We Never Know What We Are Talking About

AI, the Autonomy of Knowledge, and Why Institutions Can't Control What They Create

W. W. Bartley III, the philosopher and Hayek’s biographer, once offered a koan for epistemological enlightenment. When asked what he had learned from the two thinkers who most shaped his worldview, he said: From Popper, I learned that we never know what we are talking about. From Hayek, I learned that we never know what we are doing.

I have been turning this line over in my mind for a while now, because I think it contains, in compressed form, the deepest thing anyone has said about that can be applied to what artificial intelligence and generative artificial intelligence are doing to institutions. Not the usual story about disruption, or job losses, or even the one I have been telling myself about on “knowledge production” and how execution costs collapse and judgment becomes the bottleneck. Something more fundamental. Something about the nature of knowledge itself, and about why every attempt to govern AI by controlling its outputs is doomed to reproduce the very problem it is trying to solve

In my last post, I argued that AI shifts the institutional bottleneck upstream. When technically competent artifacts become cheap (papers, policy briefs, research proposals, or student essays), they can no longer serve as reliable signals of knowing. The proxy mechanism by which institutions have governed knowledge for centuries (evaluate the output, infer the competence) quietly fails. The binding constraint migrates from execution to judgment, and institutions organised around the old scarcity find themselves misallocating effort on a massive scale.

That argument, I now think, was correct but incomplete. It described the shift without explaining why it is so disorienting or why institutions keep reaching for the wrong solutions.

The missing piece is this: knowledge has always been autonomous. Our products have always escaped our control. AI did not create this condition. It made it impossible to ignore.

---

The unfathomable depths of what we create

There is a tradition in philosophy, largely ignored in the AI governance conversation, that takes seriously a simple but unsettling observation: we never fully understand what we produce.

Karl Popper spent decades developing this insight. When we create a theory (or a policy, or a curriculum, or an institution), we also, by logical necessity, produce an infinity of implications we cannot foresee. The content of an idea is not identical with its creator’s thoughts about it. It includes every situation to which the idea might apply, every combination with other ideas, and every consequence that follows from its internal logic. Many of these situations have not been imagined. Some are, at the time of creation, literally unimaginable.

Popper’s favourite example was from mathematics. When humans created the sequence of natural numbers, they also inadvertently created the distinction between odd and even numbers, the question of whether there is a largest prime, and eventually the incompleteness theorems that Gödel would discover millennia later. None of this was intended. None of it could have been anticipated. It followed from the logic of the product itself, not from the intentions of its creators.

The same holds for scientific theories. Part of the informative content of Newton’s mechanics was that it would conflict with a theory (Einstein’s) that could not even have been conceived at the time. Schrödinger did not fully understand his own equations before Born reinterpreted them, and even then, he disliked what they seemed to imply. The product’s content exceeded the producer’s comprehension.

This is not an occasional failure. It is the normal condition of knowledge. Our knowledge is, at any given time, what Bartley called “unfathomed”, that is, richer and deeper and more consequential than anyone who produced it or uses it can fully grasp. It generates problems independently of anyone’s intentions. It develops in directions no one designed.

And this matters profoundly for the AI conversation, because it means that the demand most institutions are making right now, that knowledge production remain under human control, that outputs be traceable to individual competence, that the epistemic chain of custody stay intact, is not merely difficult to satisfy. It is logically impossible. It was always logically impossible. AI just made the impossibility visible.

---

Why our products must escape our control

The young Marx, in his Paris manuscripts of 1844, gave voice to perhaps the most emotionally compelling version of the opposing view. He argued that our products, that is, the fruits of our labour, the expressions of our being, should remain ours. When they are wrested away, alienated, made strange to us, we lose something essential. He puts it starkly: the great evil is the consolidation of our product into an objective power over us, one that outgrows our control, crosses our expectations, and nullifies our calculations.

The sentiment resonates. It resonates when academics watch AI generate essays indistinguishable from those of their students. It resonates when policy analysts discover their carefully crafted briefs can be replicated in seconds. It resonates when the artefact that once stood as evidence of intellectual labour suddenly feels worthless. There is a genuine sense of loss here, and it would be dishonest to dismiss it.

But as Bartley showed in a brilliant essay that deserves far more attention than it receives, Marx was demanding something that is both logically and physically impossible. Not because capitalism is inevitable, or because markets are sacred, but because of the nature of knowledge itself. Our products must escape our control. They do so whether we surrender them to another person or try to keep them to ourselves. They do so because they are unfathomable, because their content exceeds what any creator can anticipate or any owner can direct.

The implications are profound. If we never fully know what we are talking about, then the knowledge we produce cannot be an “expression” of our inner states. It exceeds us. It may have consequences that are not only unforeseen but also contrary to our preferences. A theory we create may undermine beliefs we hold dear. A policy we design may produce results opposite to those we intended. An assessment framework we build may measure something quite different from what we thought it would.

This is not a bug in the system. It is the system. It is how knowledge grows: by escaping the control of its creators and encountering criticism, application, and recombination in ways no one could have planned.

---

The expressionist fallacy and the AI panic

There is a name for the error that underlies both Marx’s account of alienation and much of the current institutional panic about AI. Bartley called it the “expressionist” approach to knowledge: the assumption that an individual’s products are externalised reproductions of their inner self, that they stem from their most intimate being, and that their worth depends on the “authenticity” of the mental states that produced them.

This assumption runs deep in institutional life. Universities grade essays as expressions of individual understanding. Research councils evaluate proposals as evidence of the investigator’s intellectual command. Government agencies treat policy briefs as reflections of analytical competence. Professional credentialing systems of all kinds rest on the premise that the quality of the artifact reliably signals the quality of the mind behind it.

AI breaks this assumption, and the institutional response has been almost uniformly to try to restore it: more detection tools, more authentication requirements, more in-person assessments, more surveillance of the production process. Turnitin for the universities. Audit trails for the agencies. Provenance tracking for the journals.

But the expressionist assumption was always wrong. Not because people were cheating before AI, but because knowledge products were never reducible to “self-expression” in the first place. They were always richer, more consequential, and more unpredictable than any individual mind could encompass. What AI has done is make the gap between the product and the producer so visibly, undeniably wide that the old proxy mechanism can no longer be maintained even as a convenient fiction.

This is why the “cheating crisis” in universities is actually an assessment design crisis. This is why AI governance frameworks built around output monitoring are structurally inadequate. This is why the entire apparatus of artifact-based epistemic authority is failing. The institutions are trying to restore a relationship between producer and product that never existed in the form they imagined.

---

What the Austrian Economists understood about knowledge that the planners did not

The deepest engagement with the autonomy of knowledge in economic thought comes not from mainstream economics but from the Austrian Economics tradition, and specifically from Friedrich Hayek, whose work on the dispersed, tacit, and irreducibly local character of knowledge remains one of the most underappreciated contributions to institutional theory.

Hayek’s central insight, developed across decades from “The Use of Knowledge in Society” (1945) through his Nobel lecture “The Pretence of Knowledge” (1974), was that the fundamental problem of social coordination is an epistemic one. The relevant knowledge in any complex system is not concentrated in any mind or authority. It is dispersed among countless individuals, much of it tacit, contextual, and inaccessible to formal articulation. No central planner or central planning algorithm can aggregate it.

The market, in Hayek’s account, is not primarily a mechanism for allocating resources. It is an epistemic infrastructure: a system through which dispersed knowledge gets communicated, tested, and used, without anyone needing to possess the totality. Prices function as compressed signals that encode more information than any participant can articulate, coordinating action across the system without central direction.

Bartley generalised this insight in a way that connects directly to the AI question I’ve been interested in developing through this Substack. The market doesn’t just make use of existing dispersed knowledge. It elicits new knowledge that no participant yet possesses. In the competitive interaction of dispersed, specialised individuals, each bringing their own partial understanding to bear on an unfathomable product, they collectively discover potentialities that no one had imagined. The market is a discovery process (in Hayekian language), and so, Bartley argued, is every other institutional setting where knowledge is examined, criticised, and recombined: the seminar room, the peer review process, the policy debate.

This is the constructive alternative to the expressionist fallacy. We do not need to “own” our knowledge products to benefit from them. We need to be able to exchange them, to submit them to others who will find in them what we missed, probe implications we hadn’t considered, and discover uses we hadn’t imagined. We need criticism, not custody. We need an ecology, not a chain of command.

---

The cybernetic temptation

Here, a historical thread needs pulling, one that connects the mid-twentieth century to our current moment more tightly than most AI commentators recognise.

In the 1960s and 1970s, cybernetics offered what seemed like the definitive solution to exactly this kind of complexity problem. If the challenge is coordinating vast quantities of dispersed knowledge, then design the right feedback mechanisms (information flows, sensors, control loops), and you can steer the system without needing to understand every component. Stafford Beer took this furthest with CYBERSYN in Allende’s Chile: a system designed to regulate the entire national economy through real-time data and cybernetic feedback.

Hayek’s relationship to cybernetics was far more complex than is usually understood. He attended the 1960 Symposium on the Principles of Self-Organisation at the University of Illinois, where Beer also presented. He drew extensively on the cybernetic concept of negative feedback. He acknowledged openly that the market’s self-regulating mechanism is, in cybernetic terms, a feedback system: what Adam Smith had called the invisible hand.

But Hayek also grasped something that cybernetics, in its more ambitious governance applications, consistently underestimated: the knowledge that matters most in complex social systems is not the kind that can be captured in information flows. Much of it is tacit. Much of it is contextual. And crucially, much of it is unfathomable, that is, not yet articulated, not yet discovered, existing only as unrealised potential in products whose implications have not been fully explored. No feedback mechanism, however sophisticated, can capture knowledge that does not yet exist in any mind.

The parallel to contemporary AI governance is direct, and I think it is the most important one available.

Most current AI governance frameworks implicitly adopt a cybernetic model. If we can design the right monitoring systems, such as audit trails, algorithmic transparency requirements, compliance dashboards or risk registries, we can steer AI’s institutional effects. More data, better feedback, tighter loops.

This is the cybernetic temptation in its twenty-first century form. And it fails for the same reason the more ambitious versions of cybernetic governance always failed: because the relevant knowledge is not the kind that can be captured in monitoring systems. What AI transforms is not just information flows but the epistemic/knowledge infrastructure through which institutions constitute what they count as knowledge. The interface performs epistemological work, converting statistical patterns into institutional facts, probabilistic outputs into deterministic-seeming intelligence, and generation into the appearance of retrieval. No audit trail captures this. It operates at the level of what questions seem askable, what framings seem natural, and what counts as a coherent argument. It reshapes the landscape of the thinkable.

---

Knowledge vehicles and the ecology of institutions

There is one more idea from this tradition that I think deserves much wider circulation, because it points toward a constructive response rather than merely diagnosing the problem.

Donald T. Campbell, evolutionary epistemologist and one of the most rigorous thinkers on the social organisation of knowledge, introduced the concept of knowledge vehicles: the carriers, media, and institutional structures through which knowledge is embodied and transmitted. Every vehicle (a language, a notation, a journal system, a curriculum, a bureaucratic process) has its own structural characteristics, and these characteristics inevitably introduce distortions. The vehicle is never a transparent window onto the knowledge it carries. It shapes, constrains, and sometimes warps what it transmits.

Think of a mosaic depicting a landscape. The size of the stone fragments limits the detail that can be rendered. The rigidity of the structure constrains what can be represented. The available colours determine which hues appear. The mosaic is a vehicle for a visual message, and its vehicular characteristics inevitably intrude upon the message itself.

The same is true of every institutional vehicle for knowledge. The peer review system, essential for holding the scientific community together, also introduces distortions: it rewards conformity, encourages tribalism, and sometimes suppresses precisely the kind of critical challenge it was designed to facilitate. These are not bugs. They are vehicular characteristics, or the structural features of the carrier that serve the necessary function of holding the vehicle intact, but that inevitably distort the knowledge it transmits.

Now here is the crucial move: these distortions are not fatal, because they can be corrected. Not perfectly, and not without introducing new distortions of their own. But they can be improved, iterated upon, redesigned. The question is not how to eliminate distortion, which is impossible, but how to design the institutional ecology so that distortions are identified, contested, and progressively reduced.

Bartley called this the “ecology of knowledge”: the question of how intellectual life, institutions, traditions, and even etiquette can be arranged to expose beliefs and practices to optimum criticism, counteracting error while maintaining the fertility of the intellectual environment. Not a plan, but an ecology. Not a control system, but a garden.

This is exactly the framing that AI governance needs and overwhelmingly lacks. The question is not: How do we control AI outputs? The question is: How do we redesign the ecology of institutional knowledge, the vehicles, the practices, the protocols, the norms, so that they remain capable of identifying and correcting distortion in a world where AI is a participant in knowledge production?

---

From production to navigation

If knowledge is autonomous and has always escaped the control of its creators, always richer and more consequential than anyone who produced it could grasp, then the institutional response to AI cannot be to restore the old relationship between producer and product. That relationship was always a useful fiction. AI has simply made the fiction untenable.

The real shift is from production to navigation. The scarce competence is no longer the ability to generate knowledge artifacts. It is the ability to move through knowledge landscapes: recognising adequacy and inadequacy, detecting errors and distortions, selecting and recombining appropriately, knowing when a framing is wrong even when the output looks right.

This is what I have been calling the curational turn, the transition from authorship as creation to authorship as navigation, from the writer as producer to the writer as selector and critic. It is also, I think, the deepest version of the idea that question formulation, not execution, is the scarce resource in a world of cheap research production.

Popper’s evolutionary epistemology helps explain why this shift matters so much. If knowledge evolves (if it generates its own problems, develops its own contradictions, produces unexpected implications), then the relevant epistemic competence is not production but selection. Just as biological evolution proceeds through variation and selective retention, epistemic evolution proceeds through the generation and critical evaluation of knowledge structures. AI accelerates the variation side of this process enormously. What it does not do, what it cannot do, is perform the selective function. It generates. It does not navigate.

And navigation, unlike generation, cannot be automated, because it requires judgment about the unfathomable. It requires the capacity to recognise that an output is inadequate for reasons that are not fully articulable, because the framing was wrong, because the question was poorly specified, because the theoretical structure was unsound in ways that only become visible when someone with the right background encounters the product and finds in it something that the producer, whether human or machine, could not have anticipated.

This is what institutions need to cultivate. Not surveillance of outputs, but the capacity for navigation.

---

Alienation as transcendence

I want to end with an idea that may seem counterintuitive, but I think captures something essential about the moment we are in.

The experience that people call “AI disruption” in institutional settings, the sense that the artifacts we produce no longer fully belong to us, that the knowledge we work with has become strange, that the relationship between effort and output has been scrambled, is a form of alienation. And the instinctive response is to try to overcome it: to reassert control, to restore ownership, to demand that the product remain securely tethered to the producer.

But there is a tradition in epistemology, one that runs through Popper, Hayek, and Bartley, that says alienation, properly understood, is not the enemy. It is the condition of transcendence.

We grow by making our ideas strange to ourselves. We advance by submitting our products to criticism that reveals in them things we could not see. We transcend our earlier selves precisely by allowing our knowledge to escape our control, to be tested and transformed in encounters we could not have designed.

As Bartley wrote, the fundamental task of education is unlearning: making ourselves, and the ideas in which we conceive and create ourselves, strange and alien, and thereby transcending our old selves. The teacher presents a structure of knowledge that she does not fully understand to a student who also cannot hope to fully understand it. This is the smallest social unit in the marketplace of ideas, and it illustrates how, despite all distortions, such a market acts as a discovery process.

AI intensifies this condition. It makes the autonomy of knowledge visible in daily institutional practice in a way that was previously confined to abstract philosophy. It confronts everyone (not just epistemologists) with the reality that our products escape us, that our knowledge is unfathomable, that we never fully know what we are talking about or what we are doing.

The institutions that understand this will not try to restore the old fictions of epistemic control. They will redesign around the actual dynamics of knowledge: autonomous, unfathomable, evolving, exceeding every attempt at mastery. They will build ecologies, not control systems. They will cultivate navigation, not production. They will treat alienation not as a threat but as the price, and the precondition, of intellectual growth.

The ones that reach for more surveillance, more compliance, more output monitoring, will discover that they have optimised for a world that no longer exists, and that the knowledge they were trying to govern has, as it always does, outgrown their control, crossed their expectations, and nullified their calculations.

---

*This thought piece develops themes from Karl Popper’s Objective Knowledge: An Evolutionary Approach (1972) and his 1978 Tanner Lecture “Three Worlds”; on Friedrich Hayek’s “The Use of Knowledge in Society” (1945) and “The Pretence of Knowledge” (1974); on W. W. Bartley III’s “Alienation Alienated: The Economics of Knowledge versus the Psychology and Sociology of Knowledge” (1987) and his concept of the ecology of knowledge; on the growing literature on Hayek’s engagement with cybernetics, particularly Gabriel Oliva’s “The Road to Servomechanisms” (2015); and on Stafford Beer’s CYBERSYN project as documented in Eden Medina’s Cybernetic Revolutionaries (MIT Press, 2011).

---

Disclaimer: I used Claude as an editorial and structural tool in drafting this piece. The ideas, arguments, and intellectual framing are entirely my own, and I take full responsibility for them.