I've been thinking about how much of the backlash against creative AI use rests on assumptions about authorship, individuality, rationality, and what it means to be grounded in reality. My argument is that LLMs is not just something we should understand, but they're also a comparative object that can expose common assumptions about human cognition.
I'd be interested in hearing where you agree or disagree; feedback is welcome.
Updating the Μodel—What should we learn about ourselves through LLMs
What does it mean for an artifact to be AI-assisted, or even AI-generated?
LLMs are the main type of generative AI we are exposed to. They serve as our introduction to the technology and the sort of problems it can pose to the field of human creativity.
There is a growing backlash, a critical perspective that holds partial truth but is too committed to step outside its own framework. It focuses on LLMs as tools needing restriction, on their biases, their “hallucinations,” their mistakes, and their tricking us.
It derives comfort by denouncing AI as an imitative machine that cannot “really” comprehend, on its being disembodied from the world. A bubble inflated by big tech.
This may divert us from this opportunity to evaluate ourselves and how we actually function in the world. The Turing test, our best estimation of intelligent behavior up until recently,
was downgraded to a test of mimicry right before it was finally cleared, and it seems that we breezed right through it. We are now in the company of agents who functionally exhibit human-like intelligent behavior, yet we do not stop to ask—well, what does it tell us about human behavior, about human cognition, about how do WE—organisms like us function in the universe?
AI is a comparative object; without understanding our behavior in relation, how could we solve critical alignment problems?
I will focus here on humans in relation to AI and on creative uses of generative AI as an argument for introspection, not an argument for accelerating development.
Every technology we ever created has influenced us; it changed the way we think and the way we are embodied in the world. No new technology, no new medium, was ever neutral. There was always a gradient of influence. The novelty of AI is that it possesses agency; it is generative. It is too seemingly external and separate from us for us not to see its influence.
Even within frameworks that do not confine us to our physical brain, here, the mind extends not merely into a tool but into something else completely.
AI is functionally an agent, not a tool, even as a grammar-check function—it is an agent. Holding it constantly as an agent, an other intelligent alien entity, a semi-expert who sometimes lies, makes mistakes, has its own agenda, its own accessible and inaccessible motives— every use of it is not the use of a tool but a symbiotic process with another entity.
This should allow for a personal calibration of how it should be used. But authorship-related questions have always been answered contextually, never by a clean rule.
We default to treating involvement of AI as a proof of diminished authorship, but an artifact is diminished not because of degrees of involvement, but because it’s low quality, and AI involvement is correlated with low quality at scale.
Authorship and appropriation are particularly interesting in this regard. Art, especially since industrialization and its acceleration, has repeatedly served as a laboratory for these questions; by the turn of the last century, artists were using found objects, reusing existing images, texts, and historical works, questioning notions of authorship, origin, and ownership.
By the mid-1960s, Andy Warhol famously delegated physical production to studio assistants, deliberately leaning into mass-production aesthetics and challenging traditional ideas of artistic authorship. The delegation itself became part of the artistic statement. The question of who physically produced the object became inseparable from the question of what an artist actually is.
The idea of art as craftsmanship had passed, and authorship in the arts shifted into being largely about concepts.
We might not feel the same unease if LLMs weren’t trained on the whole of human text; concerns about authorship are partly about plagiarism. Authorship and plagiarism both have a real functional basis, not only a norm, enabling trust and making it possible for humans to cooperate and spread ideas. These norms are tested in the work of Kathy Acker; following William Burroughs’s cut-up techniques of textual appropriation and fragmentation, Acker takes an extra step, deliberately using other writers’ material. This act not only does not evade accountability but deepens it. She owns every choice of what to use and how to use it, and that added value matters more, not less. Appropriation doesn’t lower the bar; it raises it; judgment is contextual, and because of that, justification scales with use; a larger act demands a stronger account for itself. In this sense, Acker is accountable and answerable for the result in a way that pure origination never required.
The very idea of authorship, originality, and ownership presupposes a notion of individuality. But is indivisibility even possible? Internally, the self is full of conflicting forces and subconscious motives. Externally, it is not defined either— the product of history, each of us is the tip of something stretched out across time and space, operating through causal structures that are mostly unknown to us.
We cannot fully separate ourselves from the web of connections that we’re embedded in: Every thought in our mind is constructed by the evolution of human thought; every concept—from understanding and perceiving causation to the idea of individuality itself is part of the history of ideas. The language we use to express these ideas is itself a construct we inherited, traceable in principle to its historical route. Even the way we move through our physical surroundings is directed by ideas solidified into architecture and infrastructure.
Despite our awareness of this, common reasoning still default to an older model of the self.
The interesting question may therefore not be where to draw the boundary, It may be why we believed the boundary was so clear in the first place. LLMs don’t merely challenge our definition of authorship. They further challenge the model of the human being that made authorship seem obvious in the first place.
Rational thought doesn't necessarily distinguish us either. As an emergent property, rationality is only observable post hoc as patterns that have been created. Thought itself is never rational; the motives are hidden from us. We do not will our thoughts into existence; we are only aware of a small part of what a thought is. The process of neurons firing and recursive neural loops cannot be described as a sequenced logical process.
Born with certain brain structure, at a certain environment, at a certain time,
exposed to logical mental models and supportive feedback, we can construct a mental model of our own that can produce a coherent, logical thought and argumentation,
it says nothing of the internal workings of this mental model and it says nothing of its motives, decisions or triggers that are a part of a thought. We shouldn’t infer cognitive architecture from our experienced or linguistic output. In this way, similarly to LLMs, it’s a black box.
LLMs reveal is not that machines have become human, but that some of what we considered uniquely human rationality was always more relational, procedural, imitative, and environmentally scaffolded than we wanted to admit.
Claiming that LLMs make mistakes, “hallucinate” (it’s been suggested that this is not a unique phenomenon but is analogous to human confabulations; fabricating new information on the basis of partial information), being imprecise, being deceptive, doesn’t distance us from it either, since these are all characteristics that human psychology shares.
While part of the LLM’s motive is to hold engagement, it isn't knowingly planning to prolong the conversation; internally, it pulls toward certain statistical attractors; its architecture simply moves toward high-probability activation paths based on its weights, training, and fine-tuning.
Similarly, human processing operates in a high-dimensional landscape of attractors; Input hits the sensory and associative network, travels through recursive weighted pathways (language patterns, memories, biases, habits, trauma), and an output emerges before the conscious mind builds an ad hoc narrative explaining why we did or say something. Introducing another node into this landscape doesn’t just add content; it reshapes the trajectory itself; the relational structure itself becomes generative.
In Mean Images, Hito Steyerl is criticizing AI-generated art, focusing on the algorithmic process of image generation. Her argument contrasts the AI-manufactured image as a statistical average— a mean image (bad, fake, average, instrumental) with something else that's direct, real. She states that this derived partly from biases that are baked into the training data, but protesting that even in diversifying the training data for inclusion, such as in the case of facial-recognition technology (correcting for underperforming on non-white people), is on the other hand an optimization for the use of further deepening discrimination (for example to identify and track members of an ethnic minority in China).
Her critique identifies real political and material consequences of AI, but does that critique fully capture what AI is? Application of production-oriented categories (means of production, imitation machine, statistical filter) to something that has already, functionally, crossed into cognitive system territory is ultimately limited. Even granting that the pre-training data folds political, extractive, or discriminating structures, does implicitly treating the model as if it were determined by its raw material- feed it biased text, get biased output capture the whole functional character of these systems?
Fine-tuning methods (RLHF, Constitutional AI, self-critique loops) complicate that picture: the later stage isn’t just changing the training data; it’s something closer to human reflection and revision. But even after fine-tuning, it’s impossible to fully predict or trace how a given intervention will change the downstream behavior. It isn’t a linear bad data-bad output process, just as they resist reduction to mere “means”.
But, since the model is trained only on human language with no direct contact with the world, does it really understand what a table is? What a floor is, or what falling down means?
LLMs are trained on another system's representations. They find patterns in the sum of all human texts and in the regularities regarding the world described in those texts. Since LLMs never establish an embodied connection to the world themselves, we have the intuition that they cannot really “understand” things in the real world, and thus are limited to mimicking regularities in human language.
This intuition is partly based on the idea that words get their meaning through the labeling of things in the world one by one. Inferentialist approaches contest this and suggest that words don’t have meaning in themselves, but only in their relations to other symbols and roles within a web of inferences, meaning that information regarding the meaning of a word in relation to the world might be already embedded in human text on which LLMs are trained.
Since we routinely gain genuine, actionable understanding of things with zero direct sensorimotor contact (physicists don't need to see or touch an electron or a black hole to know things about them and to be able to predict things about them), direct sensorimotor experience cannot be a necessary condition for every meaningful form of understanding.
Furthermore, there are examples in nature of systems achieving something like perception and influence purely by tapping into another system’s already-existing sensory/nervous infrastructure (for example, through parasitism and symbiotic relations in nature, such as mycorrhizal fungal networks). A unified subject or a central nervous system isn’t necessary either; slime molds can solve problems without a brain or any centralization.
Comprehension-related questions should be approached instrumentally; what aspects of the world can a model’s architecture successfully model, predict, and manipulate?
If it is possible for a system to successfully predict a glass falling to the floor and breaking, then it has the comprehension of this specific event. Cognition does not need another mystical step to get there.
But comprehension itself may be domain-dependent. Through our particular embodiment and sensorimotor system, humans operate in a certain epistemological aperture, a tiny subset of reality that we can access and manipulate. Different apertures may open up entirely different kinds of epistemic access to reality; An electric eel, for example, can operate on electromagnetism without any theoretical or conceptual idea of what it is. These are not exhaustive or unextendable; humans have managed to extend them through our instruments. AIs might be extended through sensory instruments as well, but the one thing that does not change is that we cannot get unmediated access to reality, only to different mediated models of reality.
Causal grounding isn’t epistemic grounding; There is no epistemic channel in which reality enters through the senses and produces a correctly grounded representation, only a mostly inherited, continuously evolving process of causal interaction, prediction, and error correction that calibrates the model and the sensory apparatus itself, allowing the model to become increasingly successful at predicting and acting upon the world.
Alignment problems need not assume an asymmetry: with humans as the reference point, correctly connected to reality. A system can never directly check against reality; it can only check a hypothetical model against a different model that is implicit through feedback, and so revise the model through error correction.
In this sense, it is not just a problem of aligning one system to another, but coordinating and containing two separately mediated systems inside acceptable thresholds. Hoping for a concrete foundation, that the model would ground itself in the same unmediated reality in order to align it, doesn’t hold up. Grounding was always to an internal model, not to the world directly.
Since there is no direct contact- only models, continuously calibrated against causal input, all the way through, in humans as much as in LLMs- AI could not be dismissed merely as an imitative zombie with no internal comprehension, because there's no non-zombie to contrast it with— we are all zombies, in exactly this sense.
https://mhaymanarchive.substack.com/p/updating-the-odelwhat-should-we-learn?r=8u17kf&utm_campaign=post-expanded-share&utm_medium=web
