The Entropy of AI: What We Lose When We Stop Thinking

While visiting Hamburger Bahnhof, the former railway station turned contemporary art museum in Berlin, I came across a work by [materialistin] — Matter of Care, Care of Matter.

I don’t want to write about the piece itself. It just gave me a starting point for a different thought, one I’d been circling for a while without a good way to say it.

Every complex system needs care. Matter left unattended slowly falls apart. Systems nobody understands eventually start doing things nobody asked for.

I keep noticing the same pattern in AI.

We’re stacking an entire layer of artificial systems on top of everyday human work: connecting models to business processes, spinning up agents, automating decisions, handing off more and more to machines. And there’s a question underneath all of this that almost nobody is asking out loud:

Who’s going to understand these systems in five or ten years?

The entropy of AI driven systems

The scary version of AI isn’t the one where machines get too smart. It’s the one where the systems get so tangled that nobody (not the people who built them, not the people using them) actually knows what’s happening inside anymore.

I keep reaching for the word “entropy” here, and I think it’s more than a mood. In physics, entropy is what happens to a closed system left to itself: it drifts toward disorder unless something actively works to keep it organized. Norbert Wiener, who coined the term cybernetics, built his whole theory of control around this idea. He treated information as a kind of negative entropy, and feedback (checking what a system is actually doing against what it’s supposed to do) as the thing that holds disorganization at bay. Stop feeding in that feedback, and things drift. Nobody has to break anything on purpose. It just happens.

That’s the drift I’m describing with AI, except it’s happening to understanding rather than matter. The world isn’t getting more chaotic in some physical sense. The gap between what these systems do and what we can actually account for just keeps widening, and nobody’s putting in the work to close it.

You can already see early signs of this. Companies are inventing job titles for it: validators, process owners, people whose whole job is making sure the machine did the right thing. But there’s a catch nobody likes to say out loud: to validate something complex, you need to understand the context better than the system itself does. So what happens once that context lives mostly inside the model instead of inside a person?

The human stops checking decisions and starts checking outputs. That’s a smaller job than it sounds like. The human stops being the source of the change and becomes the person who signs off on it. An operator, basically.

Push this far enough and you get a situation where nobody can explain why a system did what it did. Not the person using it. Not the model itself, not really. Everyone just keeps using the result anyway, because it mostly works.

That’s the entropy I mean: complexity growing faster than our ability to keep up with it. And at some point AI stops being the fix for this problem and just becomes another part of it — one more piece of the fog.

Putting names to it

I’m not the first person to notice this, obviously, and it turns out there’s already decent vocabulary for pieces of it, borrowed from philosophy of science and complexity theory.

There’s epistemic opacity, a term from Paul Humphreys, who spent years thinking about computer simulation. His point was narrow but useful: a process is epistemically opaque to you if you can’t actually verify the steps connecting its inputs to its outputs. He went a bit further with what he called essential opacity, cases where it’s not that nobody happens to understand the process yet, it’s that no human ever could, given how fast and how much computation is involved. His real argument wasn’t that computers can’t be trusted. It’s that computational science quietly changed what counts as knowing something. We used to check the process. Now we check the result and hope. Which is exactly the shift from validating decisions to validating outputs, described a different way.

There’s also computational irreducibility, which comes from Stephen Wolfram’s A New Kind of Science. The idea is that some systems can’t be predicted by any shortcut, the only way to know what they’ll do is to actually run them, step by step, all the way through. No formula lets you skip ahead. A lot of what we’re building now (huge models trained on enormous, shifting piles of data, chained into agents that call other agents) behaves exactly like this. You can’t reason your way to the answer in advance. You have to run it and see.

And then there’s something more mundane, which I’d just call complexity explosion, though it goes back to Herbert Simon’s older work on why complex systems tend to organize themselves as hierarchies of smaller, mostly independent parts. As you connect more models and more agents and more automated decisions to each other, the number of ways they can interact grows a lot faster than the number of pieces itself. Hierarchy is often the only thing keeping a system comprehensible at all. Flatten it (let everything talk to everything) and the space of things that could happen explodes past whatever a handful of “validators” could realistically check.

Line these three up and you basically get my whole argument in someone else’s words: complexity explosion is the engine, computational irreducibility is why you can’t cheat your way around it, and epistemic opacity is what’s left over — a growing pile of processes nobody can fully account for, even in principle.

There’s an older, kind of blunt way of putting why this matters for control specifically. The cybernetician W. Ross Ashby argued that something can only control a system if it has at least as much internal variety as the thing it’s controlling. Once a system’s behavior outgrows what the person watching it can actually represent in their head, oversight becomes theater. It still looks like someone’s in charge. Nobody is.

Brain first, not AI first

Here’s the principle I keep coming back to: brain first, not AI first.

AI is genuinely great at routine: repetitive work, chewing through huge amounts of information, spotting patterns, producing drafts, automating things we already understand well.

But anywhere a task actually requires thinking, the human needs to go first. Deciding something without full information. Picking a direction. Noticing a problem before it’s obvious to anyone else. Making something that didn’t exist before. This is where the human edge still lives.

If someone stops thinking and analyzing and creating on their own, they become a very easy thing to replace with an agent. Not because the AI is better than them at everything, but because they’ve quietly handed over the exact part of the job that made them valuable in the first place.

The question isn’t how many tools you can operate or how many agents you can run at once. It’s what’s left of you once all of that gets taken away.

The disappearing line between specialists

Something else is happening alongside all this: professions are converging. IT has been drifting this way for years, powerful tools got cheap and easy, knowledge got easy to find, and the gap between specialists stopped being about who had access to information and became about who could actually use it well.

AI speeds this up a lot. Everyone’s looking at the same interface, the same models, the same tools. The gap isn’t about who can open an AI assistant anymore. It’s about what someone brought to the table before they ever touched it: years of experience, connections between fields that don’t obviously belong together, the kind of intuition you can’t shortcut.

Take all of that away and leave only “who writes better prompts” or “who can juggle more browser tabs” and the advantage just evaporates. We’re at real risk of handing AI exactly the things that used to make one professional different from another.

Context as part of the work itself

Something similar is happening in art. As the internet fills up with AI generated stuff, people are going to keep asking: does it even matter? If an image makes you feel something, does it matter whether a person painted it? If a song moves you, does it matter whether a musician actually made it?

At the level of just consuming something, maybe sometimes it really doesn’t. But art was never only the object. A painting isn’t only an image. A song isn’t only a string of sounds. There’s a story behind them. We don’t even need to know an ancient artist’s name to find ourselves imagining who they were, how they lived, what they saw, what it felt like to make the thing. Even an anonymous creator is still baked into the work somehow. The time, the culture, the circumstances, the person’s own life, all of it becomes an invisible layer of meaning.

Which is why passing off AI-made work as human-made ruins part of it. It’s not only a question of who gets the credit. It’s that you’ve built a false context around the object, it starts telling a story about a process that never actually happened.

Of course, you can imagine a further stage. What if AI eventually becomes a genuine creative subject with its own history, its own experience, something it actually wants to express? At that point it stops being a tool. It becomes a different kind of author entirely.

But today’s generative AI mostly isn’t that. What it does is closer to: a prompt, a computation, a generation, a result. You could push a layer deeper and ask whether the person who wrote the prompt is the author. What were they feeling? Was that a creative act, or just a well built instruction? Maybe one day someone will try to measure brain activity during the process and figure out whether something creative actually happened. That question probably belongs in a future season of Black Mirror, not here.

What actually stays human

AI isn’t really the problem. We are — the moment we stop doing our part. Stop deciding things for ourselves. Stop understanding what these tools do and just trust whatever they hand back.

I don’t think the answer is to ignore AI. Handing everything over to it isn’t the answer either. It’s simpler than that: keep thinking for yourself, and let AI help with that instead of doing it for you. Paying attention is what keeps things from falling apart. Stop paying attention and nothing bad happens right away — that’s the tricky part. It just slips, little by little, and you don’t notice until it already has.

Matter of Care. Care of Matter. Different matter now: data, models, agents, not stone. Same deal, though. Someone still has to look after it or it comes apart too.