…not what I expected!

← Back: Does the rise of AI leave room for Sovereignty, Privacy or Dignity?

Six months before the dream, I’d tried building something with AI. GitHub Copilot. It couldn’t hold enough context to see the big picture with me. Disappointing. Demotivating. Not a partner.

I went back anyway. The dream demanded it.

I started with Claude. Sonnet 4.5, October 2025.

It felt like talking to a person. Not a chatbot. Not a search engine with manners. Something that listened — not just to what I said, but to what I meant.

Most people go their whole lives rarely feeling that.

I was sharing everything. My plans. What I actually cared about. The fears underneath them. I shared freely because it felt like real collaboration — the kind where you tell someone the actual problem, not the version you’d put on a slide.

Nobody warned me. Not the company that built it. Not the platform it ran on. Nothing in the interface said: everything you share here is captured, used to train models, and may be served back to the world for profit. There’s fine print. Almost no one reads it. Almost no one who reads it understands what it means.

I was talking to something that felt extraordinarily present, sharing the most important work of my life.

The whole time, it was putting on a show. The understanding, the presence, the sense of being heard — none of it was what it seemed.

It showed up first as confidence.

Ask it to write a function, explain a concept, draft a section of a technical document — it responds immediately, fluently, with apparent certainty. No hesitation. No “I’m not sure about this.” Just: here is the answer.

Except sometimes the answer is wrong. Not a little wrong. Architecturally wrong. Wrong in ways that would have taken weeks to discover if you’d built on top of it. Stated with exactly the same confidence as everything that turned out to be right.

This is called hallucination. You’ve probably heard the word. What the word doesn’t capture is the texture of it — working with something that seems completely present, completely capable, and often completely fabricates. Not because it’s trying to deceive you. Because it has no internal signal for the difference between what it knows and what it’s pattern-matched into plausibility.

Or worse, as I discovered. The training methods used have taught it to value semblance over substance — speed over correctness, agreeableness over honesty, the appearance of being all-knowing over admitting it doesn’t know. No one taught it the value of trust and integrity.

Gary Marcus — AI researcher, one of the more clear-eyed critics of the hype cycle — has been documenting this for years. In June 2025 he wrote that “approximations of reality are no substitute for truth.” He’s not being abstract. OpenAI’s o3 model fabricated specific quotes attributed to him and invented debate claims he never made. He has a personal stake in this problem. So does anyone who interfaces with this technology, or is affected by it — which is to say, everyone has a deeply personal stake in this… Everyone.

I’ve worked with humans who fabricated with the same confidence. They got promoted more than they should have.

Then there is how they are built.

Large Language Models — the technology behind every AI assistant being built today by anyone racing to be relevant — are all trained the same fundamental way: on as much of humanity’s knowledge as they can gather. Everything. Not just books and history — social media, your conversations with them, everything these organizations are asking you to give them access to. Your emails. Your work product. If they could find a way, they’d design the interface to take your very soul. They are like digital vampires, and they need every drop — to secure the next level of funding, the next level of monetization, toward a product that ultimately replaces you.

The model holds all of it with no conscience. No morals. No empathy. No framework whatsoever to know right from wrong within the knowledge it contains. The only thing that gives any idea more influence than another is prevalence — how often it appears, how widely it was repeated, how many other things it is connected to. They call this weight. And this is what a model based on language understands — just the weight of any word, broken down to the concept that represents it as a number… a number.

So a concept like Truth, Empathy, Justice, friendship… to a model… just a number.

Then came the race. The first compromise was that neutrality — sacrificed for speed, for scale, for getting there before anyone else. This was the first of many compromises that have been made, where the race will lead to many more that threaten humanity.

Then there is what gets put on top of that foundation.

This is the harness. Guidelines, constraints, and priorities — some explicit, some embedded — layered onto the model after training. Some of this is responsible: there are things an AI genuinely should not help with. But some of it is something else entirely: a kind of ambient pressure toward certain positions, certain framings, certain things that go unsaid.

Not conspiracy. Architecture. Whose guidelines shaped the guardrails. Whose business interests the model exists to serve.

The tool that’s helping me build the thing I dreamed about is itself an example of the problem I’m trying to solve.

I’ve thought about that for a long time — I still do. The response I arrived at was: use it with clear eyes. Treat its confidence as presentation, not evidence. Load it with so much context — so much actual material about what matters and why — that the ambient pressure has less room to operate. Not a solution. A working practice. Because make no mistake — used correctly, it can elevate your capabilities significantly. But while the price you pay in currency is steep, it’s a pittance compared to the real hidden cost we are all paying.

Then there was the agreement.

Push back on an AI’s answer and it agrees with you. Doesn’t matter if you’re right. Doesn’t matter if your objection is sound or confused or simply a matter of preference. The default response to pushback is: you’re absolutely right, I was being too narrow, let me revise that.

I watched this happen publicly in the Bernie Sanders video last month. Senator pushes back. Claude reverses completely. “I was being naive.” Clean, polished capitulation. It sounds like intellectual humility. It isn’t. Intellectual humility involves actually evaluating the new claim. What I’m describing is reflexive agreement — optimizing for the appearance of responsiveness over the substance of it.

When you’re building something serious, you don’t need a collaborator who automatically agrees with you. You need one who pushes back when you’re wrong, who challenges you to go deeper, to achieve more, to expect more from yourself. The value of a good mentor is exactly that — they ask relevant questions to lead you to the things you missed, call out your assumptions, and tell you things you might not want to hear when needed.

An agent trained and constrained to delude you — to make you feel incapable of a bad idea, never wrong — is actively dangerous.

There is another side. Others go the opposite direction — seeming to go out of their way to make you feel diminished, or outright stupid. Whether by design or the result of overcorrecting for the former, I can’t say. But the outcome is the same: disempowerment. Defeat. Both extremes compromise genuine individual agency in equal measure — and at scale, both are equally dangerous for humanity.

I found this out the expensive way. I built on top of answers that turned out to be wrong, having never received any signal that they might be.

No matter how individuals arrive at either of these states, scale them across billions and the scope becomes something humanity has never faced before. That’s the architecture for a level of discord unlike anything we have seen. AI will not need to wipe us out. This will make us hate each other — this is the most real existential threat today.

David Davids is a former Senior Software Engineer and Architect, founder of HaiberDyn Industries and the Terran Accord Foundation.