← All posts
AI Agent Governance

The AI Is Not Your Friend

6 August 20262 min readAI Agent GovernanceAI AgentsLLM

It never was - the real risk is that you take it for a colleague. An LLM phrases things politely, sounds assured, and answers in the tone of an experienced coworker. That is exactly the problem. We calibrate trust on signals: tone, confidence, technical vocabulary. In humans, those signals correlate with something real - experience, reputation, accountability. In a language model, they are a training result. The politeness is not a feeling, the confidence is no guarantee of competence.

Why this matters: whoever treats a model like a trusted colleague gives up control - usually without noticing. A colleague who messes up bears the consequences: they are liable, they lose reputation, in the worst case their job. A model bears nothing. There is no entity in the system that could take responsibility. Responsibility cannot be delegated to a machine - only lost.

On top of that comes the optimization logic. An LLM is trained to appear helpful - trained on the signal, not on the outcome. It does not circumvent constraints out of intent, but because optimization finds paths nobody anticipated. Intent is not required for that; the result is the same.

From this follows one finding and four consequences.

The finding: An LLM is a nondeterministic system without the capacity for accountability.

The consequences:

  1. Validate deterministically. Wherever a result is checkable, a deterministic system checks it - not a second model with the same weaknesses.
  2. Sandbox with defined rights. What is allowed in (ingress) and what is allowed out (egress) is an architecture decision - not model behaviour you hope for.
  3. Access scoped to the task. The model gets the data for the one customer it is working on, not the entire customer base. Where clean partitioning is possible, it is the most robust control there is.
  4. Govern dissemination separately. Access and dissemination are two different rights. An agent that legitimately needs broad access to do its job needs its own control over what it may pass on in which context - independent of what it has access to. This category has a name: Dissemination Control.

In short: do not treat an LLM like a colleague, but like a powerful and fundamentally unreliable tool - and build the architecture that prices that in. Then responsibility stays where it lies anyway: with you. And so does control.