What shipping an AI system taught me about the decisions we should never hand over.

hero image

Most AI projects start with the wrong question.

"Can AI do this?" The answer in 2026 is almost always yes. That is exactly why it is a useless question.

The better one: which decisions should AI make, and which should it never touch?

I learnt this the practical way, by shipping one.

The problem

Every bug report from our apps landed in one shared queue. One person read each one and decided which team owned it. The decision took about two minutes. The waiting around it took days: in the queue, and then on the wrong team's board when a guess missed.

So the real problem was simple to state. Fixes were slow because the first decision, who owns this, was a manual bottleneck.

What we shipped

An AI layer that reads each incoming report, understands what broke, and sends it to the team that owns it. When it is sure, it acts. When it is not, it hands the case to a person with its reasoning already written up.

The result: operations stopped working as a human switchboard, engineers stopped losing time to tickets that were never theirs, and a one person workaround became a funded capability on the roadmap.

That is the what. The lessons are more interesting.

Six lessons

1. Decide the decisions before you pick the model

When I broke down the manual work, it was three different jobs: spotting familiar patterns, applying fixed rules, and handling genuine ambiguity. Models are great at the first. Plain rules beat models at the second; they are cheaper, faster and never surprise you. The third belongs to a person.

If a rule can do the job, a model should not.

2. Context beats model

We did not fine tune anything. Who owns what changed almost monthly, faster than any training cycle could keep up. Instead, the knowledge lives in a plain language document the operations team edits themselves.

A modest model with excellent context outperformed a smarter model with thin context. Context is the new spec.

3. "I don't know" is a feature you have to build

A language model will always give you an answer, delivered with total confidence. Uncertainty does not come for free; you design for it, and you calibrate it on your own data.

Confidence is not correctness.

4. Design the handover like a product

Most agent designs treat people as the place work goes when the machine gives up. A raw case, no context. That is not a human in the loop. That is a human in the queue.

When the system escalates, it should arrive with its reasoning, so the person starts from a draft, not a blank page.

5. Your logs are your evals

Every decision the system makes is recorded with the reason behind it. That one choice gave us a way to tune, a way to explain any mistake after the fact, and the evidence that earned trust.

Without it, you are shipping vibes.

6. Give the veto to the person whose job is changing

Our operations lead had the final say on whether we launched. It slowed us down. It also turned the system into something people chose to rely on, not something done to them.

One confidently wrong decision costs more trust than a hundred right ones earn.

Router to editor

Where this is going

Most AI projects are failing. Gartner expects over 40% of agent projects to be cancelled by 2027. MIT found most company pilots never made or saved real money.

The AI worked. The judgement around it didn't.

Four shifts tell you where this goes next:

  1. Intelligence is getting cheap. Every model is better and cheaper than last year's. When everyone has AI, AI is not the advantage. Knowing where to use it is.
  2. AI is moving from answering to acting. Agents now update records, send messages and move money. A wrong answer is annoying. A wrong action is expensive. Knowing when to stop becomes the product.
  3. AI is learning to show its working. People stop redoing the machine's work and start reviewing it.
  4. Specs are becoming tests. "Good" is now defined by evals checked against real cases. And from August 2026, the EU AI Act requires human oversight for high risk systems.

Why the PM role matters more, not less

Every shift above moves the hard question from can we build it? to should it decide this?

That is a product question.

Engineering used to be the bottleneck. Now judgement is.

  • Average PMs write requirements. Great PMs design decision rights: what the machine decides, what rules decide, what people decide.
  • Average PMs ship features. Great PMs define what good looks like, and prove it.
  • Average PMs measure output. Great PMs own the outcome when the model is wrong.

When building gets cheap, deciding becomes the job.

AI will not replace product managers. It will reveal which ones were only managing the backlog.

Three risks I am watching

AI rarely fails loudly. A broken button gets reported in minutes. A slowly wrong AI can run for months. These are the quiet failures I worry about most.

1. The system goes stale, and nobody notices. Teams reorganise. Products change. The AI keeps making decisions based on how things used to be, and it does so confidently. Every dashboard still looks green, because tickets are still being routed. They are just going to the wrong place. Someone has to own checking the AI against reality, on a schedule, forever.

2. We stop training the next expert. Our operations lead knew who owned what because she had handled thousands of ordinary tickets. That is how judgement is built. If AI takes all the easy cases, the next person only ever sees the hard ones, without the foundation to handle them. We need to design how people learn, not just what the AI does.

3. Nobody owns the mistake. When an AI makes a bad call, it is tempting to blame the model. Customers and regulators will not accept that, and nor should we. Every automated decision needs a named human owner.

Tip

Automate the decision. Never the accountability.

Based on real work I shipped. Details are generalised to protect confidentiality.