The setup
Three consumer apps across several regions, each shipping the same bug reporting widget. Someone hits a problem, shakes the phone, types a sentence, and sends. The widget captures the device, the version, the network state, a screenshot and a video. It is good product design, and people use it. That is where the trouble starts.
A capture tool cannot know who owns the thing that just broke. So every report, from every app and every market, arrives in one shared queue carrying a description of a symptom and no indication of a cause. Someone then has to read it and decide.
Reporting a problem turned out to be the easy part. Three harder problems were hiding behind it: getting each ticket to the right team, getting it into a state that team can actually act on, and then resolving it. This is the story of solving all three, and of the four decisions I would defend in a review.
The problem
What I chose to optimise: time to the right pair of hands.
Not time to resolution, which depends on engineering capacity and priority calls I do not control. Not ticket volume, which is a demand signal I have no interest in suppressing. The measurable loss was the interval between a report arriving and it reaching someone who could act on it.
The decision was two minutes of real work. Everything around it was queue, and a misroute did not add delay, it restarted the cycle. Three people paid, in different currencies. The reporter, who needed evidence the report went somewhere real, because a channel nobody trusts is worse than none. The product ops lead, reading every ticket in interruptions rather than in blocks. The engineer, absorbing tickets their squad did not own.
Naming the goal that precisely was the most useful thing I did, because it made four tensions unavoidable. Every hard decision in this project is one of them, and the trade offs later resolve them in order.
- Speed against certainty. Deciding automatically only beats waiting for a human if the answer is right.
- Cost against capability. A model call is cheap once, and not cheap at the volume of an incident.
- Elegance against resilience. The cleanest architecture is often the one that quietly concentrates the most risk.
- Usage against authority. What an AI is allowed to read is a different question from what it is allowed to decide.
Root causes
I pulled several thousand historical tickets and compared where each first landed against where it ended up. Two destinations absorbed close to half of all misroutes, so precision on a few boundaries would beat a general purpose classifier.
- Reporters describe symptoms. Routing needs causes. "The restaurant list is empty" could be four owners, and adding form fields would only reduce reporting.
- The ownership map lived in people’s heads. Documentation is accurate the way a photograph is accurate. Org structures move faster than wikis, everywhere.
- A strong signal arrived unusable, as inconsistently populated free text. And repeat reports of one fault arrived as separate work, so load peaked when people were least available.
Layer one: routing
I watched what the human was doing first, because the shape of the old decision had to be the shape of the new one.
Two of those calls are pattern matching. The third is judgement about ambiguity, which is where automation does damage, so it stayed with a person.
A deterministic duplicate check runs first, so repeat reports never reach the model. Survivors get context assembled, the model returns an owner with a written justification and a confidence score, and a gate decides whether we act or escalate. Every path logs one row.
What I would defend hardest is not the model call. It is what goes in front of it.
What routing did not solve
Routing worked, and the complaint changed shape. Tickets arrived at the right team fast, and sat there.
A correctly routed ticket is not a workable ticket. It arrives without the order reference, the market, or what the reporter expected, so the engineer chases, waits a day, then starts. Every team paid that cost separately, and detail is cheapest to collect at the moment of reporting. The second gap was severity, because reporters set priority by how annoyed they are, which is honest and useless for sequencing.
Layer two: enrichment, so tickets arrive triage ready
Enrichment asks the reporter for what is missing, in the ticket, at routing time. Two tracks share one pipeline. Track one returns which items are absent against the owning team’s checklist. Track two returns which severity scoring inputs are absent. It never proposes a severity, and it reads priority without ever writing it.
Reaching a human was the most interesting investigation here. The reporter field was always populated, which made it look solved, and it was always the integration service account rather than the person. The real human sat in a details table inside the description, in every ticket across a thirty day sample. The fix was a resolver with two branches converging on one contract, so everything downstream stays source agnostic.
I mention the person rather than reassign the reporter field, because duplicate detection keys on reporter identity and rewriting it would have silently broken a working feature. Both tracks cap at five items, because a comment asking for fifteen things gets ignored.
Layer three: the resolution agent
With tickets arriving routed and complete, the remaining cost was a person reading, investigating and answering. Most of that follows a few patterns: the same voucher questions, the same lookups only a few people can run.
The agent reads a short plain text playbook and works three tiers. Tier one is a known answer, so it explains and closes. Tier two needs live data, so it runs read only lookups and either explains the behaviour or establishes a real defect. Tier three is a defect, where it does not attempt a fix but hands the engineer a complete ticket instead of a vague one. Below confidence it steps back.
Two choices carry it. Playbooks are plain text that ops and product can write, so extending the agent does not queue behind engineering. Tools live in one catalogue, declared once with an owner and a read only flag, so a lookup built for one domain is available to every other. The second domain costs a playbook and sometimes one new tool.
Trade offs
Four decisions, each resolving one of the tensions above. For each one: why I chose it, what it cost, and what the reverse choice would have cost instead.
Where I built, and where I needed engineering
I drew the line where the work stopped being configuration and became infrastructure.
Mine, end to end: the historical analysis, the context and prompt design, the routing pipeline, the confidence gate and path logic, the duplicate check, both enrichment tracks, the reporter resolver, and the severity input logic. All in a tool where a change is an edit, not a deploy. That was a first principles constraint. Ownership boundaries move monthly, and any design needing an engineering ticket to reflect a reorg rots inside a quarter.
Engineering, necessarily: production hosting and credentials, the runtimes for layer three, and the read only lookups owned by domain teams. Routing is fast and linear, so it stayed on the workflow tool. The resolution loop may wait hours for a reply, so it needs a runtime that holds state across restarts. That is infrastructure, and pretending otherwise would have produced something fragile I would have handed over anyway.
What I learned
- Context is the product. The model is a commodity. The intelligence lives in a careful description of who owns what and which boundaries are contested. The model is stateless, so every call is a new hire on their first morning.
- Solving the stated problem reveals the real one. Routing answered the question I was asked, then moved the bottleneck one step right to readiness, and one further to resolution. I now expect the second problem and build the first layer to carry it.
- Decide where AI gets authority separately from where it gets used. Reading, gathering, drafting and explaining are safe to delegate. Committing your team to a service level is not.
- Design the second customer in from the start. Shared playbooks, a shared catalogue, plain text where non engineers can reach it. The first domain is a proof. The second is the product.