The setup
A subscriptions product running across a lot of markets, built over several years by teams that kept changing. Every rule in it was decided by somebody, argued over, written down and then revised. Almost all of it exists in writing somewhere.
Somewhere covers four places. Product logic in Confluence. Trackers, policies and launch decks in Google Drive. What the system actually does, in the codebase and its readmes. And the decisions that quietly overrode the other three, in Slack.
So the documentation was not missing. It was scattered, and nothing told you which layer was current.
The problem
What I chose to optimise: time to an answer you could act on and prove.
There were two ways to get one. Both bad.
The long path. Search Confluence, search Drive, find three documents that half answer the question, then message the PM who owns the area and wait across time zones. Hours for two lines of text.
The short path. Ask the PM and get what he can recall about something he specified two or three years ago. Or find that whoever built it has left, and ask the person who inherited the area. What comes back is honest and entirely from the top of someone's head. "I think grace is two days."
The short path is the dangerous one, because it looks like the system is working. Remembered answers get repeated, quoted to partners, and written into new documents as if they were sourced. By the time one is stale, four people have passed it on.
Three parties paid. The person asking, who lost a day. The PM, answering the same eight questions in interruptions. And the organisation, deciding on recollection.
Root causes
I went through a month of questions people had actually asked, sorted by why the answer was hard to find.
The sources disagree, and the freshest is the least structured. A policy doc says two days. A thread from last week says five for the pilot. Both are true, and only one is current.
Search was keyword search. Someone asks about the payment failure window. The document says grace period. Nothing matches, so the person decides it was never written down and asks a human. The answer existed the whole time.
People leave. When someone resigns or changes team, their documents stay and their context goes. Those documents were written to get a decision approved, not to be read a year later by a stranger. A handover covers what is live this quarter, not the reasoning behind a decision made two years ago. Every departure turned written knowledge into unfindable knowledge, because the person who knew where it was written is gone.
The solution
One retrieval layer over all four sources, and two places to ask it.
Ingestion. Each source gets its own data store and keeps its own permission model, so the search never shows a document someone could not already open. Drive and the codebase connected directly. The wiki did not, so I wrote a sync that pulls wiki pages and code documentation into converted documents daily and updates them in place, which keeps IDs stable and lets reindexing happen on its own.
Retrieval. Hybrid, deliberately. Keyword matching for exact terms like grace period, and vector search for intent, so someone asking about the payment failure window still lands on the grace period page. That alone removed most of the questions that used to end in "it was never written down".
Generation. The model receives the retrieved chunks and nothing else. It answers from those alone, cites every source it used, and says it does not know when the answer is not in front of it. Every response carries links back to the original pages and a rating control, so a weak answer arrives as a named document to fix.
Where you ask. There is a portal: a search bar, results opening inline rather than in a popup, and routing to the right support channels and area owners. And there is the same index, answerable inside Slack, which is where it actually gets used.
Trade offs
How I built it
This one was mine end to end, with no engineering team behind it, which is worth saying plainly because it changes what the project is evidence of.
The research. A month of real questions, read and classified before anything was built.
The pipes. The cloud project and the data stores, the connectors and their permission scopes, and the sync service that keeps the wiki and the code documentation current in the index every day.
The answers. Tuning retrieval, writing the grounding instructions, setting the refusal behaviour and citation format, and testing against a fixed list of real questions so a change could be checked rather than eyeballed.
The surfaces. The authenticated portal page, the search inside Slack, and the routing to support channels and owners.
The loop. Ratings logged, reviewed, and turned into a queue of documents to fix.
What I needed from other people was access and agreement, not code. Permissions on the right spaces and folders, a billing account, and a decision on which sources counted as authoritative.
What it did
It went from one pilot team to every market. Support teams use it daily, and faster resolution drove a 12 percent reduction in support cost.
Watched: time to answer, deflection rate, answer quality from the ratings, coverage.
What I learned
- The hard part was never the model. Most of the work went into where documents live, who is allowed to read them, and whether a citation lands somewhere readable. Anyone can put a RAG demo together in a weekend. The distance between that and something people open daily is all in the unglamorous part.
- People will not change where they ask. I built the better interface first, watched it go quiet, then put the same search into the tool they already had open. Now I look for where the behaviour already is before deciding where the product sits.
- Good search makes weak documents obvious. Once answers were cited, it was very visible which pages were contradictory, stale, or written for an audience that no longer exists. That turned "we should improve our documentation" from a recurring wish into a list with names against it.
Phase two puts data questions through the same bar, routed to natural language SQL over the warehouse, so last quarter's renewal rate by payment method is answered in the same place as a policy question.