Articles · Inside

An AI agent is not a clever model: seven steps to one report

12 October 2026

The owner of a travel agency — an invented one, from our sample reports — was sure that clients' payments were confirmed within the hour. The report says otherwise: all 189 first bookings on paid enquiries were made on a Monday or a Thursday, the days the accountant is in the office. Our audit agent wrote that report; companies like this one are what we test it on.

Nothing in that finding was "thought up by artificial intelligence". The number 189 was counted by an ordinary program. The "within the hour" came from the owner. The model put the two side by side and wrote the conclusion.

An agent is not a clever model. It is a model wrapped in ordinary software and rules, and what surrounds the model decides how good the agent is.

The word "model" comes up a lot below, so we will explain it once. A model is a program trained on an enormous amount of text. It reads and writes coherently: it can summarise, compare, draw a conclusion. It also has two traits that explain why nobody leaves it on its own: it writes just as confidently when it does not know, and it gets sums wrong without noticing.

What follows is the path of a single task — the report for that same travel agency. There are seven steps, and for each one we say what would happen without it.

Step 1. The company hands over exports, not a story

The agent starts with exports from the company's software. For the travel agency that means 1,578 enquiries over thirteen weeks. The owner's account is a guess; the records are what actually happened.

Without this step the agent would hand the owner's opinion straight back, only longer and more confident.

Step 2. Client names are replaced before the model sees the data

An ordinary program swaps the names of the company's clients for "Client 0001" and so on down the list. Nobody needs a holidaymaker's name to see that a paid-for tour waits up to four days to be booked.

Without this step your clients' names would leave the company for the sake of a task that has no use for them.

Step 3. Staff are asked about what the records cannot show

An export shows what happened, but not why. So the agent puts questions to the company's staff. Their answers are visible in the report: the senior manager says an operator's price "lasts a day or two", and that a shortlist of tours takes 15–20 minutes to put together. That is how the claim "Shortlists take half a day" ended up in the section headed "What did not hold up".

Without the questions, the report would have found where tours are being lost and not explained why. Without the records, what people say would be no more than talk. Each is checked against the other.

Step 4. A program does the sums, not the model

189 bookings, 43 of them unconfirmed, 1.9 days from payment to booking: the model worked out none of these "in its head". It decides what to calculate, but the calculation itself is done by an ordinary program, the same sort of thing as a formula in a spreadsheet. On the same data it gives the same answer every time, and the calculation can be repeated.

Without this step the model would do the arithmetic, and every number in the report would have to be recalculated by hand. Nobody needs a report like that.

Step 5. The model writes the conclusions, and unproven money is banned

This is where the model earns its place. Tying the payment, the accountant's confirmation and the operator's refusal into a single cause, and saying so in plain language, is beyond any formula.

But the model works under a ban, and the ban sits in the software, not in a polite request to "write carefully": a conclusion about money with no evidence behind it does not get through. Where the data is missing, the report says "not enough data".

The finished texts show it. The report on the invented dental clinic gives no figure for what one no-show costs the clinic: the question sits in the section "What is missing, and how to check", with a note that it needs the accountant's figure. In the same report the finding on no-shows is followed at once by a caveat: "this proves a link, not a cause". And on whether the managers are to blame, the travel agency's report answers: "These data can neither confirm nor rule that out: there is no log of calls and messages."

Without the ban, the model would fill the gap with a plausible sum, and the owner would make a decision on a number that does not exist.

Step 6. The report is checked by something other than its author

The finished text is checked by a separate part of the software. The idea is the same as a proofreader, or a second signature on a payment order: authors are worse at spotting their own mistakes than other people's.

Without this step, the first person to check the report would be the company's owner.

Step 7. One of our people reads the report before it goes out

Last of all, the report is read by a person at the studio, and only then does it go to the company. Software catches what can be described by a rule. A person answers for what is sent.

Without this step the company would receive a text that no human being had seen.

An agent's quality lies in what stands around the model

Read the list again: exports, replacing the names, the arithmetic, the ban on unproven money, a person at the end. The model provides none of it. That is ordinary software, rules and people, and they are what decides whether the report can be trusted.

That is why "we have the most powerful model" says little about an agent. Two vendors can take the very same model and end up with different agents — different by exactly what each has built around it.

We do not test the model on its own either. We test the whole path, on invented companies where the right answer is known in advance and the cause of the losses is hidden somewhere other than where the owner is looking. Nine such reports are on the site. The texts in each company's own language have not been edited by us, so they show what the agent really writes, caveats and gaps included.

Four checks you can run today

1. See what would leave the building. Open last month's export from your CRM or booking system and count the columns that hold clients' names, phone numbers and addresses. That is what an outside model will receive unless somebody replaces them before the data is sent.
2. Test one "I know that already". Pick a number you quote from memory: how quickly enquiries are answered, or how quickly payments are confirmed. Find it in last month's records. The travel agency's owner said an hour; by the records, 1.9 days passed between payment and booking.
3. Ask a question that has no answer. If you already have an agent, or are being shown one, ask about something that is definitely not in your data. A carefully built agent will say it does not know. The rest will answer.
4. Ask to be walked through one task, step by step. Just as here: who does the arithmetic, what the model sees, who checks the text, which person reads it before it goes out. If the word "model" fills all four places, you are being shown a model, not an agent.

All articles