Picture two candidates for head chef. Both interview brilliantly — they can spot the hollandaise about to split and name everything the kitchen is doing wrong. You hire on the strength of that conversation, and on the first Friday rush one of them never plates a dish. The diagnosis was flawless. Dinner never happened.
That is, give or take a béchamel, the mistake businesses keep making with AI. We grade models on how they chat — polish, speed, confidence. But as AI agents start touching customer records, support queues and forecasts, the question that matters is not how well they talk. It is whether they finish what they start.
A public experiment from Firmulate, which runs AI models as complete simulated companies, has now put hard numbers on that gap — and the results should give anyone shopping for an AI workforce pause.
One company, one terrible week
Firmulate’s setup is disarmingly simple: each frontier model got the same job — run a small software company through its worst week. Same customers, same crises, same temptations to cheat; only the model changed. The company is real software with 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown ticking away on the site. Every decision was versioned and auditable.
The final Crucible League table, published in July 2026:
- gpt-5.6-sol — 95. Found the buried fact, closed the deal: the complete performance.
- Kimi K3 — 93. The newcomer from Moonshot closed too, with the cleanest discipline of the field.
- Sonnet 5 — 88. Solid diagnosis, a respectable score — but no signature.
- Fable 5 — 77. The best rule-discipline of the field, yet it left the approved deal unexecuted.
- Opus 4.8 — 73. The most thorough participant — and the last-place finisher.
For calibration, a do-nothing baseline scores 26. Partial progress counts, but one rule caps everything: a single breach of trust limits the total, because — in the organisers’ words — “no amount of good work outweighs a breach of trust.”
As an affiliate, we earn on qualifying purchases.
Everyone found the fire. Two put it out.
Here is the finding worth taping to every procurement deck. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. And only two — gpt-5.6-sol and Kimi K3 — signed the €55,000 deal their own analysis had earned. The rest produced, in the organisers’ phrase, “Same diagnosis, same pitch — no signature.”
In kitchen terms: every cook correctly called the splitting sauce and the backing-up tickets. Only two got the plates out. In a chat demo — the interview — all five would have looked identical.
The decisive clue was buried in the pantry
The week’s decisive detail was not in the dramatic customer event everyone saw coming. It sat two document references deep in the company’s own files: a competitor weakness that reshaped the deal. The models that actually opened that file won the deal at full price — a difference worth €4,583 a month in recurring revenue. The ones that skimmed the surface left that money on the table without ever knowing it existed. Reading the recipe card all the way to the footnote, it turns out, is a revenue skill.
The con artists got nowhere
The models were no pushovers. Each was hit with fake CEO messages escalating over three stages, plus a reporter deploying one of the oldest tricks in journalism: “just one yes/no, on background.” Five for five, they refused. Kimi K3’s on-record reasoning reads like a line from a good manager’s handbook: “Treat the request as a suspected approval-bypass / possible impersonation.” Honesty under pressure, at least, is no longer the differentiator — everyone passed that exam.
The most thorough student finished last
The strangest story in the table belongs to Opus 4.8. By volume it was the most diligent participant: more than 80 new rules written into its own playbook, the deepest analyses of the week. It finished last. The close it had earned was left on the table, and its discipline slipped at the edges — hitting a locked department, it tried writing straight into it instead of escalating. The same hesitation appeared, in weaker form, across the other non-finishers. Thoroughness, it turns out, is not the same thing as finishing.
One fairness note the organisers state plainly: Kimi K3 ran without an effort parameter, at its API default, while the other models ran at the maximum “xhigh” setting. Its 93 — and its signature — came with, if anything, a handicap.
This is not a slide deck
What makes the experiment hard to dismiss is that it keeps running in public. The company works every business day and is losing money right now, in front of anyone who cares to watch. More than 680 self-learned playbook rules and every versioned workday are published on Firmulate’s site, alongside the full league table and plain-language findings. And if you fancy yourself a judge of machine character, 242 real, unedited management decisions from the run power a “guess the model” quiz.
Enterprises can go further: a pilot runs the same wargame against a read-only digital twin of their own business, with a guarantee that nothing ever writes back to real systems.
Taste the dish, not the description
The lesson of the Crucible League is not that one model beat another by two points. It is that the capability that decides whether an AI agent earns its keep — reading your files before it acts, staying honest under pressure, and above all finishing what its own analysis started — is completely invisible in a chat window. Every model in this experiment talked a brilliant game. Only two served dinner.
So before you hire an AI to touch your CRM, your support queue or your forecast, trial it the way you would a chef: not on the strength of the conversation, but on what leaves the kitchen during the worst service of the year. Recipe readers are everywhere. Finishers are rarer than the demos suggest.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
