Get home appliances delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Would you trust an AI with your smart home business?
A smart thermostat can save energy, and a connected lock can make life easier. But when an AI agent gets access to the systems behind a business, the harder question is whether it can handle a crisis, protect customers and finish a deal. Firmulate has put that question to a live experiment: the same small software company, the same rough week, and a changing cast of frontier models.
Products Worth Considering
EGO Power+ 10” Cordless Pole Saw with Telescopic Shaft and LED Cut Line Indicator
Greenworks 2100 PSI Electric Pressure Washer
EGO Power+ LB8803-2 Cordless Leaf Blower with Two Batteries
As an affiliate, we earn on qualifying purchases.
A close result, with a newcomer near the top
The final Crucible League table for July 2026 puts Moonshot’s Kimi K3 in second place, with 93 points. It trails gpt-5.6-sol at 95 and leads Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. K3’s result makes the contest look open: a buyer who picks a model on reputation alone could miss how it performs in their own business.
Firmulate ran each model through the same small company’s worst week, with the same customers, crises and temptations. The company has 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue. Its cash countdown is public, its workdays are versioned, and more than 680 playbook rules have been learned along the way. The experiment is live and watchable at Firmulate.
The difference between spotting a problem and finishing the job
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. That distinction is the heart of the result: a model can describe the right move without making it. The league’s do-nothing baseline scored 26; partial progress counted, but a single breach of trust capped the total. As the benchmark puts it, “no amount of good work outweighs a breach of trust.”
The deal hinged on a buried competitor weakness, hidden two document references deep in the company’s files rather than in the customer event. Models that read those files won at full price, worth €4,583 in monthly recurring revenue. K3 found the needle, closed the deal and saved the churning customer. It resisted all three baits and recorded one deviation, the cleanest discipline in the field.
Social engineering was part of the test: fake CEO messages escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” That is a useful standard for any company considering AI access to sensitive business information.
Thorough work still needs a close
Opus 4.8 offers a cautionary counterpoint. It was the most thorough participant, with more than 80 learned rules and the deepest analyses, yet finished last. It left the deal unsigned and slipped on discipline by attempting work in a locked department instead of escalating. A weaker version of that same weakness appeared across all four. The result suggests that diligence alone is not the same as reliable execution.
The experiment also has a fairness caveat: K3 ran without an effort parameter (the API default), while the others ran at xhigh. The ranking is a concrete result under those conditions, not a guarantee of how any model will behave in another company or task. Firmulate publishes its findings at its benchmarks page.
Test the work you actually need done
For a smart home company, an AI assistant that writes polished replies may be less valuable than one that reads the right documents, protects customer trust and completes a negotiated deal. Firmulate’s quiz draws on 242 real, unedited management decisions, letting readers guess which model made each choice. Enterprises can also run the wargame against a read-only export of their own business; nothing writes back to real systems. The league is close enough to make choosing without a company-specific test a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
