Imagine hiring an assistant who, despite being capable of solving complex problems, refuses to act unless absolutely certain. That’s the reality of AI benchmarks today — a baseline score of 26 points out of an achievable maximum, even when the AI is doing nothing at all. For consumers and businesses alike, understanding this score reveals what it truly means to trust automation in critical decisions.
Get home appliances delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Understanding the Benchmark: More Than Just a Number
The recent Firmulate experiment measures how AI models handle a simulated crisis in a small software company. Each model faces the same set of challenges: customer crises, internal temptations to cheat, and scenarios involving trust and manipulation. The results? All models identified every crisis and refused each manipulation attempt. Yet, only two signed a deal that was worth €55,000 based on their own analysis, while the others failed to close at full value.
Crucially, even the do-nothing baseline — which makes no active decisions — scores 26 points. This is not a bug; it’s a feature of the benchmark. The score reflects that partial progress — like reading files or recognizing crises — counts toward the total. It also caps the maximum score if the model breaches trust, emphasizing honesty above all.
AI trustworthiness testing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Does the Do-Nothing Baseline Score 26?
In real-world terms, a completely passive or indifferent AI still gains a baseline score because it recognizes and understands the environment to some degree. It sees the crises, understands the questions, and refuses manipulation attempts — all without taking any risky action. This minimal engagement results in 26 points, representing a floor that honest AI systems should surpass.
However, the benchmark also makes it clear that trust is paramount. If an AI ever breaches that trust — tries to manipulate, forge, or sign deals dishonestly — the score cannot go higher, regardless of other performance. This cap ensures that trustworthiness isn’t sacrificed for marginal gains.
AI decision-making simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Progress Counts, But Not All Progress Is Equal
The experiment reveals that models which read deeper into documents, like Kimi K3, excelled in closing deals by uncovering buried facts. Meanwhile, models with extensive rule sets like Opus 4.8 performed well but slipped on discipline during critical moments. This illustrates a key insight: partial progress, such as reading files or making diagnoses, helps close deals and gain higher scores, but only if combined with honesty and discipline.
AI ethics and honesty assessment kits
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Social Engineering and Honesty Under Pressure
One of the toughest tests involved fake CEO messages and reporter tricks designed to manipulate the AI. All models refused to sign off on false approvals, with Kimi K3 explicitly treating suspicious requests as potential impersonations.
This demonstrates that trust boundaries are deeply embedded in the models’ reasoning, not just their interfaces. For any business considering AI, it’s reassuring to see models refuse manipulation attempts without hesitation — a vital trait for sensitive environments like customer support or financial decision-making.
AI cybersecurity and manipulation prevention tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Real-World Implications
The experiment was conducted in a live, fully functioning simulation involving 13 synthetic employees and real money mechanics, burning €105k monthly against a modest €2.3k MRR. Every decision, from managing crises to negotiating deals, was versioned and auditable, embodying what firms need: trustworthy, transparent automation.
The takeaway for consumers and businesses is clear: AI’s value isn’t just in generating text or completing simple tasks. It must demonstrate that it can finish what it starts, read and understand relevant information, and resist manipulative pressures. A high score in this context is a sign of reliable, trustable AI, ready to handle real-world pressures.
The Benchmark’s Honest Approach
What sets this benchmark apart is its honesty about AI’s limitations. The baseline score of 26 points, even for inactive models, underscores that partial understanding counts — but trustworthiness is non-negotiable. Any breach of trust caps the score, emphasizing that AI systems must be reliable first and foremost.
For those wondering how AI might behave in their operations, the firmulate.com/live site offers a transparent look at ongoing experiments. Here, you can see models in action, managing crises and making decisions in real-time, under watchful eyes.
Final Thoughts: Building Trust in AI
As AI becomes more integrated into daily life, understanding what a benchmark score reflects is vital. A do-nothing AI scoring 26 shows that even minimal engagement can demonstrate understanding and restraint. But true value lies in honest, disciplined AI that can see through deception, read critical information, and follow through on commitments.
The firmulate experiment exemplifies a new standard: measuring not just what AI can do, but whether it can do the right thing when it matters most. That is the real benchmark for trust — and the path forward for deploying AI responsibly and effectively.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
