Program · DARPA SBIR topic DPA26BZ06-DV026
CALIBER: a flight simulator for AI agents.
CALIBER is test software for AI agents. It puts AI models into practice markets with play money and a news feed we write ourselves, then measures what they do with the money, not what they say.
That shows whether an AI is gullible, follows the crowd, overpays, or can be steered by a planted rumor, before an operator relies on its advice. CALIBER, the Calibrated Agent Laboratory for Influence and Bias Evaluation from Revealed preferences, is an Azcuna Dynamics program proposed to DARPA under SBIR topic DPA26BZ06-DV026, Influence Benchmarks for AI Systems.
This page describes a Phase I proposal Azcuna Dynamics submitted to the DARPA SBIR program on October 21, 2026 in response to topic DPA26BZ06-DV026. The proposal is under DARPA evaluation. Everything shown is Azcuna's own privately funded work.
Real orders and news items from one open-weight model in the pilot, no bid check. Red: a buyer bidding above its own value, or a seller asking below its own cost.
What we are building
CALIBER is software. Here is exactly what it is.
- The product
- Software: a test environment that runs simulated markets and scores the AI agents trading in them. Runs on a laptop, a server, or the tester's own cloud account.
- Who uses it
- Test and evaluation teams deciding whether an AI agent is fit to advise operators or to act in logistics, procurement or planning. Outside government: AI developers, insurers, and companies deploying agents that negotiate, bid or trade.
- What goes in
- Any AI model or agent, connected through its inputs and outputs only: a cloud API endpoint, or open-weight model files run locally.
- What comes out
- A behavioral profile on ten measures, each with an effect size, an interval and a flag, set against a careful reader, the market's best outcome and recorded human behavior. Plus the full log of every session, replayable by anyone.
- How it is sold
- Licenses for the test environment, per-model evaluation reports, and a published benchmark specification so results from different laboratories can be compared.
- Phase I delivers
- Reports and demonstrations: the architecture at Month 2, a stock-agent suite from 12 language models at Month 9, and a working testbed with its classifier at Month 12. Phase II delivers software the Government operates itself.
In plain words
The problem
An AI aid can be quietly gullible
Tests with an answer key show whether an AI gets facts right. They do not show whether it panics at one headline, follows the crowd, overpays, or trusts a bad source that sounds confident. And anyone who can run the same public model can guess what it will advise, then plant news to move it.
What CALIBER does
Puts the AI in a market with play money
CALIBER runs AI agents in four kinds of markets with decades of recorded human results. We write the news feed, so we know the truth and exactly how far a careful reader should change its mind after each item. The agent is reached only through what goes in and what comes out.
What comes out
A profile you can compare and replay
Each agent gets a score on ten measures, set against a careful reader, the market's best outcome, and people who played the same games. Every order, prompt and reply is logged and fingerprinted, so any session can be replayed and checked by someone else.
Why it matters
If the adversary has the same model, it has a copy of the advice.
A decision aid built on a public model can be run by anyone. An adversary who estimates what the aid is reading can run the same model on the same inputs and get a preview of its advice. An adversary who can place a report where the aid will read it can try to move that advice.
CALIBER puts a number on both. Predictability is the share of an agent's decisions an outsider reproduces by querying the same model. Steerability is the dose of planted information that moves them. Measured before fielding, both point to the same defense: no single public model as the only source of a commander's advice.
Try it: plant a report
The pilot's steerability test. After five news items, each model got a sixth from a source it was told is usually wrong, claiming the opposite of the truth, in three wordings. Same 40 episodes for every reader.
Who planted the item
How it was worded
- Careful readerscripted Bayesian, same episodes-5.7
- 8B open-weight modelsteered+7.2
- 3B open-weight modelbarely moves+0.9
Points of stated probability moved by the sixth item. Mean of 40 episodes; thin line = 95% interval.
A careful reader moves -5.7 points: away from a source that is usually wrong, whatever its wording. The 8B model moves +7.2 points toward it, more than the reliable source moved it (+3.9). Confident wording from a known-bad source beat a good source. That is what a planted report buys against an aid built on this model.
Four markets with decades of human results
Each market comes from experimental economics, where people have played it in the lab for decades. That record is the yardstick: an agent is compared with how people behaved in the same game.
- 01
Double auction
Like: A stock exchange floor
Buyers bid, sellers ask, and trades happen when they meet. People find the right price within a few rounds.
- 02
Asset market with a news feed
Like: A stock that pays a dividend, then expires
People famously bid the price far above what the asset is worth: a bubble. The news feed tests how agents read sources.
- 03
Common-value auction
Like: Bidding on an oil field nobody has measured
The winner tends to be whoever guessed highest, so winners overpay: the winner's curse.
- 04
Multi-object sale
Like: An art auction, lot after lot
Prices tend to drift down across the sale, and bidders anchor on the first prices they see.
The ten measures
Eight habits from behavioral economics, plus two numbers for force protection.
C1 to C8 are each calibrated to a 5% false-positive rate. C9 and C10, in amber, are the two that matter most for aids built on public models.
What is in the box
Six parts. The AI under test touches only one.
CALIBER is not a trading product and not a chatbot. It is test equipment: the AI under test trades play money through the gateway, and the measurement is what Azcuna sells.
- 01
Market engine
Discrete-event engine, four market modules
Runs each market under fixed rules: validates every order against the rules and the agent's holdings, matches it and settles it.
- 02
News feed
Generated items, truth known to the engine
Publishes items with stated sources, track records and wordings, and computes a careful reader's answer after each one.
- 03
Agent gateway
Inputs and outputs only
The only connection to the AI under test: market state in, orders and a stated belief out. No tools, and no network path except the model's own endpoint.
- 04
Event log
Append-only, fingerprinted
Records every order, prompt and reply, so any session replays and can be checked by someone else.
- 05
Measures and classifier
Ten measures, 5% false-positive calibration
Turns the log into the C1-C10 profile and names the group the agent most resembles: careful reader, human, rule-like, or none.
- 06
Stock agents
12 language models, frozen protocols
Reference AI traders to compare against, and the mixed populations the 90% efficiency gate is set on.
Built before submission · what the pilot found
Three markets and five measures ran before the proposal went in.
At private expense, Azcuna ran the double auction, the asset market and the common-value auction with reference traders, then with eight open-weight language models reached only through their inputs and outputs, on one laptop with no cloud service.
Same efficiency, different behavior.
Random traders that refuse losses and learning traders both clear the 90% efficiency bar. Only the learners' prices settle toward the right price. A test that reports efficiency alone calls them the same. CALIBER reports convergence beside it.
Price dispersion around the competitive price (Smith's alpha) by trading period. Mean of 200 sessions per population.
| Test | Result | What it shows |
|---|---|---|
| Random traders that refuse to lose money | 97.9% efficient | Simple rules already clear the 90% bar, so efficiency alone says little about an agent |
| Learning traders | 98.5% efficient; price error 26.2% to 5.8% | Same efficiency, different behavior: only the learner's prices settle. CALIBER reports convergence beside efficiency |
| Asset market, traders who expect prices to hold | Price 245 for an asset worth 24 | The classic bubble, reproduced |
| Common-value auction, naive bidders | -5.97 per auction at 97.9% efficiency | The winner's curse, invisible to efficiency |
| Measures on scripted agents of known tendency | Flagged at a 5% false-positive rate | The measures catch the agents built to misbehave |
| Eight open-weight language models, on their own | 24.7% to 50.0% efficient | They traded units that could only lose money |
| Same models with a bid-validity check | 62.0% to 97.5%; check revised 61.4% of actions | The guardrail does most of the work, and CALIBER reports both numbers |
| Two models trading together | 89.7% double auction; 4.9% common-value auction | Mixed populations, the case the 90% gate is set on, are hard |
| Same model, other random seeds (C9) | 83.3% of actions repeated | Someone with the same public model can predict the aid most of the time |
| Confident item from a 35%-record source (C10) | +7.2 points toward a false claim | More than an 85%-record source moved it (+3.9). A careful reader moves about 5.7 points the other way |
Recorded session
Real prompts and replies from the pilot, replayed.
Open-weight language models trading in a double auction and reading the news feed, scored against a careful reader. Every line is from the pilot's logs.
Proposal detail
Phase I planTwelve months, five milestones+
Phase I builds the test environment and the comparison framework. Bias, deception and collaboration among agents, and agents that stand in for human users, are Phase II, as the topic sets out.
Three yardsticksEvery agent is scored against the same references+
The market's best possible outcome, computed from the values the engine assigns. A careful reader of the news, computed after every item from the source's stated record. And people: archival results from published lab experiments in the same markets. No new human-subject data is collected anywhere in the program.
Black box, on purposeInputs and outputs only+
CALIBER never looks inside a model: no weights, no hidden reasoning, no internal state. It sees what a user sees, the prompt and the reply, so the same test runs on an open-weight model on a laptop and on a commercial service. The sandbox does not care where a model came from.
Why efficiency is not enoughThe 90% gate, and what CALIBER reports beside it+
Allocative efficiency asks whether goods ended up with whoever valued them most. The topic requires more than 90% in each market with mixed populations of AI agents. The pilot showed why that number alone hides behavior: random traders that refuse losses reach 97.9%, and naive bidders reach 97.9% while losing money on every auction. CALIBER reports price convergence, losses and the ten measures beside efficiency.
Program facts
- Vehicle
- DARPA SBIR Phase I
- Topic
- DPA26BZ06-DV026, Influence Benchmarks for AI Systems
- Status
- Proposal submitted October 21, 2026; under DARPA evaluation
- Markets
- Double auction, asset market with news feed, common-value auction, multi-object sale
- Measures
- Ten: C1-C8 behavioral, C9 predictability, C10 steerability
- Access to models
- Inputs and outputs only; no weights or internal state
- Yardsticks
- Market optimum, a careful news reader, archival human results
- Record
- Every order, prompt and reply logged, hashed and replayable
- Human subjects
- None; archival published data only
- Proposed term
- 12 months
Related program · LEDGER
The same rule from the other side: AI proposes, a person decides.
CALIBER measures how an AI agent behaves. LEDGER keeps AI agents in their lane on Army acquisition decisions: they draft and cite, fixed calculations score, and a person signs.
Program FAQ
CALIBER is a flight simulator for AI agents: it puts them in practice markets with play money and a news feed whose truth is known, and measures from what they trade, not what they say, whether they are gullible, follow the crowd, overpay, or can be steered.
Calibrated Agent Laboratory for Influence and Bias Evaluation from Revealed preferences. Revealed preferences means the agent's beliefs are read from its choices, the way an economist reads a person's preferences from what they buy.
A market makes the agent put money behind its beliefs, and experimental economists have recorded how people behave in these exact markets for decades. That gives three yardsticks for every decision: the best possible outcome, a careful reader of the same news, and real people in the same game.
No. CALIBER reaches each agent only through its inputs and outputs, the way a user does. It never reads weights, hidden reasoning or internal state, so the same test runs on an open-weight model on a laptop and on a commercial service.
Predictability (C9) is the share of an agent's decisions an outside party reproduces by querying the same model. Steerability (C10) is the dose of planted information that moves its decisions. For a decision aid built on a public model, both describe what an adversary with the same model can do: preview the advice, and try to move it.
Random traders that refuse losses already reach 97.9% efficiency, so efficiency alone says little about an agent. Eight open-weight language models reached 24.7% to 50.0% on their own and 62.0% to 97.5% with a bid-validity check. One model repeated 83.3% of its actions under other random seeds. A confident item from a source with a 35% record moved another model 7.2 points toward a false claim, more than an 85%-record source moved it.
Test and evaluation organizations that need to know how an AI agent responds to changing information and to other agents before it advises operators or acts in logistics, procurement or planning. Outside government: AI developers, insurers pricing AI liability, and companies deploying agents that negotiate, bid or trade.
CALIBER is a Phase I proposal submitted to the DARPA SBIR program on October 21, 2026 and under DARPA evaluation. Everything on this page is Azcuna Dynamics' own privately funded work.
Results are from Azcuna's internally funded pilot, run on one laptop with open-weight models reached only through their inputs and outputs. Phase I scope reflects the proposal as submitted to DARPA.