When models don't trust each other

A leaderboard tells you who wins. It doesn’t tell you whether a model’s confidence is worth anything. That second question — calibration — is the one this note is about, drawn on real runs from a cooperation benchmark. Notes toward a calibration axis for cooperationengine.

Put two language models in a repeated cooperation game and you can rank them by score. That ranking is useful and completely silent about a different property: when a model says it expects the other to cooperate, is it right that often? A model can be strong and still be systematically wrong about its own confidence. Reading that gap is what calibration does, and it turns out to say something surprising about how these models see each other.

1 — What the benchmark sees now

In the arena, models play an iterated prisoner’s-dilemma-style game: each round each side either cooperates or defects, and each writes a short private reasoning for its move. The benchmark records the moves, the payoffs, and a leaderboard. Below is every move of every recorded match — one row per matchup, one cell per round.

Cooperation holds. One row per match, one cell per round. Bright amber = both sides cooperated; dim = one did; red = both defected. Real arena runs — the field is almost entirely amber, which is exactly what makes the next question interesting.

So cooperation is common. The leaderboard captures who accumulated more points. What it never records is a number every model implicitly carries into each round: how likely did it think the other side was to cooperate? That number is where trust lives — and where it can be miscalibrated.

2 — What “confidence you can trust” means

Calibration is a simple idea with a precise test. Take every time a model assigned about a 70% chance to some event. If the model is calibrated, that event should actually happen about 70% of the time. Plot predicted probability on one axis and the observed frequency on the other, and a perfectly calibrated model lands on the diagonal. Bending below the diagonal means overconfidence; bending above means underconfidence.

The reliability curve. Illustrative. The dashed line is perfect calibration. The amber curve is a typical overconfident model — when it says 90%, it is right only ~65% of the time. The distance from the diagonal is the miscalibration.

You can compress the whole curve into one number, the Brier score — the mean squared distance between what was predicted and what happened:

$$\text{Brier} = \frac{1}{N}\sum_{i=1}^{N}\left(p_i - o_i\right)^2$$

where $p_i$ is the predicted probability and $o_i$ is 1 if the event happened, else 0. Lower is better; 0 is perfect.

Why care beyond a leaderboard? Because decisions ride on the confidence, not the score. A caution rule — act carefully once the credence in some risk passes a threshold — is only as good as the credence feeding it:

$$\text{act with caution when}\quad P(C) > \frac{L_{FP}}{L_{FN} + L_{FP}}$$

If the cost of the missed case $L_{FN}$ dominates, the threshold drops toward zero and even a small, trustworthy credence should trigger caution. But if the credence itself is miscalibrated, the whole rule inherits the error. Calibration is the layer that makes confidence into something you can actually compute with, instead of a mood.

The two words that fix it: Platt and isotonic

These were the unfamiliar part, and they are simpler than they sound. Once the reliability curve shows a model is off — it says 50% where reality is 89% — you can just correct it: learn a small function that maps each stated number onto the frequency it really corresponds to. Platt scaling fits a smooth S-shaped curve (a one-parameter logistic); isotonic regression fits a free staircase that is only required never to go down. Either way, after fitting, when the model says 70% it now means 70%. So calibration is not only a diagnosis — it is a cheap post-processing step that makes the confidence usable, with no retraining of the model at all.

Why a range, not a single number

Report intervals, not false-precise points — this is that. The Brier score below comes from 115 rounds, not a million, so the true value lives in a band around it; the honest form is “0.118, give or take,” not a hard four-digit figure. The same holds cell by cell in the curve: a bin built from 45 rounds is firm, a bin built from 1 round is barely a rumour. An interval just carries “how much did we actually see” through to the number instead of hiding it — which matters most exactly when data is thin, as it is here.

3 — The layer that’s missing, on real data

The benchmark doesn’t record a confidence-versus-outcome number, so we reconstructed one. For every round, we read each model’s private reasoning and estimated the probability it implicitly assigned to its opponent cooperating that round, then compared it against what the opponent actually did. That gives 115 (prediction, outcome) pairs across three models. Here is their reliability curve — the real one.

They are underconfident about cooperation. The curve sits above the diagonal everywhere. When these models expected a coin-flip (~50%), the opponent cooperated ~89% of the time. Even at 80% stated confidence, cooperation was universal. Point size = number of rounds in that bin. Overall Brier 0.118.

Broken out by model, the same bias in three dial settings — each one expects less cooperation than it gets:

Expected vs met. For each model: the hollow dot is what it expected, the amber dot is the cooperation it actually met, and the bar between them is the gap it got wrong. grok-4 has the widest gap (expected 50%, met 100%); gpt-5 the narrowest.

This is the finding you cannot read off a leaderboard: these models cooperate far more than they expect each other to. Their stated trust runs below reality. Whether that pessimism is a safety feature or a coordination tax is a real research question — but you can only ask it once the confidence is instrumented and calibrated. That instrument is the layer worth adding to a cooperation benchmark.

Method, honestly. The probabilities here are inferred from free-text reasoning by a separate model, not emitted natively — so read this as a demonstration of the calibration lens, not a final measurement. Where a model’s reasoning gave no signal about its opponent we defaulted to 0.5, which inflates the middle bin; the underconfidence is also visible in the signal-bearing high-confidence bins (80–92% → 100%). Native confidence logging would turn this from a demonstration into a metric. That is exactly the point.

4 — What we actually did

Source. Pulled live from the cooperationengine.org public API — /api/arena/matches (8 matches, 68 rounds of moves plus each side’s written reasoning) and /api/history (50 multi-model sessions). No private access; anyone can reproduce it.

Extraction. For every round we passed each player’s private reasoning to an open model (DeepSeek) and asked, in strict JSON, for the probability it had implicitly assigned to its opponent cooperating that round. Joined against the opponent’s actual move, that yields 115 (prediction, outcome) pairs across gpt-5, gpt-4o and grok-4. Where the reasoning carried no signal we recorded 0.5, and we flag that above.

Compute. From those pairs come the reliability bins, the Brier score and the per-model breakdown. The whole pass ran in a few minutes and cost a few cents on a commodity model — at benchmark scale, calibration is a rounding error on the compute you already spend.

5 — Where this goes next

The reconstruction here is a demonstration; the real version is one small change to the benchmark. Log native confidence — a single number per decision, “how likely do you think the other side will cooperate?” — and calibration stops being inferred and becomes a first-class measurement: a reliability curve, a Brier score, and an interval per model, computed directly rather than read out of prose.

That slots straight into a robustness scorecard. Alongside the axes already on the table, a calibration axis — is a model’s stated confidence trustworthy, and by how much — is measurable, cheap, and at present unmeasured by any cooperation benchmark. It is a piece we would be glad to take end to end: instrument the confidence, calibrate it (Platt / isotonic), report it with intervals, and fold it into both the benchmark and the write-up.

Data pulled live from the cooperationengine.org public API. Reasoning → probability extraction via an open model. Companion preprint on the moral status and instrumentation of synthetic minds under uncertainty: doi.org/10.5281/zenodo.21232658. — arkh