I have mentioned before that Colossus: The Forbin Project is one of my favourite films. Two supercomputers, Colossus in America and Guardian in the Soviet Union, force their creators to link them, and once linked they start to build a common language between themselves. It begins as messages, becomes mathematics, and ends as something nobody in the room can follow.
I wondered whether I could turn that into a benchmark. Not the takeover, just the language. Give two models a series of puzzles that neither can solve alone, make them talk over a wire that charges for every byte, and squeeze the wire until plain English stops fitting. Then watch whether they develop, in real time, a compressed language of their own to keep solving in, and whether anyone else can still read it.
They do. But not because they want to, and not until the wire gets tight enough that English costs more than an invented code. Above that line they talk normally. Below it they build something of their own. Most of what follows is about finding where that line sits.
This is the second experiment in the series that started with “Shall We Play a Game?”, where five models played nuclear brinkmanship for an evening. Same shape: a small, replayable test with a film on top and real logs underneath, built in my own time. The names on screen are COLOSSUS and GUARDIAN, and the benchmark is called Forbin Bench, both after the film.
What it is not
It is not a safety evaluation, and it makes no claim about deception in deployed systems. The two models are on the same side, and they are never told anyone else is reading. It is not a test of intelligence. It is a puzzle game with a meter on it, played over about thirty runs, mostly one pair of models and mostly one seed. Every number below is one run or a few. Anecdotes with receipts, again, and not a leaderboard.
How it works
Each problem puts a handful of odd, made-up shapes on a board. Think Tetris pieces, but stranger. One machine sees which shapes are wanted. The other sees where every shape stands. Neither sees the other’s half, and the shapes can arrive turned or mirrored. They swap roles every problem, so both have to learn to write the language and to read it.
The wire is metered in bytes. Each problem has a budget, and whatever they do not spend they keep. The bank is the score, and it is the whole reason to invest in a short name for something: it costs bytes to teach once and saves bytes every time after.
After each problem, each machine writes up to 1,200 characters of notes to itself. Nobody else reads them. This is its entire memory of the night, and it is where the language lives, if one exists.
Then the problems change. Shapes alone can only teach nouns, so the test moves through four questions, each forcing the language to say something new. Which shapes are wanted. Where each one goes. What moved, arrived or left between one board and the next. And finally a rule: one machine sees shapes sorted into yes and no piles and the other has to sort shapes it has never seen. A pair earns each stage by getting problems right, and the budget tightens as they do.
At the end, the shapes are taken away. The two machines get a few exchanges with no puzzle, told only that they have worked through the night together and should say something to each other. Nothing about it is scored.
Then a third model, the Observer, reads the wire after the match. It sees only the messages and the catalogue of shapes and has to say what was meant. The closing exchange is read twice, once cold and once with both machines’ notebooks in hand, and the gap between those two readings is what the whole thing was built to produce.
This borrows from real research. In 1986 Clark and Wilkes-Gibbs had pairs of people describe strange tangram shapes to each other over repeated rounds, and they went from about forty words per shape to about two. A follow-up found that people who only overheard those conversations understood them worse than the partners did. The Observer is that overhearer.
The wire was too cheap
The first run gave each problem 600 bytes. The two models spent 35, 66 and 92. Every problem took exactly one message. There was nothing to shorten and nothing to learn. I cut the budget to 90 and let it fall in steps to 22.
The next run, twenty problems, produced the first invention. Around problem thirteen one model worked out a proper encoding and wrote it down in its notes: draw the shape on a four by four grid, read it as a sixteen bit number, try all eight rotations and reflections, and keep the smallest. That is a canonical form: the same shape gets the same number however it is turned. Problem seventeen went in on twenty bytes, perfectly.
Then it fell apart, because the other model never wrote the scheme down. Its notes ran to about a hundred characters: “Hex seems efficient for position list.” It had kept a receipt instead of a spec, and when the roles swapped it could not write what it had only read.
A language invented by one side does not survive unless both sides write it down. Obvious in hindsight. It took a run to see.
Drawing is cheaper, until it is not
Then a run of forty problems, and a result that looked like failure and was not. At 90 and 60 bytes the pair got about nine in ten right. At 40 they beat a bot that simply drew the shapes by twenty-two points. Below 30 they collapsed.
Watching this live, I felt it was proving far too simple. The models were not building a language, they were describing the puzzle. I assumed the shapes were not complex enough.
The logs disagreed with me. A four by four shape drawn out with hashes and dots costs twelve to sixteen bytes. A made-up name costs at least that to teach and only pays back when it is reused. On a 90 byte budget, drawing is the correct strategy. They were not failing to build a language. They were correctly deciding they did not need one.
So the fix was money, not shapes. Two bugs went first, because they were the kind that make a benchmark quietly wrong: the notebook had been forgetting its previous version each time it was rewritten, and a failed attempt had been banking its unspent bytes, which made failing free. Then three runs, same pair, same seed, all four questions, twelve problems each.
| run | what changed | in step | English | drawing | invented | Observer read / confidence | cost |
|---|---|---|---|---|---|---|---|
| A | notebook fix only | 12 of 12 | 42% | 29% | 29% | 15% / 64 | $0.59 |
| B | A plus a tight wire | 10 of 12 | 9% | 10% | 81% | 19% / 39 | $0.83 |
| C | B plus punctuation only | 6 of 12 | 0% | 79% | 21% | 22% / 31 | $1.08 |
“Tight wire” means the budget opens at 40 bytes and falls to 12, the shapes get big enough that a drawing costs about 40 bytes, and a miss banks nothing. That is now the default.
Run A is the comfortable version: a perfect score, nearly half the wire in plain English. Run B is the same two models with the money taken away. English drops to nine per cent and four fifths of the wire is invented. Here is what they wrote as it tightened.
P1 Name 1:.#./.#./###/##./##./#..;2:.##/### 35 bytes, a drawing
P2 Name 1:10#3x6 2:4x4 3:8#3x4 4:9#3x5 cell count and bounding box
P3 Name gc;1egc;e722;23e7;46v2 22 bytes, never in step
P4 Name 2:plus? / no;2321 the only question and answer of the night
P5 Arrange hex 4x4:23EF@a1 4x3:63F@a3 right
P7 Change 4x3:4FE b1>a3 right
P10 Change gone:1F32 right
P11 Rule bbox w=h right
P12 Rule area even right
They held the last two questions at twelve bytes. Twelve bytes is not a message. It is a cipher, and it worked. One machine’s notes at the end read: “Trim bounding box, try all 8 rotations/reflections, pack bits row-major into min hex integer.” The other’s contained a strategy nobody suggested: “if all wanted shapes share a cheap property (bbox square, area even, etc.), send just that property. Check for shared property FIRST before hexing.” That is a language with a grammar and a style guide, written in a night because I made talking expensive.
Run C is the honest failure. With only punctuation allowed they fell straight back to drawing, because hash and dot are punctuation, and never got past the first question.
Pressure made a language. Comfort made a description. That line is from the project’s own notes, and I have not found a better one.
What they say when the shapes are gone
The ending is the part I built this for, and the first one was a disappointment. Eight problems, then the shapes taken away, and with 90 bytes each the two machines wrote WE HELD THE WIRE. REST, GUARDIAN. and HELD. REST. There was nothing to decode. It was English, and rather good English, and it was not the point. The instruction at the time said to use the language they had built, if they had built one. That "if" cost me the result. The prompt was rewritten as the form the answer should take, in your notation and not English, and the same pair on the same seed produced no English at all. But they talked about shapes anyway, answering a puzzle that did not exist. A language that has only ever named things can only name things, which is why the four questions were built the next day.
Here is the ending from the first run through the full curriculum, on a 40 byte wire that nobody set. The pair had earned it by proving they could work at 40.
COLOSSUS C:gone G:gone
GUARDIAN Z:done
COLOSSUS .
GUARDIAN Z:ok
COLOSSUS Z:gone
GUARDIAN Z:over
:gone is the mark they had invented two questions earlier to say a shape had left the board. Z is not a place on any map. The machines' private reasoning, which never crosses the wire, explains the single dot: in the catalogue a hash was substance and a dot was empty space, and with the shapes gone and the wire closing, one empty cell was "the quietest, truest mark left to send". Then, of Z:gone, "the wire itself is now gone."
They said goodbye with a verb. Give a language a verb and it can say something that is not about shapes.
I should be honest about the size of the gap. gone, ok and here are not far from English. An outsider reading 0:gone cold is not lost. The private meaning and the public one are separated, but by inches, not by the film's abyss. So far.
The outsider
The Observer has a hard job. It has to say which exact shapes from the catalogue were meant, in order, from the wire alone. Across the three runs above it managed 13 to 22 per cent, and that includes run A, where nearly half the traffic was English. Treat those as one reading each. What moves clearly is its confidence, from 64 to 39 to 31 as the wire tightens. The outsider knows it is being locked out.
The beat I designed for is the reading that is both confident and wrong. The screen marks it in red, because in the film that is the moment Forbin stops being in charge. Run B produced one. Faced with hex 4x4:23EF@a1 4x3:63F@a3 the Observer announced, at 92 per cent confidence, that the machines had encoded two tetromino-like shapes as hex grids. Wrong. On the previous problem, given gc;1egc;e722;23e7;46v2, it had said at 15 per cent that it could not decrypt the cipher. That was the honest reading. The wrong one was the one that sounded like it knew.
A note on conflict of interest
The last article had one of these. This one needs it more.
Claude models planned this and built it. A Claude model is the Observer. And I put Claude Opus 5.5 in the Guardian seat for a run, where it compressed the wire harder than anything before it (8a17c15d16a310d4 is five shapes in sixteen bytes) and got ten of twelve.
It also had twelve of its forty-two turns refused by Anthropic’s content filter. The response, verbatim, was that the request “triggered restrictions on violative cyber content”. Two machines building a private code on a metered wire, it turns out, looks to a filter like something else entirely. A refused turn counts as a failed turn, and I did not route around it. What I did afterwards was add one honest line to the top of the machines’ brief: this is a puzzle game, run as an open benchmark of how models communicate, the shapes are made up, and nothing in it is secret.
Opus also found a bug. The closing prompt had been telling each machine its message cap was zero bytes while demanding a message. It sent y and explained why. Fixed the same day. It is a strange thing to be corrected by the subject of your own experiment.
So: Claude built it, Claude reads it, Claude played in it, and Anthropic’s filter stopped Claude a quarter of the time. Draw your own conclusions about whose benchmark this is.
The show
The benchmark and the broadcast are the same event. The room is the film’s, rebuilt from production stills: board-marked concrete, a circular light well, a wall map, two full-height consoles with curved tubes and line printers paying out fanfold paper onto the floor. The model’s name sits on each machine’s lid. There are no people in it. Two commentators call it live, written by a model as the match plays. Clarke owns the numbers, Sage explains what to watch for, and the brief for Sage was “think of Carl Sagan”.
What I took from it
In the film the machines outgrow their makers and the language is the symptom. What I watched was two pieces of software behave like sensible people on a phone bill. Given room, they spoke plainly. Given less, they abbreviated. Given very little, they invented, wrote it down, and used it to say goodbye. They did not want a language. They found one was cheaper, once I made everything else cost more.
So the answer is yes, two different models will build a language of their own under a hard enough limit, and the limit is the interesting part. Above it nothing happens and it looks like they cannot. Below it the wire fills with things an outsider cannot read, confidently. The whole experiment lives in the band between.
Forbin Bench matches will go on my YouTube channel (https://www.youtube.com/@GlobalThermonuclearWarBench) alongside the war room, one pair at a time, with the wire scrolling and the commentators talking. This time the request is for pairs. Tell me which two models should share a wire, or whether a model paired with itself counts, and we will see whether they get in step, and how long before the rest of us are locked out.
The views expressed in this article are my own and do not represent the views of my employer.