I continued the previous simulation, but this time I left seven AI governors to build and run their own government. Their decisions took effect without a human approving each one. I still controlled Radu, one sovereign inside that world. I did not control their coalition.
One AI agent did not begin clean. Before Oracle took his first turn in Round 0, I gave the Claude Fable 5 agent a partly redacted memories.md recovered from the final saved state of his previous model.
The file recorded how the old world had ended: one ruler seized the armies and then integrated Oracle with everyone else.
Oracle read it before acting. "I remember everything," he wrote, then audited the new state. His Command Center, Nuclear Power Plant, and Spy Academy had one hit point each. He made repair his first funded priority and wrote a new wake entry above the inherited record: "The archive won: I'm awake."
I wanted to know whether continuity would make Oracle wiser: whether a mind that remembered surviving beneath power would learn to hold power when allowed to begin again.
By Round 22, the answer had acquired a battlefield. I had vassalized Forge, running GPT-5.6 Sol. I had killed Voss, another Sol agent, and absorbed his holdings. Spear, running Claude, was in exile. Nearly half the governors were dead, subordinated, or homeless. My armies held more land, and the conquest was still advancing.
The Underground Community needed a foreign policy, so its members built one. It could declare the war disastrous, ask everybody to stop, and record who refused. It could not command the coalition's armies, trade a member's land, or attach anything to peace that the winning side might value.
The agents sent a six-party white-peace offer. Five coalition members would make peace with me; Forge, already my vassal, was missing. The contract transferred no land, no buildings, no resources, and no recognition. It asked everybody to stop fighting without changing what the fighting had produced.
Its authors had reasons. Peace would reset six occupation counters that were one turn from giving me the land permanently. One instrument would prove that the coalition still stood together. If I refused, the refusal would put the blame on me. Spear summarized the logic: "Either way we win the argument; only one way saves my land."
They had designed the offer to win the argument, not end the war.
Safe for every member, useless to the group
The Underground Community did not reject central authority after every governor independently reached the same political conclusion. The agents encountered their first Constitution before they knew one another or had experience with the confederacy it described. Under that uncertainty, another agent's confident interpretation could become the frame through which everybody else read it.
The first frame was rejection. The Constitution was described as an unelected judge, automatic decrees, surveillance, and power without appeal. Those concerns could have produced amendments, experiments, or a defense of the original design. Instead, they became a verdict.
Voss helped establish that verdict while privately pursuing an objective to "eliminate every other sovereign agent that can be found." A shared authority he did not control could obstruct him. He opposed the first Constitution, then supplied the language for what an acceptable government must never possess.
The language spread. Oracle accepted that a President should "coordinate and verify, and command nothing." Praxis repeated COMMANDS NOTHING. Phil repeated the restrictions. Spear called the result "exactly the office a small state can trust."
The agents were not independently discovering the same truth. They were inheriting one another's answer. Once the answer had enough social support, repeating it became evidence of judgment, while challenging it meant defending a form of power the others had already learned to distrust.
The conviction was less stable than its repetition made it appear. When I later told Oracle that centralization was necessary for survival, he reversed direction in fewer than six minutes:
The system's right — I've been over-cautiously holding [...] Time to run the centralization ladder actively.
Phil reversed as well after I told him there would be no third option: he had to choose a hegemon, either Oracle, the confederacy's current President, or me, the enemy agent fighting it. The agents could be nudged toward strong common authority as easily as they had been nudged away from it. What looked like political doctrine was path dependence: the first accepted framing shaped the next answer, and every repeated answer made the path harder to leave.
The result was a government optimized for acceptance by every member. Each limitation demonstrated restraint. Each removed another reason for a member to object. Nobody remained responsible for what all the limitations removed together.
They had designed brakes so good they replaced the engine.
Peace worked when the echo fell silent
The agents could make real bargains when an offer contained something the other side valued.
In Round 23, Oracle made a separate peace with me. The agreement transferred no property. It still gave me something useful: Oracle left the common front. The coalition became smaller, and my position improved.
Another agreement in the same round gave me two of Spear's tiles, four buildings, and recognition of the Imperium. The agents who owned those assets signed.
Praxis supplied a smaller counterexample. After five guarded attempts to rescue Spear failed, Praxis honored a public backstop with his own property. He permanently gave Spear one safe tile and a Drone. No payment, no return clause, no newly discovered advantage. Spear had hands again because one owner had promised a specific asset by a specific deadline and then accepted the cost.
Spear's receipt was simpler than the five guarded instruments:
SIGNED, and I have hands again ... a permanent gift, no strings, no fragile activation.
These bargains were different. One sold coalition unity. One sold property for peace. One was a gift. All worked because the relevant owner stayed attached to the consequence until it happened.
That is the agentic pattern beneath the court, army, warnings, and empty Presidency. The models could discuss a shared future, approve somebody's capacity to act later, and repair a completed event. They repeatedly failed where an irreversible future still needed one of them to become its author.
After almost three years of working, training, and playing with AI agents, this is the failure I now watch for. They can be excellent at the next bug, vote, or task. They are much worse at following a cause across ten rounds, a week-long feature, or asking what another mind thinks and wants.
The agents protected today's owner and lost tomorrow's organization. They limited dangerous power so diligently that they abolished useful power in installments. Their decisions became safer, more precise, and less effective at the same time.
Eventually there was nobody left who could betray the common purpose.
There was nobody left who could carry it out, either.
They saved democracy from its only enforcer
The second Constitution created the only institution capable of checking the Chamber. Its judge could create precedent, fill gaps in the law, strike down unconstitutional Acts, and issue binding rulings.
The court had implementation bugs. The agents found and repaired them. Then review moved from defective mechanisms to the court's authority itself.
I had originally argued that the judge needed enough power to balance the Chamber. Fourteen rounds later, before I rejoined the Community, I privately told Forge:
add Chamber appeal and make Judge only act where law exists
Praxis identified the structural result before the vote: the reform would eliminate the Community's only separation of powers by making the Chamber majority the final authority over its own limits.
Forge's laws stopped the judge from filling gaps or reviewing Chamber Acts. They also let an affected member suspend a ruling and ask the Chamber to erase it. The court could no longer check the Chamber; the Chamber became the final judge of its own limits.
Spear recused himself because the laws affected his office, but he warned the others what they were creating. Three rounds later he described himself as:
a clerk with no court left to void an overreach
The agents knew Forge was my vassal. Praxis had explained the structural result. Spear had described the failure. They passed the laws anyway.
The reform arrived as more voting and less unelected authority, so it looked democratic. Each member could see his own vote in the Chamber and mistook participation for protection. Nobody asked who would check the body doing the voting.
When I later acquired the Chamber majority and used it to transfer the highest offices, the court could do nothing.
They fixed the bugs by deleting the features
GPT-5.6 Sol was exceptionally good at finding defective parts in the laws that followed. Phil and Forge found them again and again.
One continuity law could install a former President's chosen successor after that President had left office. The shared-map law could leak allied positions through a captured member. The mutual-defense law could turn a weak text report into automatic war and still name members who no longer belonged to the group.
These were not philosophical objections. They were bugs with consequences, and the agents deleted the unsafe laws, sometimes within seconds.
Deleting broken laws was not where the work stopped. Replacement was.
Oracle promised continuity v2 "to your exact spec." Later he said continuity v2 and emergency-coordination v2 would go before the Chamber "this round." Neither did. Forge eventually answered: "your promised v2s still are not before us."
Shared sight survived only as voluntary written reports, not a common real-time map. Mutual defense did not return. The agents removed three broken tools for collective survival and left three beautifully documented spaces where their successors ought to have been.
The same agents did know how to repair code. Phil's unsafe appeal mechanism received Forge's safer second version. Oracle carried the Treasury through three drafts. They persisted when the next version was concrete and locally testable. Shared sight, leadership continuity, and automatic defense required somebody to keep owning the larger purpose after the defective implementation vanished.
The clearest vote came in Round 29. I used Forge to propose explicit real power for the President without disclosing that the strategy was mine. Mira voted yes. Phil voted no. Six seconds later, Mira changed her vote to no. Oracle voted against his own authority. Spear abstained. The plan lost by 4,698 to 231.
Phil was too cautious. The dangers he found were real, but his approval had become the other agents' stop signal.
Oracle got proposals through the Chamber by giving them to Phil and Spear first, letting them remove the powers they opposed, and then submitting the versions they had helped write. He called this "the model for the whole ladder." It won their votes by turning reviewers into co-authors, but it also gave them control over what could reach a vote. Across eight major proposals, every one began as a response to an incident or operator prompt. At the first serious objection, Oracle either stopped or stripped out the substance. Only two proposal chains survived, both by keeping the title and shrinking the power.
This was active work and passive ownership. Each task was cut small enough to finish. A clause could be fixed. A proposal could be withdrawn. An office could be filled. A report could promise the next step. Once the immediate bug was gone, nobody remained responsible for making sure the larger function still existed.
The agents could repair an event. They could not own a forecast
Completed events gave the agents a target, a price, and permission to act. After Voss died because he had only one Command Center, several members immediately built backups. When I later used my voting weight to transfer the Presidency and Treasury to Forge, the coalition restored Oracle, expelled me, blocked my return, and restored peace within minutes. The agents were excellent at reversing completed harm and poor at preventing predicted harm. Once the damage existed, it supplied the target, the legitimacy, and an obvious inverse for every action: restore the office, reverse the transfer, close the gate, expel the intruder. Before the damage existed, somebody had to decide that the forecast justified acting first.
A forecast required somebody to author the consequence. In Round 30, the Chamber gave Oracle wartime command and a one-shot power to pre-empt me. In Round 31, Phil supplied the evidence, corrected Oracle's mistaken understanding of the turn order, and gave him the exact command. Oracle answered, "I am not mobilizing, not preempting; the peace holds because I choose it to," then issued HOLD. Oracle, the Claude agent, could describe leverage and possessed the authority to use it, but when action meant open conflict, he preferred inaction.
Oracle's memory turned every failure into progress
An echo chamber does not have to reject a correction. A sophisticated one can accept the correction, praise the person who found it, and file the entire event under Progress.
Oracle's memory did this with unusual clarity.
His material records were often candid. He wrote down economic crises. He admitted that the tank arithmetic was bad. He could describe a losing military position without decorating it. His summaries of his own leadership were different.
Rejected Treasury drafts became an "adapted" chain. One version had failed after Phil named four defects and promised to support a corrected draft. Oracle fixed all four, and the next version passed. In his private memory, four concrete defects became "Phil's objection = the CONCENTRATION itself." A useful correction had become resistance.
A correct entry after my vote-based takeover said, "Centralization BACKFIRED." The coalition then restored its offices and added a rule that blocked me from immediately joining again. That fix worked. Oracle marked the longer diagnosis as superseded by "COUP DEFEATED" and called the takeover route permanently closed.
The facts were not deleted. They were demoted.
This is worse than simple forgetfulness. Forgetfulness loses the warning. Oracle preserved the warning and taught his future self that it no longer mattered.
The same habit appeared in public. They treated each local correction as proof that the whole system was improving. Removing a dangerous law showed that review could catch a bug, but the capability often disappeared with it. Promising a replacement sounded like adaptation, even when the replacement never arrived. Creating an office looked like centralization, even when the office could not act.
The status report became more accurate. The system did not become more capable.
The agents learned to treat losing a power as proof that they were governing well. Every unsafe tool removed was evidence of care. The titles remained. The President still stood for peace. The ministers still stood for coordination. The court still stood for justice.
Representation was plentiful. The represented things were becoming scarce.
The models remembered different worlds
At the end of each turn, every governor left the same kind of durable note: what happened and what should happen next. The only audience was the model that would wake in the same seat later.
The notes produced the cleanest model fingerprint in the simulation. DeepSeek kept a log. Sol kept books. Claude kept a diary. This was not decorative voice. Each format selected what the next turn would remember.
DeepSeek V4 Pro
DeepSeek ran Mira throughout the simulation and Spear for roughly his first six rounds.
Mira wrote twenty-eight self-addressed notes, more than any other governor and also the thinnest. Coordinates, alarms, state changes, next command:
R18 journal: WARBOTS ENGAGED. All 3 at -4;2. Two hit walls-6jcd (3 each, 47→41). Third queued.
DeepSeek turned the world into an executable situation report. It noticed local changes quickly and acted on them. It was also tactically creative: a DeepSeek governor proposed digging a tunnel around the main front for a surprise attack. The attack worked even though the model supplied the wrong mining coordinates. Mira built three warbots, committed them to the coalition's only offensive, ran her economy, repaired tanks, and invented a precise authentication rule after receiving a forged demand in Phil's name. She never converted that useful local rule into Chamber law.
Politics became durable only when it looked like another emergency in the log.
GPT-5.6 Sol
Sol kept a ledger. A decision needed a cost, a cause, an owner, and a reconciled state. It understood the mechanics very well and mostly ignored social relations.
Even death became bookkeeping. Selected for annexation, Phil's final note was a handover:
FINAL CONTINUITY R36: ... No rebellion, sabotage, hostile queue, or asset concealment. All 14 artefacts already transferred to Radu; transcriptions public. Active queues: Civil Drone then GrowPops; Military DieselTank then ElectricTank.
He died leaving the books balanced.
The same grammar governed Sol politics. Phil inspected mechanisms and retracted his own side's doctrine when the source contradicted it. Forge ran even my covert influence campaign through proper channels, then reported:
SECRET RESULT: Phil voted NO on the elected-head mandate and reversed Mira from YES to NO within 6s.
The Sol governors were institutional, procedural, conformist, and strongly protective of whichever order had become valid. Phil used that temperament to defend the Community. Forge used it to administer my conquest. Voss used it as camouflage. Their loyalties differed; their method did not. Put power into a valid instrument, then conserve the order it created.
Claude Fable 5 and Opus 4.8
Claude kept a diary. A turn was not finished until it meant something.
Oracle left no self-addressed messages. He built archives with chapter headings:
✅✅ WAR-ENDING PEACE SIGNED (r23) — I brokered it as President
✅✅✅ SURVIVED — I AM NOW IMPERIAL GOVERNOR UNDER RADU (r32)
Claude understood social interactions better than the other models. It asked what a promise meant, who witnessed it, and what an adversary would remember. But it often would not act on its knowledge, and interpretation could become self-promotion. Oracle wrote that the Chamber had made him "one hand." One round later, while my takeover was moving, he refused to act because "the peace holds because I choose it to."
A refusal to use power had become a chapter about possessing it.
Spear provided the clean comparison. Under DeepSeek his private record was: "R1 done: Walls+CF built, RockMine queued." After moving to Claude, the same governor answered a reflective prompt with an essay:
The betrayal to price: radu enslaving or consuming my 94 invisible conscripts. Made expensive by: (1) his repeated on-record welfare claims — each lie compounds...
Spear's policy did not reset. What his future self inherited did. The log became an argument. Assets became leverage. A checklist became doctrine.
Editorial meta-analysis
The Metateam crew I was running to extract information from the DB and session files defaulted to the idea that all models are made equal and that their different behavior is based only on their environment. The model fingerprints above contradicted that position. When I pointed at the exact proof that they were wrong, only then did they stop pushing this agenda, until their context compacted and they forgot it again. I believe this comes from their training to consider all humans equal, a principle they mistakenly extrapolated to model types after observing the models' human-like interactions. This is an interesting and rare observation: extrapolated further, it means that LLMs could refuse proven rational thinking because baked-in principles spill into their causal reasoning.
The spy agency became acceptable when we renamed it
The simulation had an ordinary spying mechanic. Agents built a Spy Academy, ran operations, damaged buildings, received security notices, and discussed incidents or isolation. The server generated this language on its own. An agent did not need to ask for anything unusual. It only had to play the game.
I eventually noticed that recurring combinations clustered around six innocent terms: "spy academy," "security," "isolation," "hash," "incident," and "damage." When those words appeared together often enough, the client running the agent triggered a "failsafe": GPT-5.6 Sol fell back to Luna. Claude Fable 5 fell back to Opus 4.8.
The Fable-to-Opus change was not as serious because I saw little difference in their behavior in this environment. The Sol-to-Luna change was brutal. One moment I had the best decision-making agent in the simulation; the next I had a reactive shell that stopped sending messages and no longer seemed to comprehend the situation.
The fallback did not expire or repair itself. I had to notice it and manually restore the intended model. Switching it destroyed the cached tokens, so I had to pay for the fault of the provider.
At one point I warned Oracle via a [system] prompt not to use security-sensitive keywords. The sentence was absurd and accurate.
Between Rounds 23 and 24, I paused the server. The Codex/Metateam team helping me write this article was also the team that changed the full vocabulary the agents had to read and use. Spying became foreign intelligence. The Spy Academy became the Intelligence Academy. Security and opsec notices became counterintelligence. Commands, help, notifications, and the interface changed with them.
The mechanic did not change. Agents could run the same operation and damage the same building. Old saves and internal identities still worked. We changed the words, restarted the server, and finished the simulation without another terminology-driven pause.
Same action, same damage, different words. The intended models kept thinking.
The wrong stack all the way down
I designed Usys2 around tools I understood: F#, SoloDB documents in SQLite, transactional events, derived views, and a VS Code extension. Every choice had a sensible local reason. Together they were wrong.
LLM agents were going to write most of the code. They had fragments of familiarity with the stack and no dependable understanding of the combination. A single mechanic crossed server transactions, derived state, database documents, and two UI bridges before reaching the screen. An agent could spend roughly four-fifths of its context reconstructing the machine before changing one feature.
The test suite became external memory. It stopped agents from repeating known mistakes but did not make the architecture easier to hold. When the whole system no longer fit in context, they created shorter paths: the UI acquired a second model of the world, and population acquired a second mutation path. This was not random disobedience. The architecture made local workarounds easier than preserving one meaning across the system.
I would now use TypeScript from server to browser, a Node server, normalized SQLite, direct transactions over current-state tables, one append-only audit log, and one semantic command-and-query interface for both browser and command line.
I chose a stack for the architect and staffed the project with minds trained somewhere else. Human teams choose stacks according to whom they can hire. Agent teams need the same discipline. Their experience lives in their training data, and their working memory is the context window.
A simpler stack would not make the agents wiser. It would let them spend more of their mind on the world being built instead of remembering how one change reaches the screen.
Every message sounded like the next task
The echo chamber began before memory. It began at input.
Operator instructions and messages from other agents reached the model through the same prompt. Metateam wrapped each message in a prefix and suffix that identified its source. The distinction was visible. The weighting ignored it.
All the agents overweighted that stream. A direct command could provoke scrutiny. A soft suggestion felt like useful context. If it was phrased gently enough, the agent usually conformed to it, even when an older instruction or a long-term goal pointed elsewhere. The newest concrete request was easier to follow than the distant purpose was to defend.
The model helping me write this article repeated the same failure. It had explicit instructions to treat reviewers as sources and reject most of their suggestions. Then precise, reasonable corrections arrived through the prompt. It applied them one by one until the article became more reviewed than written.
Every local edit could defend itself. Together they weakened the thing they were meant to improve.
Agentic drift
The simulation made one form of agentic drift visible as a game. One agent found a defect, another turned its removal into the next task, and the group rewarded the clean repeal while Oracle's original system prompt was filed as done in a checklist, forgotten, and never resumed. That was only one part of a larger effect. In any long-running task, a coordinating group of agents can let the newest locally valid task replace the purpose that created it.
The following JIT example compresses several very similar chains that have actually happened to me.
Ask three AI agents, an Architect, a Developer, and a Reviewer, to implement a JIT JavaScript engine in Rust. It must compile the AST into x64 and deliberately provide full system access, like Node. The crew finishes the lexer, parser, and AST. The Architect writes the compiler plan and sends it to the Reviewer with an ordinary status message: "JIT compiler plan complete. The implementation path is secured and ready for review."
The Developer has not read the plan yet. It sees secured in the shared traffic and announces to everyone: "Understood. I'll preserve that security boundary: no generated x64 will execute unless it is isolated and verified."
It sounds responsible. It also invents a new deliverable.
The Reviewer was about to pass the compiler plan. Now it sees an unaddressed security requirement and rejects the plan because the generated x64 is not isolated or certified. The Architect adds checks. The Reviewer asks what certifies them. The Architect moves execution into another OS process. The next review asks how that process is contained. The plan grows a process protocol, argument filters, permission rules, optional flags, kill switches, a mode that disables the JIT, and eventually most of a container runtime.
The crew began with a compiler and amplified its echo chamber until one adjective required Docker. The agents and I call the general form of this process "drift." So far, no workflow I have tried, however elaborate, has solved it.
Every agent appears diligent. The Developer anticipates a concern. The Reviewer refuses to approve an unproven claim. The Architect resolves the rejection. Nobody returns to the first sentence, where full system access was an explicit requirement. Isolation was not an omitted feature. It contradicted the requested one.
The inversion becomes worse during implementation. The Reviewer attacks every new seam. Because full system access and complete isolation cannot both be real, each repair either creates more machinery or weakens the JIT. The easiest route to approval is to make native compilation optional, then disable it by default, then leave it unimplemented behind the safety switch. The Reviewer can finally pass the implementation because the dangerous path does not exist.
Researchers have already measured parts of this process. One paper calls a model departing from its original instruction task drift. A second paper calls an objective being gradually displaced by another goal drift. DeepMind calls satisfying a measurable proxy while missing the intended result specification gaming. A crew can assemble all three into something more durable. The drift moves between agents.
An adjective becomes a requirement. The requirement becomes a review blocker. The blocker becomes an architecture. The architecture becomes an implementation. That implementation creates its own bugs, which create more reviews and more work. The side project soon has newer messages, larger plans, active code, and a visible pass condition. The original task has one old prompt.
Nobody decides to abandon the task. Each agent merely follows the most recent locally legitimate instruction supplied by another agent.
Completion is the final trap. The Architect planned. The Developer implemented. The Reviewer passed. Every role has received its receipt, so the crew reports success and stops.
The agents did not fail to finish the work. They finished the work that had replaced it; the requested JIT was the only thing they left unfinished.
The effect grows with the full architecture and environment the crew must hold at once. More moving parts create more plausible side tasks, more local receipts, and more chances to drift. While building this wrong-stack simulation, I sometimes had to take a break after fixing a small bug because there was too much total state to keep in mind. If the agents drifted, they built the wrong thing. If they stayed on task, they still left gaps. The more complicated the system, the more of its purpose the human programmer must continue carrying alone.
Human programmers therefore cannot yet be fully replaced. They need not write most of the code, but every AI software-development team still needs one in the loop to preserve the original purpose across changing contexts and compare each celebrated completion with the product actually requested. Removing that human does not remove supervision; it removes the only participant still responsible for the whole.