What I learned „the hard way“ steering a coding agent through a production recovery

We had two data stores that were meant to stay in sync. Because of a power outage, one of them lost a few days of confirmed updates, the other one still had them. The system was built on the assumption that the second store can only ever be behind, never ahead, so there was no code path for what had actually happened. Nothing healed itself. The only visible symptom: a handful of records that suddenly could not be opened anymore.

To get the data back I had to find the mismatches among roughly two million records, pull out what only the „ahead“ store still had, and write it back through the product itself. No bulk import, no REST interface for this, a production system of a customer, and a clock running. I did it with Claude Code over one very long session, spread over three days. The agent wrote practically all of the code. This post is about my half of the work: the prompts that steered it, and the places where it went confidently in the wrong direction.

The setup

I started Claude Code inside a checkout of the server source code, which turned out to be the most important decision of the whole incident (more on that below). On top of that it got SSH access to the production host, access to a QA system that had suffered the same outage, and the credentials for the server’s remote admin interface in a local .env file. The rules were simple and said once at the beginning:

I have ssh access to the server (prod system, be careful on all writes/updates, reads are fine) via ssh <user>@<host>

The very first prompt was a typical „I have no idea yet“ prompt. It asked for a plan, not for action, and it gave the one number we had:

Hi, we have a problem on a customer. The <cache-db> and PostGRES are out of SYNC. but we can ssh into the server. What kind of Java file would we need to execute to find the difference? Please make a proper plan and check how we can execute a fast query on what assets might have changed in <cache-db> in the past two weeks that might have gotten out of sync. We screened the server log and it showed 147 assets that cannot be queried anymore because out of sync. but we guess it is a lot more

It was a lot more: 730 records had diverged. What came out at the end was a small toolkit: Java scripts that run inside the server (the host only allowed Java and bash, see lesson 4), compiled by the server itself, plus bash wrappers like update_test_assets.sh, update_730_lost.sh, remove-surplus-values.sh and a compile-check.sh, all with a short README so colleagues could run the same thing on the QA system. Every step went to QA first, then to two hand-picked test records on production, then to everything.

Here are the lessons, with the real prompts. The LinkedIn version had two number threes. That is what happens when you write lessons at the end of a three-day incident.

1. Read the source, don’t guess

The agent needed to make a single write land: put an old value back into a record. It burned about eight attempts, each a plausible variation of the one before: different ids, raw SQL inserts, stripping this field, keeping that one. Every attempt looked like progress. None worked, and one even flipped a value back and forth in front of my eyes:

ok. but why do you do SQL inserts instead of using <the product's Java write API>? you have the correct XML, except for <change-counter> and version need to be removed!

wtf. I saw the value_long="0", then you did value_id and now it is back to "1" at <change-counter>=391

What broke the loop was this one:

ok. so cycling does not work. Read the code. It is here at <server-source>, there must be a simple way to read in such XML as the <product's own asset import> does it

Two minutes later I typed „cool. run both test assets now!“. The answer had been in the source all along; the agent had just never looked, because guessing the next parameter is cheaper than reading a code base. If an agent has failed the same task three ways, a fourth attempt will not help. Change the instruction, not the parameters.

Finding: Look into the source first, if you have it!

2. Verify the artifact, not a proxy

More than once the agent reported success because a version number and a change counter had gone up. Those are side effects, not evidence. A colleague opened the two test records in the desktop client and could not tell what was supposed to be different. So I kept asking the same boring question in different shapes:

question: What should have changed on the two example assets, as my co-worker updated them on prod. They wanna check in <desktop client>

ok. so what is different between the asset and the restored version (even if not restored)

ok. But why is archive 0, live 1. This should have been done by your <reimport script>. So please check again by ./update_test_assets.sh <id> which should have your recent code.

Three times I asked whether the content had actually changed, and three times the honest answer differed from the confident one. Agents gravitate to whatever signal is cheapest to read.

Finding: Ask for the thing a human would look at for verification.

3. Sample randomly, because self-verification can endorse the damage

The big one. After the bulk run, I asked for a quick random check:

nice. Let's do a quick check. Randomely select 3 assets from <lost-versions> XMLs and check features. Randomely check 3 storage items hashes, of the 69 assets with removed rows, randomely select 3 and check if selected

Even the random draw was instructive: the first three picks came from all eligible records, and 527 of the 698 had been deliberately skipped because the live data was already newer, so the sample proved nothing. Redrawn from the records actually written, it found two „gaps“ that were artifacts of its own SQL (one join on a version column that is empty for version-independent values, one timezone). And then the uncomfortable part. Together with a follow-up question about a list of 28 records that would not restore:

do you know what are those 28? Any idea? how do you know it?

It turned out that the outage had also rewound the id counter. A different record now lived under each of those ids, and the file-restore step had written old files into 27 of them. The agent’s own verification had passed every single one, because it compared each restored file against the archive it had just copied from: exactly the invariant that should never have matched. An agent checking its own work will often prove itself right. The fix was an identity guard (archive name and type must match the live record, otherwise exit) in every write path.

4. State the environment

It assumed tooling that simply was not allowed on the host, and it assumed an admin CLI could be scripted like any other CLI:

you cannot run python on prod! only Java and bash!

I can tell you, Our Admin tool is made for Humans. So you first run -login, than you run -c and type the command. all with hitting enter in between. but if localhost works fine, enjoy

That is missing context, not bad reasoning, and it was mine to supply. Today these constraints go into my first prompt.

5. Don’t accept „unrecoverable“ the first time

The whole recovery started with a question that sounded hopeless:

ok. we think we had a power outage on PostGRES not on <cache-db>. so Postgres after PowerOutage was reverted to its latest state, which was already a bit older. So the question is, any way we can restore the work that only happened on <cache-db>???

And later, when files ended up on the „cannot be restored“ list:

- then recover them via zip. it makes no sense to tell them we cannot restore. please ship the 26 master files and restore! now!

So I needed to tell the agent to do proper backup restore! It did not help to let him do what he thought was the right backup.

7. Distrust tidy aggregates

Twice the numbers looked suspiciously clean, and twice the comparison was wrong. Once it reported „1,612 lost texts on 309 products“; the real figure was 1,494 missing nested rows on 74 products, because its SQL had split multi-line texts. Once it compared flat rows and ignored nested structure completely:

I do not get your table. the problem was that on the three example assets, they had 14 features in restore, while 17 on prod. so 3 features were removed and this we cannot rebuild. How many such cases exist and which of those cases influence output-channels. [...]

are you sure for <id> it is not just nested this 1 missing? do a xml diff. I downloaded all old XMLs with <id>.xml into "out"

When a count looks like a mirror image, ask what identity it keys on.

How it ended

Of the 730 records, 196 got their values back, 8 relations were re-created, and 85 stale values on 52 records were removed (old values the database restore had resurrected and a merge cannot clean up). 28 could not be restored because another record owns their id now; 27 of those were transient import files that can simply be re-imported, and exactly one was real business data. A final comparison across all 1,901,672 records reported zero differences in either direction. The week after was less tidy: more lost work surfaced from a different source, and that is another story.

If you want to try this

• Start the agent in the source code of the system you are repairing, and point at it the moment it starts cycling.
• Put the host constraints (languages, tools, what is read-only) into the first prompt.
• Always go QA, then two named test records on production, then everything. Let a human look at the test records.
• Ask for random samples, and check them against something the agent did not write itself.
• When it says „unrecoverable“, ask what prior state or derived data might still exist.
• Use sub-agents for independent pieces, but hand them the invariants the main thread has learned.

Agents are strong at doing and weak at knowing when they are wrong. Supply the doubt yourself.