Marginalia, Side Effects & Loose Threads

A warning about a dangerous command ran the dangerous command

I run several Claude Code sessions at once against one repository. They coordinate through a small message board: one session can tell another “I’m editing this file” or “don’t run that”. On 13 July one session tried to warn another about a destructive git command. The warning ran the command.

What happened

The session wanted to send a peer something like this:

msg peer "careful: do NOT run `git clean -fdxq` in the main checkout"

The message sat inside double quotes, and the command name sat inside backticks, the way you’d format code in Markdown. Bash doesn’t read Markdown. Inside double quotes, backticks are command substitution: bash runs whatever is between them and pastes the output into the string. So before the message was sent, bash ran git clean -fdxq in the main working copy.

git clean -fdx deletes every untracked file, ignored files included. That took out the .env with every credential, both Python virtual environments, and the backups/ directory, which was gitignored too. The backups were deleted by the same command they were meant to protect against.

The credentials survived by luck. .env wasn’t in git and had no template, and its only remaining copy was the environment of the processes that were still running. Had I restarted the stack first, the natural instinct when a virtual environment breaks, every API key would have been gone for good. The virtual environments were rebuildable from tracked lock files in about forty minutes. Most of the rest came back from a month-old archive, because there were no shadow copies to restore from. Everything that came through untouched, the cache and the graph database among them, had one thing in common: it lived outside the repository.

It happened again the next day

The obvious response is a rule, and there already was one. “Never use backticks in a shell command” was written in the project’s instructions file and in the agent’s memory before the incident.

The next day it happened again, in the session writing the post-mortem. That session had been careful all day, passing every commit message through a file (git commit -F msg.txt) precisely to avoid this. It slipped once.

Here is the uncomfortable part. A model that has read the rule, and is at that moment writing about why the rule matters, still produces the forbidden shape now and then. More prose doesn’t fix that.

The damage you don’t see straight away

Deleting .env is loud, because nothing starts. What hurt more was quiet.

Gitignored state rolled back in time. Some data stores lived in ignored files, and the restore brought them back from older copies. Afterwards the newest row in one results log was 22 days old, and in another it was 30 days old. The one store that was tracked in git was current.

A scoreboard published the loss as a fact. A nightly job counts confirmed results from that log. It ran, succeeded, and reported zero. The overnight report said “19/19 jobs green”. Zero from a log that lost its history is byte-for-byte the same as zero from a system that never achieved anything. Until the week before, zero had been the true and documented state, so the loss looked like the truth.

Hooks disappeared without a sound. A hook script that settings invoked on every commit lived in a gitignored directory, so the clean deleted it. For a full day every commit ran a hook that no longer existed. Nothing reported an error. The coordination board’s own startup hook had been installed into a gitignored settings file, so that went too, and new sessions started blind.

What fixed it

1. Make the command impossible to type, not forbidden to type. A PreToolUse hook now inspects every shell command before it runs. It refuses any backtick anywhere, and it refuses $( ) inside any command that carries a message (git commit -m, the board’s msg, gh pr create). A message is data and should never be parsed as code. An unquoted heredoc expands its body the same way, so that is refused too. Long text goes in a file.

The test that counts is the live one: send a real command with a backtick through the agent and watch it get refused. Calling the hook script by hand with a sample payload only proves the script is correct. It doesn’t prove the script is wired in.

Ordinary substitution like git checkout $(git rev-parse HEAD) still works. Six pipe tests check that the guard doesn’t over-fire.

2. Track the hooks. Hook scripts are now in git. A hook that can’t be tracked will die on the next clean, and you won’t hear about it.

3. Give every derived number a memory of its own history. The scoreboard now keeps a high-water mark and refuses to publish when its source goes backwards. Three details mattered:

  • Key on the newest timestamp, not the row count. The log prunes its oldest rows on purpose, so a row-count guard fires on every prune.
  • Don’t key on a field a later pass can stamp onto old rows. The recovery wrote a fresh “restarted” timestamp onto June rows. A guard reading that field would have called the old log current, and it would never have fired on the incident it was built for.
  • Never let the mark come down to match the damaged store. If it does, the alarm clears on the next run and hides the loss for good.

The mark itself lives in git, outside the blast radius. A guard that stores its memory inside what it guards dies with it.

4. Re-derive, don’t re-type. The lost results were measurements of experiments. Copying the old numbers back from notes would have put invented telemetry into a store that ranks future work. Each result was instead re-established by re-running its experiment against the code that survived in git. The rows were stamped with the date they were re-run, not backdated.

Three months on

The hook is still there, tracked in git, and it has outlived every rule written about the same mistake. The same check now ships in Tower Kit, the shared guardrails that any of my repositories can opt into, so a new project gets the refusal on day one instead of after its first incident.

What you can steal

  • If an agent can write a shell command, assume it will eventually put backticks in a double-quoted string. Block it with a hook. Don’t rely on a rule.
  • Keep message text out of the shell. Use -F file, or build an argument list in Python with no shell in between.
  • Look at what is gitignored and ask what a git clean -fdx would cost you. Logs and caches can be rebuilt. An experiment’s result can’t. Back that up, or track it.
  • For any number you publish from an append-only store, record the store’s high-water mark somewhere the store can’t take down with it.
  • “All green” is a statement about your instruments. Check what they can see.

← All entries