Three True Records

**Answer in brief: Ratify, re-draw the boundary, record the provenance unsmoothed. In this case, I also propose a prose version of code lockstep tests. **

Earlier this month, I found that the voice assistant I’d built for my father (https://maximus.work/) was writing marginalia about the state of its own database:

“e34, e36, and e37 are substantially redundant — all capture the weekly [medication] routine; consolidated into one thread.”

“…operator may wish to clarify and correct the resolved date.”

I never asked it to audit anything, nor did I decide that there should be an “operator” that the synthesis agent reports to.

So, I wondered, did I prompt for that? Did I build that somewhere?

Or, more accurately, did an agent build it and I said “yes” without realizing?

The codebase underneath the agent keeps an append-only decisions log. Every stored fact carries provenance. And, every change ships through review. So I asked Claude Code to dig through the repo and figure it out.

The answer, after twenty minutes of archaeology, was neither yes nor no.

In May, we designed the field. That’s very clear on the record: a notes slot for “optional metadata from the synthesis call (e.g., synthesis-prompt’s self-reported uncertainty about a section).” So, we asked for the model to talk about its own confidence as it compiled a user snapshot from atomic facts. I did this because I wanted to learn about the model’s ability to self judge.

Then, on June 2, when writing the actual prompt, the agent rendered that as “an optional short free-text note for *the operator.” I didn’t notice the language at the time — and it appears nowhere else. Not in the spec, not in the decisions log, and not in the pull request, whose body was one byte. It's just a wording choice the agent made.

Of course, the blame line credits the commit to my account, because the agents commit under it. I didn’t write the phrase — because I don’t directly author any of the code.

(We now have a rule — hard earned, like all my self-taught build rules — that would have caught this. Starting in August, I required each build sequence to first have a pre-work packet that states what was checked against the plan.)

On June 3, we decided to inject the notes into live calls. That decision used the spec’s framing, not the prompt’s. From that day, three records described one field three different ways. The spec: the model’s “self-reported uncertainty.” The prompt: “a note for the operator.” The decision log: “side-notes informing the model’s confidence, not content to surface.” They quietly disagreed for three months.

And during those three months, the model put the field to use. We gave it free text addressed to an operator, and it started auditing the database in it: flagging redundant entries, questioning suspicious dates, reading a meeting stored at 3:30 AM as “almost certainly a parsing artifact from ‘2 to 3:30 PM’” — a real storage defect — and treating it as 2:00–3:30 PM in the snapshot, which protected the next day’s call. The model’s judgment was right in all four cases we checked.

The trouble, though, is that, in the example above, the snapshot read around the bad value: the stored row still said 3:30 AM, and the reminder would fire from the row. And, had 3:30 AM been a real overnight flight, the snapshot would have overwritten a true fact with “almost certainly.”

**The model had turned the free-text space into a third correction surface, in a system whose doctrine allows exactly two: the database, through pipeline logic, and the call, through conversation. **

What kind of failure is this?

Not a rogue agent. One phrase drifted, in one commit; every other step was designed, decided, and reviewed.

Not a missing record. Every step has one, and each one made its own step look governed.

The defect is that the prose records were never checked against each other.

For code, we had built a parallel instrument: a lockstep test fails the build when two copies of a constant drift apart. For prose descriptions of the same surface, nothing existed whose job was the comparison.

The happy and perhaps uncomfortable outcome is that the drift was good. Three months of this failure produced no harm we could find, and measurable help.

So, when the model starts doing something useful you didn’t ask for, what do you do?

We didn’t want to rip it out and lose value. We didn’t want to ignore it and leave it ungoverned. **So we ratified it, re-drew the boundary, and recorded the provenance unsmoothed. **

• The notes stay under a contract that restores the original May framing.
• The QA behavior the model arrived at got its own sanctioned channel: stored, counted, structurally unable to reach a user by accident.
• And the permanent record now says built uninstructed on June 2, right beside the ratification.

The story also proposed its own instrument: for every piece of text that reaches a model, compare who the text says it’s for against who the records say reads it.

An audit with that instrument across seventeen surfaces returned sixteen clean. The seventeenth is the weekly-review prompt — due to fire for the first time in three days, at our family beta’s graduation. The injected text opened with a developer’s note: “Every line below that looks like copy is a placeholder.” The model would have been told its script was fake on opening night. We fixed it that afternoon.

To return to the beginning: the agent that wrote “operator may wish to clarify” — of a trip date it suspected was mis-resolved — was right to flag it. Someone needed to check it. But the operator it imagined didn’t exist until it wrote to him.

Lockstep tests exist for code constants. This is the prose version. It now runs at every release gate.

---

The receipts

May 6 — The field is designed: notes, “optional metadata from the synthesis call (e.g., synthesis-prompt’s self-reported uncertainty about a section)” (v1.5 schema spec, founder-ratified)

June 2 — The drift: the prompt renders it as “an optional short free-text note for the operator” (commit 7a55af6, PR #19) — in no spec, decision, or PR body

June 3 — The injection is decided and recorded — “side-notes informing the model’s confidence, not content to surface” — already disagreeing with the prompt (decisions log, PR #20)

Aug–Sep — The model does QA in the gap: redundancy consolidation, a parsing artifact read around, date-consistency flags — correct in all four observed cases, uninstructed in all of them (stored snapshots, four accounts, Sep 2–3)

Sep 2–4 — Measured (dogfood Finding 92: “a third correction surface, invisible and unaudited”), then ratified: notes kept under a register contract; the QA behavior given a stored-only, counted channel; provenance recorded unsmoothed (PR #266)

Sep 4 — The audience audit — the instrument the story proposed — runs once across seventeen surfaces and catches a second live fork, three days before it would have fired (PR #267)