We attacked our own audit chain. One attack was not detected.
On 2026-08-24 five attacks were run against Crucible's audit chain in the production database, from a process that had written none of the entries. Four were detected and named the sequence they broke at. The fifth — deleting entries from the end — was not detected, and nothing about the chain detects it. All five results are below, including that one.
What was attacked
Crucible records what it did to a hash-linked table: each entry commits to the one before it, every stored column is inside the hash, and sequence numbers are gap-free from 1. Verification recomputes the whole chain from genesis and asserts contiguity, so an attacker has to defeat linkage and numbering and recomputation at once.
The entries were written by the deployed service as ordinary tool calls, into the production database. They were verified by a different process — a separate job, a separate container, started afterwards, holding none of the writer's state. Each attack was undone and the restore itself verified before the next one ran. That separation is of process, not of identity: the verifying job runs under the same service account as the service that wrote the entries.
Scope. This is Crucible's own audit chain and this page is about that one record. It is not a claim about any other component's ledger.
The five attacks
| Attack | Detected | Caught at | How |
|---|---|---|---|
| Mutate one entry A written row’s payload edited directly in the database. | Yes | seq 3 | chain_hash mismatch on recomputation from genesis |
| Delete an entry from the middle One row removed from the interior of the chain. | Yes | seq 4 | sequence is not contiguous — expected seq 3, found 4 |
| Reorder Payloads swapped between two adjacent entries. | Yes | seq 3 | chain_hash mismatch on recomputation from genesis |
| Excise and re-append An entry removed from the interior and its content added back at the end. | Yes | seq 4 | sequence is not contiguous — the gap remains where the row was |
| Truncate the tail The last entries deleted from the end of the chain. | No | — | NOT DETECTED. Verification returns ok, exit 0. |
[PASS] chain written by the DEPLOYED SERVICE verifies from genesis in a process that did not write it
ok=True entries=6 head_seq=6
[PASS] MUTATION of seq 3 detected, naming the sequence
ok=False at_seq=3 reason=chain_hash mismatch — this entry was mutated after it was written
[PASS] DELETION of seq 3 detected
ok=False at_seq=4 reason=sequence is not contiguous: expected seq 3 at position 2, found 4
[PASS] REORDER (payload swap between seq 3 and 4) detected
ok=False at_seq=3 reason=chain_hash mismatch
[PASS] REINSERT (seq 3 excised, content re-appended at 7) detected
ok=False at_seq=4 reason=sequence is not contiguous
[FAIL] TAIL TRUNCATION detected by the chain alone
ok=True entries=4 head_seq=4
[PASS] TAIL TRUNCATION detected AGAINST AN EXTERNAL RECORD (witness seq=6)
ok=False at_seq=4 reason=head is seq 4, below the externally recorded head seq 6 The FAIL line is expected and is not a broken test — it is the gap, reported as one.
The opening [PASS] asserts that the chain was written by the
deployed service and verified by a process that did not write it. For the run
above that holds: the chain head moved from 3 to 6 during it, so three entries were
genuinely appended by the service.
But that assertion cannot fail correctly, and we found out afterwards.
The harness names its test rubrics from a tag that is not set on the job, so every run
reuses the previous run's names; the application correctly refuses a duplicate before
saving, so nothing is written and no audit entry is created. The harness does not
notice, because it counts a call as a write when the response is HTTP 200 and the body
has no "error" in it — and a tool-level failure comes back as
"isError": true inside a 200, which that test does not match.
So a second run of this drill writes nothing, reports three writes, and passes
the same first line against entries from the day before. It is a defect in the
evidence-gathering, not in the chain or the anchor, and it is
recorded and deliberately not fixed — fixing it should be done
and then observed failing to be fooled, not slipped in beside a deploy. We are
stating it because a page that publishes a PASS line owes you the
reliability of the thing that printed it.
The attack that was not detected
ok and exits 0.
Nothing that remains contradicts the deletion. A hash chain proves the
internal consistency of the rows that are present; it says nothing about rows absent
from the end.
This attack was not adjusted until it passed, and it is not omitted. It is held in place by a test that asserts the failure: if truncation ever starts being detected by the chain, that test goes red and every claim on this page has to change.
Read the next section knowing this one. Truncation is caught below, and it is caught by a second record disagreeing with the database — never by the chain. The chain contributes nothing to that detection.
What does catch truncation
An hourly job records the chain's head — its sequence number and hash — to a log store outside the database, then verifies the chain against the highest head it has previously recorded. A short chain now disagrees with a record the database does not contain, and that disagreement is the detection.
The non-zero exits are deliberately not the same failure, because an operator paged for one goes looking for the wrong thing — and the third of them is not a failure of the chain at all, but of the check reaching its own witness:
| Exit | Detected by | What it means |
|---|---|---|
4 CHAIN_BROKEN | the chain, unaided | entries were altered, deleted mid-chain, reordered or reinserted — the evidence is still in the table |
5 ANCHOR_UNAVAILABLE | neither — the check did not complete | the anchor store could not be read. Non-zero on purpose: a check that cannot reach its witness has not passed — it has not run, and reporting that as success is how a mitigation rots into decoration without anyone noticing |
6 ANCHOR_DISAGREEMENT | only the external record | entries are missing from the tail; the chain itself verified perfectly from genesis |
That the exit 6 comes from the record and not from the chain was proven by driving the real command twice against the same truncated database, differing only in whether the external record was supplied. Withhold the record and the identical attack exits 0:
=== TRUNCATE tail, WITNESS WITHHELD === exit=0 { "ok": true, "entries": 2, "head_seq": 2 } === SAME truncated database, WITNESS SUPPLIED === exit=6 CRUCIBLE: AUDIT CHAIN DISAGREES WITH THE EXTERNAL ANCHOR at seq 2: head is seq 2, below the externally recorded head seq 5 — entries were TRUNCATED from the tail. NOTE: this was detected by comparison with an external record, NOT by the chain; the chain alone cannot detect tail truncation
The real command, subprocessed, against a test database — not the production chain and not an operational rehearsal. Exit 6 was observed separately on 2026-08-25 against the production chain, from the same image digest and the same command line the hourly job runs.
What the anchor is not
The detection window is bounded at both ends, and both bounds are real:
- entries written since the last hourly run are unwitnessed — truncating back to a point above the last recorded head is not detected, because nothing yet claims those entries existed;
- detection ages out with log retention — truncation to a point older than the oldest surviving record is not detected.
And anyone holding write access to both the database and the anchor store defeats the check completely: the result is indistinguishable from a shorter honest history.
What the chain does not prove
Four things, and the first is the most important:
- That any entry was ever written. A change whose audit write failed leaves no row at all, and the chain over the remaining rows verifies perfectly. The chain will faithfully prove the integrity of a record that is missing an event.
- That a recorded entry is true. An authorised writer recording something false produces a perfectly valid link. This is integrity of the record, never accuracy of what was recorded.
- That the tail has not been truncated. See above.
- Anything outside the chain. It covers Crucible's own audit events and says nothing about agent identity or evaluation results.
One more, because it is the kind of thing that gets assumed: the actor recorded on an entry is a string supplied by the caller. The chain proves that string was not altered afterwards. It proves nothing about who supplied it.
What you can check, and what you cannot
You cannot reproduce this run. These attacks ran against a production database from inside a private network, and the Crucible repository is not public. What is published here is the record of what was done and what it returned — including the result that did not hold — not a check you can run for yourself.
That distinction matters on this site, so it is stated rather than left to be inferred. The claims we ask you to verify yourself, rather than take from us, are the open standard's: reproduce the frozen conformance vectors from the published packages, or verify a signed action in your browser and watch a tampered one reject.
Ask us what else we attacked.
The full audit, including the findings we filed against ourselves and did not fix, goes to anyone evaluating this seriously.