- Work
- Recovery Console
Reliability tool · simulator · Recovery Console
Reconnected does not always mean recovered.
See what survived, what is still waiting, and what needs a deliberate retry.
- Acknowledged
- 270 s of audio
- The receiver confirmed it.
- Not yet confirmed
- 0 s of audio
- Held, or sent without a confirmation.
- Lost
- 90 s of audio
- Evicted to make room. Not coming back.
360 captured = 0 held + 0 sent, unconfirmed + 270 acknowledged + 90 evicted
Overview
Product study · 2026Design and engineering
The problem
When a recording app loses its connection, the screen usually shows two words in turn: Reconnecting, then Connected. Neither says what happened to the audio captured in between, or whether the system downstream accepted the record it was sent. Someone looking into a missing minute has nothing to go on.
The change
Recovery Console replays an incident one event at a time and keeps the accounting in view. Every second of audio is held, sent, acknowledged or evicted, never two at once, and the totals have to add up. A separate integration scenario shows a rate-limited send, a newer revision arriving mid-flight, and a replay that needs a reason.
My part
I worked on the capture buffer and the integration worker this study draws on. For the study I designed and built the simulator, its state contract, the console and its tests.
Three decisions
Sent is not received
The console keeps apart three things a status light merges: audio held on the device, audio handed to the connection, and audio the receiver confirmed. Just after reconnecting, the status already says Connected while nothing is confirmed: part of the queue is still held, and part is sent without a confirmation. If the connection drops mid-send, those seconds stay unconfirmed instead of quietly counting as delivered.
Mechanism and evidence
Source mechanism
- Queued audio is sent oldest first once the connection is streaming again (capture controller @ de1291f).
- Once a block is handed to the connection, the client keeps no copy, so it cannot resend it.
Simulator protocol
- Acknowledgements per batch of ten. The source reads acknowledgement messages but does not track them per block.
- Capture freezes when the outage ends, so the accounting has a fixed scope.
- Acknowledged
- 0 s of audio
- The receiver confirmed it.
- Not yet confirmed
- 120 s of audio
- Held, or sent without a confirmation.
- Lost
- 0 s of audio
- Evicted to make room. Not coming back.
Accounting at this event
| State now | Chunks | Bytes |
|---|---|---|
| Held on the device | 108 | 6,912,000 |
| Sent, not confirmed | 12 | 768,000 |
| Acknowledged | 0 | 0 |
| Evicted | 0 | 0 |
| Captured | 120 | 7,680,000 |
Reconciles: 108 + 12 + 0 + 0 = 120
- Ever sent
- 12 chunks, including acknowledged. Not added above.
- Never evicted
- 120 chunks, in any state.
- Queue
- 6,912,000 B of 17,280,000 B (40% full)
Capacity is a number, and loss has an address
The queue holds 270 seconds. A six-minute outage cannot fit, so each new second evicts the oldest one still held. The console names the gap exactly: chunks 1 to 90, from 0:00 to 1:30, 5,760,000 bytes. Reconnecting changes the status, not the gap.
Mechanism and evidence
Source mechanism
- A cap of 270 × 64,000 bytes. When a new block would exceed it, the oldest block is dropped.
- Audio is 16 kHz mono with 4-byte samples, so 64,000 bytes is one second. This is arithmetic from the handshake, not a named constant.
Simulator protocol
- This scenario models the byte cap only. The source also stops reconnecting after 270 seconds; the session then ends and the held audio is cleared. A real six-minute outage would lose more than this scenario shows.
Event 273 of 662
Source mechanismCaptured, oldest evicted
Chunk 271 would take the queue past capacity, so the oldest held chunk, 1, is evicted.
- Virtual time
- 4:31.0
- Actor
- capture
- Chunk
- 271
- Audio covered
- 4:30.0–4:31.0
- Size
- 16,000 samples · 64,000 B
- State now
- held
- By the end
- acknowledged at 6:19.3
- Evicted
- chunk 1 · 0:00.0–0:01.0 · 64,000 B
- Queue after
- 17,280,000 B of 17,280,000 B
Retry waits for time, the right revision and a person
The work board answers 429 with Retry-After, and Send now refuses until then. A success for revision 1 arrives after revision 2 exists, so the row stays open. A permanent rejection goes to dead letter. Replay asks for a reason and a confirmation, and Cancel creates nothing.
Mechanism and evidence
Source mechanism
- 429 is classified as rate-limited, and Retry-After seconds set the next attempt (integration worker @ 9945464).
- Every final write requires payload revision = claimed revision; otherwise the row is released.
- Permanent 4xx goes to dead letter. Replay exists only as a command-line script.
Simulator protocol
- Replay as an in-app action with a reason, one idempotency key per replay intent, and a single half-open probe. The source sends no idempotency key, and nothing here claims exactly-once delivery.
Attempts
| Attempt | Revision | At | Response | Wait | Key |
|---|---|---|---|---|---|
| 1 | 1 | 0:00.0 | 429 · Rate-limited | 30 s → 0:30.4 | obx-7f3a:r1 |
| 2 | 1 | 0:30.4 | 200 · Stale success refused | — | obx-7f3a:r1 |
| 3 | 2 | 0:30.8 | 422 · Permanent failure | — | obx-7f3a:r2 |
| 4 | 2 | 0:32.3 | 200 · Accepted | — | obx-7f3a:r2:replay-1 |
The state contract
Every value on screen is derived from these records. The buckets are exclusive, so they must reconcile at every event.
- Captured = held + sent, not confirmed + acknowledged + evicted, at every event.
- Ever sent counts unique chunks and includes acknowledged ones. It is never added to the line above.
- Never evicted counts chunks in any state that were not evicted. It differs from the queue held right now.
- A retry or a duplicate acknowledgement never adds a captured or sent chunk.
- Chunk
sequence, capturedAt, sampleCount, bytes, state (held, sent, acknowledged, evicted), sentAt, ackAt- Incident
id, scenarioVersion, seed, virtualTime, capacityBytes, transportState, sourceVersion, events- Delivery
outboxId, sourceRecordId, payloadRev, claimedRev, destinationAlias, attempt, status, responseClass, nextAttemptAt, idempotencyKey, lastError- Audit event
incidentId, eventType, virtualAt, reason, revision, actor
What is checked
- Same seed and conditions give the same events, and Reset returns to them.
- The queue never exceeds 17,280,000 bytes after any event.
- Scenario B loses exactly chunks 1 to 90, 5,760,000 bytes, and reconnecting does not change that.
- Chunks are sent oldest first, and none is acknowledged without an acknowledgement event.
- The four buckets reconcile at every event, under every condition.
- Send now is refused before Retry-After, and creates no attempt.
- A success for a stale revision never finalizes the row.
- A canceled replay creates no attempt; one replay keeps one key across retries.
- A malformed event is refused and changes nothing else.
Where it stands
- Built
A deterministic simulator: the same seed and conditions always give the same events.
- Built
A console that steps, plays and scrubs through every event, by keyboard, mouse or touch.
- Built
The invariants above, checked in one test suite.