# Playbook — Crash an agent mid-write and bring it back without duplicating anything

Reproduce an interrupted incident trail, resume it from scratch, and verify that the graph, the receipts and the audit log all agree afterwards.

## What you need

- Docker 24 or later
- Python 3.9 or later
- About 3 minutes

## 1. Start a disposable Corrobore

```bash
mkdir -p .corrobore-tls .corrobore-secrets
printf '%s\n' 'change-me' > .corrobore-secrets/http-token
openssl req -x509 -newkey rsa:2048 -sha256 -nodes \
  -keyout .corrobore-tls/server.key -out .corrobore-tls/server.crt \
  -days 30 -subj '/CN=localhost' \
  -addext 'basicConstraints=critical,CA:FALSE' \
  -addext 'keyUsage=critical,digitalSignature,keyEncipherment' \
  -addext 'extendedKeyUsage=serverAuth' \
  -addext 'subjectAltName=DNS:localhost,IP:127.0.0.1'

docker run --rm -d --name corrobore-playbook \
  -p 127.0.0.1:8080:8080 \
  -e CORROBORE_STORAGE_MODE=ephemeral \
  -v "$PWD/.corrobore-secrets/http-token:/run/secrets/corrobore-http-token:ro" \
  -v "$PWD/.corrobore-tls/server.crt:/run/secrets/tls.crt:ro" \
  -v "$PWD/.corrobore-tls/server.key:/run/secrets/tls.key:ro" \
  ghcr.io/estance-labs/corrobore:latest

curl --insecure -H 'Authorization: Bearer change-me' https://127.0.0.1:8080/health/ready
```

## 2. Install the notebook dependencies

```bash
pip install requests
```

## 3. Run the notebook

```bash
jupyter lab notebook.ipynb
```

The notebook loads `dataset.json` from the same directory. Keep the three files together.

## 4. What to observe

1. `/health/ready` reports `storage_recovered` separately from liveness; that is the check to gate writes on.
2. The resumed agent reissues every write and creates no duplicate: each replayed receipt carries the original committed version and audit correlation id.
3. Reusing an idempotency key with different content returns `IDEMPOTENCY_CONFLICT`; omitting it returns `IDEMPOTENCY_KEY_REQUIRED`.
4. Session logs carry an `audit_parity` block reconciling input and output events, with missing and orphan ids listed explicitly.
5. A mutation sent to the read route returns HTTP 200 with `status: Rejected` and `WRITE_PERMISSION_REQUIRED`, not an HTTP error.

## 5. Change one thing and re-run

- Change `crash_after_step` in the dataset and confirm the number of replayed receipts follows it exactly.
- Kill the container between the two write phases and restart it with persistent storage to see recovery run for real.
- Start the server with `CORROBORE_MEMORY_PERMISSIONS=read,trace` and watch every mutation return `PERMISSION_DENIED`.
- Drop `CORROBORE_HTTP_AUTH_TOKEN` from the request headers and confirm 401 before any handler runs.

## Boundaries

- The incident, the pipeline and the trail are synthetic.
- The notebook runs against ephemeral storage; persistent recovery is a deployment concern covered by the database operations runbook.
- Snapshot, restore, index rebuild and object-storage export sit behind the admin token at `/v1/admin/storage/*` and are out of scope here.
- `DURABILITY_FAILED` is not reproducible on a healthy local server; it is the failure surface of the configured durability gate.

## Tear down

```bash
docker rm -f corrobore-playbook
```

---

Documentation: https://docs.corrobore.org — Source: https://github.com/Estance-Labs/corrobore