# Rollout contract

Done is not "merged", and launch is not one event. A release is a controlled experiment:
every stage has an entry condition, something to observe, an advance threshold, and a
rehearsed way back.

Source: lesson 0049 · Test, Observe, and Roll Out Safely.

---

## Blank — risk-to-test map

Test according to risk, not a universal pyramid quota.

| Risk or driver | Test type | What it proves | Owner | Evidence location |
| --- | --- | --- | --- | --- |
| `[from the driver table or risk list]` | example / unit · integration / contract · end-to-end · quality (load, security, recovery) · exploratory charter · drill | `[the specific claim]` | `[name]` | `[CI job, dashboard, drill report]` |

---

## Blank — telemetry map

Every measure needs an owner and a decision it can change. A green error dashboard does not
prove that returns resolve faster.

| Layer | Measure | Segment by | Owner | Decision it can change |
| --- | --- | --- | --- | --- |
| Outcome | `[the thing the initiative exists to move]` | cohort | | |
| Guardrail | `[what must not degrade]` | cohort | | |
| System health | `[rate, latency, errors, saturation]` | | | |
| Business state | `[distribution and age of entities in each state]` | | | |
| Dependency | `[external system health as we experience it]` | | | |
| Data quality / privacy | `[completeness, deletion job success]` | | | |
| Manual load | `[queue depth, support contacts]` | | | |

Segment enough to expose who is harmed, while protecting privacy and sample size.

---

## Blank — exposure stages

One row per stage. No stage without all five columns.

| Stage | Entry | Observe (measures · segments · minimum volume and duration) | Advance (threshold · decider) | Pause / reverse (trigger · authority · action · recovery time · communication) | Data and side effects |
| --- | --- | --- | --- | --- | --- |

**Separate deployment from release.** Deploy dormant behavior behind a short-lived flag,
validate it internally, then expose cohorts gradually. The flag needs an owner, a removal
date, a safe off-state, and tests for both configurations.

**Define rollback before rollout.** Who decides, what signal triggers it, how long it takes,
what happens to data already written, and how customers in flight recover. A code rollback
cannot undo an email sent, a refund issued, or a schema contracted. Some effects need
roll-forward or compensation.

### Operational readiness — part of "done"

- [ ] Dashboards exist and someone has looked at them when nothing was wrong.
- [ ] Alerts are actionable: each names a threshold, an owner, and a runbook step.
- [ ] Structured events carry a correlation id across every boundary, including the worker.
- [ ] Runbook covers the known failure states, with the disable path first.
- [ ] Support brief written by someone who will answer the contacts.
- [ ] On-call and steady-state service owner named — a person, not a team alias.
- [ ] Access, capacity assumption, and data recovery documented.
- [ ] The disable path has been exercised, not just written.
- [ ] Every temporary flag has a deletion owner and date.

Run a 30-minute failure drill before the first customer stage: the dependency returns
timeouts, the worker backlog grows, one request is duplicated. Can the team detect, diagnose,
limit, recover, and explain?

---

## Completed — returns resolution pilot

### Risk-to-test map

| Risk or driver | Test | What it proves | Where it runs | Owner |
| --- | --- | --- | --- | --- |
| A1 · policy correctness | The 200 sampled agent decisions as table-driven tests against `assess` | Every rule and refusal reason matches what an expert would have decided | Unit, every build | Sam |
| D2 · 800 ms p95 | Load test at 40 rps against a production-shaped dataset | The decision path holds its budget under peak load | Pipeline, nightly | Sam |
| D1 · carrier outage | Recovery drill: the carrier sandbox forced to 100% timeouts for 30 minutes, then released | Requests are still accepted, and 99% of the backlog drains within 15 minutes | Staging, before every stage | Lea |
| Duplicate labels | A contract test with an induced timeout and retry, plus a daily production reconciliation against the carrier's own label list | Exactly one carrier label per return, in test *and* in production | Contract suite + production job | Lea |
| D3 · cross-customer disclosure | Negative authorization tests on every read path; a log-scanning fitness function that fails the build on an unredacted order id | No identifier leaks through a response, an error, or a log line | Contract + build | Ola |
| D4 · one-day policy release | The import-rule fitness function; measured lead time on the last three policy changes | The seam is real, and the claim about it is measured rather than asserted | Build + monthly report | Dana |
| Accessibility guardrail | Automated checks in the pipeline, plus manual keyboard and screen-reader passes on the four critical flows, plus one external audit | WCAG 2.2 AA on the flows that matter. Automation alone would have missed two of the three findings | Pipeline + manual + external | Marco |
| The unimagined | Two exploratory charters per slice: "try to get a label you should not have" and "try to make the system contradict itself about a return" | Behavior the examples did not imagine. The second charter found the receipt-reversal case that became an acceptance example | Timeboxed, per slice | Whole team |
| D5 · reversal | The reversal rehearsal, below | The disable path works when a human uses it under time pressure | Staging, before every stage | Nils |

### Telemetry map

| Measure | Kind | Segmented by | Owner | Decision it can change |
| --- | --- | --- | --- | --- |
| Median and p90 time from submission to decision communicated | Outcome | exposed vs. control; device; decision outcome | Priya | Continue / change / stop |
| Support-contact rate within 7 days of submission | Outcome | exposed vs. control; accepted / referred / refused | Priya | Continue / change / stop; slice order |
| Fraud loss per 1,000 eligible returns | Guardrail | exposed vs. control | Risk | Stop — Risk's unilateral authority |
| Manual review queue depth and daily arrivals | Guardrail | referral cause | Nils | Pause the exposure ramp |
| Referral age; oldest unanswered referral | Guardrail | cause | Nils | Roster change; pause the ramp |
| Decision-outcome mix by reason code | Business state | reason code; policy version | Anya | Roll back a policy change |
| Eligibility decision p95 | Health, tied to D2 | — | Sam | Block the next stage |
| Carrier latency, error rate, label backlog age | Health, tied to D1 and D5 | carrier error code | Lea | Disable label attempts |
| Age of the oldest accepted return with no terminal state | Data integrity | state | Lea | Escalate to Dana after two days |
| Records older than 30 days in the returns schema | Privacy | — | Ola | Halt exposure — it is a compliance breach, not a bug |
| Accessibility check failures on critical flows | Guardrail | flow | Marco | Block the release |

Baselines the measures are read against: eligible domestic demand about 130 requests/day,
rising to about 160/day from mid-November. Decision mix from the A1 sample and the first
exposed week: 63% accepted, 26% referred, 11% refused.

### Exposure stages

| Stage | Cohort | Entry | Observe (minimum) | Advance when | Pause or reverse when · authority |
| --- | --- | --- | --- | --- | --- |
| 1 | Team only | All tests pass; a support path exists; the runbook has a first draft | All flows, 2 days | No critical security or accessibility defect open | Any critical defect · anyone |
| 2 | Staff accounts | Runbook exercised; alerts route to a human; **privacy sign-off**; reversal rehearsal passed | Errors, retries, comprehension; 50 real cases | Zero duplicate labels; decision p95 within 800 ms | Any duplicate label · Dana |
| 3 | **1% of eligible demand**, selected by a stable hash of the customer id, with a **10% holdback control** that stays unexposed for the whole pilot | Support briefed; reversal rehearsed again; the control cohort instrumented | Outcome and guardrails; 3 days and at least 120 exposed requests | Median resolution improved against control; fraud not worse; queue under 50/day | Fraud loss above control by more than 1.0 per 1,000 · **Risk**. Contact rate worse than control · Priya |
| 4 | 10%, then 50% | The previous gate signed off by Dana and Nils | Capacity, referral age, manual queue, and segments by device and decision outcome; 3 days each | All stage-3 thresholds still hold at the larger n | Referral SLA missed for 4 hours · Nils. Queue above 45/day · Nils |
| 5 | 100% of eligible demand, minus the holdback, released only after the outcome has held for two consecutive weeks | A steady-state owner is named and has accepted the runbook | Continuous; weekly review | Outcome and guardrails hold for two consecutive weeks | Any guardrail breach · that guardrail's owner |

**The holdback is the point of stage 3.** "It improved after we launched" is not a finding.
A control cohort that stays unexposed for the whole pilot is what turns the outcome numbers
into evidence rather than into a coincidence with a date on it.

**Data and side effects at every reversal.** Disabling exposure stops *new* returns entering
the new path; returns already in flight continue, because abandoning them would strand
customers mid-journey. Labels already bought are not voided. Refunds already requested are not
recalled. Reversal is therefore "stop the intake", never "undo the week", and the contract says
so rather than leaving the team to discover it at 2 a.m.

### Reversal rehearsal — 30 minutes, before every stage

| Field | Entry |
| --- | --- |
| Scenario | The carrier starts returning timeouts at 09:40 on a weekday. Label errors cross 2% within four minutes. The on-call engineer is someone who did not build the worker. |
| What must be demonstrated | Detect (the alert fires and names the carrier) · diagnose (the runbook link resolves the carrier's error code) · limit (disable label attempts) · recover (re-enable; the backlog drains) · explain (a support message the agents can actually send). |
| **First rehearsal, 18 September** | The disable flag did not do what everyone believed. It was checked when a *request* created an intent, so it correctly stopped new label attempts — but the worker read the flag once at start-up and kept retrying the 63 intents already in its backlog against a failing carrier. The stopwatch said "disabled in 3 minutes"; the carrier's dashboard said we were still calling it eleven minutes later. |
| Fix | The flag is read inside the worker loop, once per intent, and a test asserts that the worker stops within one iteration of the flag flipping. Added to D5's evidence line and written into the acceptance examples as an explicit expectation. |
| **Second rehearsal, 22 September** | Disable worked in 90 seconds. The gap was the support message: nobody could say what a customer with a pending label should be told. It was written, put in the runbook, and is now part of the stage-2 entry condition. |
| Why every stage, not once | The disable path is code, and code changes. A rehearsal that happened in September proves nothing about the flag in November. |

An untested control is a belief. The rehearsal cost thirty minutes and found a defect that
would otherwise have been discovered at 02:00 during a real outage.

### Operational readiness at the staff gate

- [x] Dashboards exist and someone has looked at them when nothing was wrong.
- [x] Every alert names a threshold, an owner, and a runbook step.
- [x] Structured events carry a correlation id across every boundary — **fixed in week 2**, when the walking skeleton showed the worker did not receive one.
- [x] The runbook covers the known failure states, with the disable path first.
- [x] A support brief written by Nils, who answers the contacts.
- [x] On-call named, and the steady-state owner identified before stage 5.
- [x] Access, capacity assumption, and data recovery documented.
- [x] The disable path exercised twice, not merely written.
- [x] The release flag has a deletion owner and date.

**Traces to:** every test row cites a driver from `architecture-driver-table.md`; the stage-3
thresholds are the guardrails in `initiative-frame.md`; the disable flag and the daily
reconciliation are follow-up work from ADR 0003; assumption A6 in `assumption-map.md` exists
here as a stop threshold rather than as a test, which was a deliberate choice.