# Phase 3 — reference solution Code in this directory: `p3_expand_contract.py`, `p3_idempotency.py`, `p3_tolerant_reader.py`. Verified by `verify_phase_3.py` (19 tests). Every exercise in this phase has more than one defensible answer. The deliverable is the decision and the cost you accepted. --- ## 3.1 Build or buy the carrier **The decision: buy, and own the market rule.** The seam is `../phase-1/p1_carrier.py`. **The ledger.** Building a label service means carrier contracts, rate negotiation, customs paperwork for CA and MX, a document renderer, and a team that keeps it running — call it four engineer-months to a first label and one engineer permanently. Buying costs an integration (two weeks), a per-label fee, and an outage you do not control. At the pilot's volume the fee is noise and the build never pays back; the break-even is at a volume you will not see for years, and by then you would buy a different vendor rather than build. **What the port does and does not buy.** It buys testable outage behaviour, a market rule you own, and a switching cost measured in one adapter. It does not buy an actual second carrier: a real swap still means a new contract, new rate cards, new tracking-number formats in every email template, and a support team retrained. Do not let the abstraction masquerade as a migration plan. **Where the market rule goes.** `SUPPORTED_COUNTRIES` is a product decision about which markets you serve, and it currently lives in the vendor's module — so swapping carriers would silently change the product. In the reference it moves to `Settings.served_countries`, and the adapter's `DestinationNotServed` means "this carrier cannot", which is a different sentence from "we do not". --- ## 3.2 Abstraction or duplication Three candidates, three different answers: 1. **Validation** — false duplication. Two copies, one purpose, and they disagree. Unify (exercise 1.10). 2. **The five refund branches** — unify, because the *reason* is the same, not because the code looks the same. `p1_refund_policy.py`. 3. **Five order-system calls per submit** — leave it, and write it down. **The arithmetic for (3).** Five calls to an in-process fake cost nothing. Five calls to a service with a 120 ms p99 cost 600 ms out of a four-hour resolution target — 0.004% of the budget. The cost is not latency, it is that you are five times as likely to hit their outage. Fix it with a per-request cache when the order system's availability becomes your availability problem; the trigger to watch is their error rate, not your latency. **A second valid solution.** A request-scoped cache is four lines and is defensible now. Do it if you like — but the cache must not outlive the request, or you have created a staleness bug that exercise 3.4 will have to reason about. --- ## 3.3 Synchronous or asynchronous **Per step, not per system.** | Step | Side | Why | |---|---|---| | Eligibility | sync | The customer needs the answer to continue. | | Fraud | sync | It decides which answer they get. | | Carrier label | async | The customer does not need the PDF in the response, and the carrier is the least reliable dependency. | | Refund | async, triggered later | It happens after the item is received. | | Notification | async | Nobody should wait for an email gateway. | **The worker's four defects and their fixes.** - *The retry unit is the batch.* One delivery record per channel, each with its own state, so a failed SMS retries the SMS. The lab's version re-sends the email that already arrived. - *The row says `sent` before the send.* Move the update after the transport returns. Write `attempting` first if you want to detect crashes mid-send. - *The queue is in memory.* An outbox table plus a poller. The lab loses everything queued on restart, and `test_the_queue_lives_in_a_module_global` proves it. - *Exhausted retries vanish.* A `dead_letters` table with the message, the attempt count, the last error and the correlation id, plus a metric on its depth. Four columns, and it converts "we lost some emails" into a number. **Consequence accepted.** Making the carrier call async means the response no longer contains a tracking number, and the customer web app currently reads one — `test_the_app_needs_tracking_and_label_url_on_the_submit_response`. That is a client-visible change and needs exercise 3.6's versioning. It is the good kind of coupling discovery: the test told you before the other team did. --- ## 3.4 Idempotency → `p3_idempotency.py` Look up the key, return the recorded answer, otherwise perform and record. The record is written in the same transaction as the effect, or you have moved the race rather than closed it. **The design decisions inside it.** - *The key comes from the client.* A server-derived key (`refund:{return_id}`) makes a second, legitimate refund impossible — which is a product decision, not a technical one. Ask whether partial refunds exist before you choose. - *Reusing a key for a different request is a conflict, not a silent replay.* `IdempotencyConflict` exists so that a client bug is visible instead of returning them somebody else's answer. - *The key needs a lifetime.* The reference does not expire them; a real deployment picks 24 hours or 30 days and says which, because "forever" is a table that grows without bound. **The availability side.** Refusing to act when you cannot verify uniqueness is the consistent choice, and it makes refunds unavailable during a database blip. The reference takes it: a duplicate refund is a real financial loss and a support escalation, and a few minutes of refund unavailability is not. For label purchases the reference leans the other way — a duplicate label costs $7.80 and annoys nobody. --- ## 3.5 Expand and contract → `p3_expand_contract.py` Four separately-runnable steps with a version marker, and the test that matters is `TheMigrationIsSafeAtEveryStep`: finance's unchanged job is run after each step and must still be right. | Step | What deploys | Reversible by | |---|---|---| | 1 expand | a schema change only | dropping the column | | 2 backfill | a job | doing nothing; it is idempotent | | 3 switch | a code deploy | deploying the previous build | | 4 contract | a schema change | a restore — the point of no return | **Details that matter.** - `to_minor` uses `Decimal(str(amount))`, never `float * 100`. The test pins `to_minor(76.49) == 7649` against the lab's `7648`. - The backfill is `WHERE refund_amount_minor IS NULL`, batched, committing per batch, and returns a count so a scheduler can loop until zero. It is safe to run twice and safe to kill halfway. - `step_3_switch` refuses to run while any row is unconverted. A migration step that can be run out of order will be. - Step 4 is not yours to take alone. Write the message to finance: which column, which release, what their query becomes, and the date after which the old column is gone. **Consequence accepted.** Four deploys instead of one, a period where two columns hold the same fact and can disagree, and a coordination dependency on a team that does not report to you. **A second valid solution — when you would choose it.** One deploy inside a maintenance window. It is genuinely cheaper if you can get the window: no dual writes, no backfill job, no waiting on finance. Choose it when the system has a quiet period you are allowed to use, when the data volume fits inside it, and when a failed migration can be restored inside the window too. For a pilot on 30 September with 50 cases a day, a 04:00 window is a defensible answer — and you should still write the four steps, because you will need them at ten times the volume. --- ## 3.6 Version the published contract The envelope grows a `version`, and the response is produced from a declared type rather than a dict literal in a handler. **The status rename, compatibly.** Exercise 1.9 collapses `APPROVED` and `approved`. The support console branches on the raw string, so a rename breaks their release. The compatible path is expand/contract applied to an API: emit `status` unchanged *and* a canonical `state` for one release window, let consumers migrate, then drop `status`. Same four steps, different substrate. **Version the envelope, not the field.** A version per field is a configuration graveyard and nobody can say what "v3" means. **A second valid solution — when you would choose it.** Coordinate instead: one flag day, everyone deploys together. Conway's law decides this more than taste. If those three teams are in your standup and share a release train, coordination is cheaper than two code paths and a deprecation calendar you must actually run. If they are in another timezone with their own roadmap, version. --- ## 3.7 The tolerant reader → `p3_tolerant_reader.py` Every field the system depends on is named once, in `parse_order`. Unknown keys are ignored on purpose. A missing optional key degrades to a documented default *and logs that it did*. A missing required key raises `OrderContractBroken` with the field's dotted path. **The important half is the logging.** "Tolerant" without a log is silent, and a default that quietly kicks in is a bug that takes months to surface — the customer who stops getting SMS because the order team moved `phone`. **Consequence accepted.** You will not notice upstream changes as fast as a strict parser would. That is the trade: strict parsing makes you break loudly and often, which is right when the upstream team is yours and wrong when it is not. **Unit prices come back as strings.** `Money.parse` refuses floats, so the reader hands over `"89.99"` rather than `89.99`. That is the boundary doing its job. --- ## 3.8 The debt ledger A worked example. Yours will differ; what matters is that every row has an owner and a trigger, not a hope. | Debt | Monthly cost | Repayment trigger | Owner | |---|---|---|---| | Five order-system calls per submit | ~0 today; 5× exposure to their outages | their error rate over 0.5%, or our p99 over 800 ms | returns team | | No API version; three consumers coordinated by hand | ~2 hours per release of scheduling | a fourth consumer, or the first consumer outside our release train | returns team | | Notification outbox in memory | est. 3–5 lost messages per deploy at pilot volume | volume over 500/day, or the first customer complaint about a missing email | returns team | **Not on the ledger, and why.** The duplicate refund (3.4) and the cent lost to float arithmetic (1.8) both move real money incorrectly. Correctness debt in a payments path is not debt, it is a defect, and it goes in the sprint rather than the ledger.