Lesson 0060 · System Architect · Module 3

Design the Deployment and Release Path

A design is only real when it ships repeatedly. The deployment pipeline, data migration, and release strategy are the last stages of the sequence — and the ones that make failures observable and recoverable.

Reading ≈ 10 minPractice ≈ 25 min

Mission tie-in: lessons 0047–0049 planned the initiative's delivery and rollout. Here you design the durable machinery — the pipeline and release path the system keeps for its whole life.

Knowledge: one artifact, many environments, designed recovery

The deployment pipeline is the automated path from committed code to production: build, test, package once, then promote that same artifact through environments. The rule is one build, many deploys — if each environment rebuilds from source, you are testing different things and shipping unproven code.

Three design moves keep deployments boring. First, separate deploy from release: a feature flag can let code arrive before behavior is exposed. Turning it off limits further exposure, but it does not undo emitted messages, money movement, or incompatible data writes. Second, treat data migrations as controlled release steps: automate and rehearse the schema sequence from lesson 0058, with an explicitly approved runbook where full automation is unsafe. Third, release in observable stages: canary, then rolling, each with entry evidence, an observation window, and a stop threshold (rollout gate, lesson 0049).

Environments are a liability as much as a luxury. Each one is software to run, data to refresh, and drift to chase. You need enough to prove the artifact — typically build, staging, production — and every additional environment must justify itself. Staging earns its cost only if it runs the same artifact, same migration, and a representative shape of data.

Finally, design for the day the release is wrong. The recovery may be code rollback, disabling exposure, forward-fix, compensation, or restoring data. A down-migration is safe only when it preserves writes made under the new schema; often it does not. Decide per release which recovery paths are valid for code, data, and external side effects, and rehearse the chosen path before the deploy rather than inventing it during an incident.

# A rollout gate as a fitness function: the deploy is not "done"
# until the evidence says so. This runs in the pipeline after canary.
def gate_passed(metrics: dict[str, float], budget: dict[str, float]) -> bool:
    for metric, threshold in budget.items():
        if metrics[metric] > threshold:
            return False
    return True


assert gate_passed(
    {"error_rate": 0.002, "p95_latency_ms": 180},
    {"error_rate": 0.01, "p95_latency_ms": 250},
)

Worked: designing the gate around the irreversible parts

The first thing the architect wrote for the returns release was not a pipeline diagram. It was a list of the things this system does that cannot be undone: it moves money, it buys carrier labels at about €0.40 each, and it sends customers email. Everything else in the design follows from that list, because those three are what a rollback cannot reach.

The third stop rule is the one that comes from the irreversibility list rather than from a dashboard template. Latency and error rate are the usual canary signals, and they would not have caught the failure this team was actually afraid of: a botched idempotency key, which produces two labels for one return without raising a single error. At 5% of 3,000 returns a day, thirty minutes of that is only a handful of duplicate labels — trivial money, and exactly the point: the cheapest place to notice is where the exposure is small. So “duplicate label rate must be zero” became an entry in the gate, measured directly, alongside the metrics everyone already had.

# The gate as data: thresholds, and the irreversible effects they protect.
GATE = {
    "error_rate": 0.01,          # fraction of requests
    "p95_latency_ms": 800.0,     # the eligibility driver from lesson 0043
    "duplicate_label_rate": 0.0, # the effect a rollback cannot undo
}


def stop_reasons(observed: dict[str, float]) -> list[str]:
    """Return every breached threshold, not just the first — the whole picture."""
    return [f"{metric}={observed[metric]} exceeds {limit}"
            for metric, limit in GATE.items()
            if observed.get(metric, 0.0) > limit]


assert stop_reasons({"error_rate": 0.002, "p95_latency_ms": 640.0,
                     "duplicate_label_rate": 0.0}) == []
assert stop_reasons({"error_rate": 0.002, "p95_latency_ms": 640.0,
                     "duplicate_label_rate": 0.004}) != []   # stop: labels bought twice

The alternative the team considered was blue/green with an instant switch — two full environments, flip the router, flip it back if anything smells wrong. It was rejected for a reason specific to this system: both colours write the same database, so the switch does not make the schema change reversible, and the schema change was the largest risk in the release. Blue/green solves for code and buys nothing against data. The expand/contract sequence from lesson 0058 is what makes the data reversible, and it does so per step rather than per deploy.

Across the pilot the shape held: 34 deploys, two canary stops (one latency, one duplicate label), and a mean time from a bad deploy to stopped exposure of about eleven minutes. That last number is the one to design for. It is not a measure of how good the code is; it is a measure of how quickly the system can admit it was wrong.

When this is the wrong move. A canary of 5% is nonsense for an internal tool with twelve users — use a named beta group, or ship and watch. A pure code change behind a flag that is already off does not need staged exposure at all; the exposure is the flag flip, and that is where the gate belongs. And a system with no data migration and no external side effects can run a much shorter pipeline honestly: the ceremony in this lesson is priced for money, labels, and email. Match the machinery to what is actually irreversible, and be willing to say out loud that a change has nothing irreversible in it.

Skill: make the path recoverable

Practice: draw your pipeline and migration sequence

Use your next change or the supplied returns release. Write pipeline stages with entry gates and exit evidence. Choose staged exposure or a flag only where it controls the identified risk, and write stop thresholds. Place the lesson 0058 migration into the sequence, then name separate recovery for code, data, and external side effects; do not assume one rollback handles all three.

In the lab and the capstone: the companion lab's phase-5 checkpoint is where the service finally has to ship. List its irreversible effects first, then design the gate around them rather than around whatever metrics are easiest to collect. The worked capstone carries the release plan in its rollout artifact and the thin production-shaped slice it starts from in the walking skeleton.

Read and watch deeper

• Continuous Delivery, Humble & Farley — ch. 5–6 and 10–13: the deployment pipeline, environments, data management, components, and release strategies.
• Accelerate, Forsgren, Humble & Kim — the delivery-performance evidence behind small batches and fast, reversible releases.
• Site Reliability Engineering, Beyer et al. — ch. 8, Release Engineering: hermetic builds, one artifact promoted, and release as an engineering discipline rather than an event.
• Related lessons: 0047, 0049, 0035; glossary deployment pipeline, feature flag.

That is the last lesson. Sixty of them have moved from reading coupling in a single function to designing a system's deployment path, and the way to find out whether the judgment transferred is to use it on something whole. The capstone asks you to produce the full artifact set for a returns-resolution initiative and then compare it against a worked strong version and a deliberately weak one. The capstone assessment has you review your own packet against the rubric before you look at either. And the exit diagnostic puts scenarios comparable to the entry diagnostic in front of you, so the change in how you answer is visible rather than assumed. Do them in that order; the self-review before the reveal is where most of the learning is.

Ask your agent-teacher to simulate the release: it should inject a staging failure, a canary regression, and a mid-migration incident, and you decide the next move each time.