THE PIPELINE

Simulated: the fleet, the stages and the traffic. 1,500 made-up requests a second, with lognormal latency around 80 ms and a 0.3% background error rate. The bug you pick changes what the canary's three servers do with their share. Real: the decision. Every half second the page runs a one-sided two-proportion z-test on the canary's errors against everyone else's, and a Mann-Whitney U test on their latencies, on exactly the requests you can see. It rolls back when either p-value falls under 0.001 — not 0.05, because it looks twice a second and repeated peeking at 0.05 would roll back good commits — and never on fewer than 400 canary requests. A slow endpoint must also move the p99 by 15%, so a statistically real but harmless shift does not page anyone. The service, commits and bugs are invented. Drag to turn it, pinch or scroll to zoom, double-tap to reset.