A platform org of about ninety engineers came to us with a familiar complaint: releases used to take a day, now they take most of a week, and nobody could point to exactly why. Headcount had roughly doubled in eighteen months. Deploy frequency hadn't moved.
We ran our usual two-week diagnostic: trace an actual change from ticket to production, write down every wait, every handoff, every manual step. Not the pipeline in the wiki — the one engineers actually route through when the wiki's version is slower than asking someone directly in Slack.
What we expected to find
Going in, the obvious suspect was the CI pipeline itself. Slow test suites, flaky infrastructure, that kind of thing. It's usually where teams look first, because it's the part they can see.
What we actually found
The pipeline was fine. Fifteen minutes, mostly test execution, nothing embarrassing. The other four days and change were almost entirely queueing: waiting for one of two people who understood a particular legacy service well enough to approve changes to it, waiting for a weekly change-advisory meeting that had quietly become mandatory for anything touching billing, waiting for a staging environment that was shared across six teams and perpetually in a broken state left by whoever used it last.
None of this showed up in any dashboard, because none of it was a system anyone was measuring. It was organizational scar tissue — reasonable decisions made individually, two years' worth of them, that added up to a four-day queue nobody had designed on purpose. Laid out by hours actually spent, the pipeline is barely visible:
What we changed
- Split the legacy-service approval bottleneck into a documented ownership boundary with a second trained approver, cutting median wait from a day and a half to under two hours.
- Replaced the blanket change-advisory meeting with an actual risk classification, so only the changes that needed a human review before shipping got one.
- Gave each team its own disposable staging environment instead of a shared one, which also — as a side effect nobody asked for — cut a class of "worked on staging, broke in prod" incidents to roughly zero.
Where it landed
Median lead time from merge to production dropped from just under five days to a little over three:
Cost per deploy — engineer-hours burned on process rather than the change itself — came down by a bit more than a third. We left the pipeline-tracing setup running so the platform team can watch the number themselves instead of taking our word for it next quarter. Deploy frequency recovered steadily as each fix landed, not in one jump:
The fix, in the end, was mostly deleting things: a meeting, a shared environment, a single point of failure in an approval chain. That's usually where it is.