Two stacks, one root cause: reading an incident that lied about where it was

Context

Two production systems, built by different people at different times, with nothing obvious in common. One handled form submissions across the public sites. The other handled file uploads in an internal tool. Separate codebases, separate repositories, separate deploy pipelines.

They failed within the same minute of each other. Neither had been deployed that day.

The problem

The error surface pointed everywhere and nowhere. Users saw a form that would not submit. A different set of users saw an upload that would not complete. Two support tickets, two teams, two plausible stories — and each team, reasonably, started debugging its own stack. That is exactly how an incident like this eats an evening: everyone is busy, everyone is looking in the wrong place, and the one fact that matters is sitting in plain sight.

The fact was this: nothing had shipped. No code change to either system. When two unrelated things break together and nobody deployed, the cause is almost never in the code you are staring at. It is in something underneath that both of them touch.

How I isolate it

I do not start from the error message. I start from what the two failing things have in common, and I work down until there is exactly one shared thing left.

The reasoning is deliberately boring, because boring is what survives contact with a stressful evening:

• No code changed — so the fault is not in a change. Stop reading diffs.
• Two unrelated stacks, same symptom, same minute — so it is not a coincidence and not a single bad feature. It is something upstream of both.
• What do they share? — list it honestly: runtime, network path, storage, a queue, a credential, a dependency. Cross off everything only one of them uses. Here, the list collapsed to a single piece of shared infrastructure both quietly relied on.
• Confirm, then fix, then prevent — reproduce the failure against the shared component, repair it, and only then ask the real question: why did this reach users before it reached me?

The fix

The repair itself was small once the cause was named — it usually is. The part that mattered was the last step. This failure had been invisible until it was an outage, which means the system had no opinion about its own health at the exact point where both stacks depended on it.

So the incident did not close with a fix. It closed with alerting defined as code on that shared dependency: a check that fails loudly, owned and version-controlled like everything else, so the next time this component degrades it announces itself minutes before a user ever sees a broken form. A post-mortem that ends with a repaired symptom is a document. A post-mortem that ends with a control that would have caught it is a process.

What this is really about

The specific component is not the point, and I have deliberately not named it — it would be different at the next company anyway. The method is the point, and the method does not change:

• When something breaks, separate where the error shows up from where the fault is. They are frequently not the same place, and the error is the less reliable of the two.
• Treat “nothing changed” as information, not as the end of the investigation.
• Reduce to the single shared cause before you touch anything, so the fix is aimed rather than hopeful.
• Never let the analysis finish at the repair. It finishes at the control that makes a repeat visible — and the control ships like code, because anything that lives in someone’s memory instead of the pipeline is already decaying.

That is what I mean when I say I own incidents. Not that I am in the channel at 9pm — anyone can be in the channel. That I leave the system more observable than I found it, every time.

What I would change

I would have had the alert before the incident, not after it. The shared component was a known dependency; the fact that it had no health signal was a gap I could have found in a calm afternoon rather than a loud evening. The honest lesson here is not about debugging under pressure — I am fine under pressure. It is that the cheapest incident is the one a preventive control turns into a quiet alert nobody else ever notices.