A Pipeline That Fails Loudly Is a Feature
Field note on why silence is the most dangerous output a system has.
A request I've heard more than once, in various words: can we make the pipeline more resilient — so it doesn't break when a source is late? It's a reasonable-sounding ask, and it's usually pointing at the wrong fix.
Most of the time, "don't break when a source is late" really means "keep serving data even if it's stale." And serving stale data without saying so isn't resilience. It's the system choosing to lie by omission. The pipeline didn't fail — it succeeded at producing a wrong impression.
I'd rather a system fail loudly and locally: stop, raise its hand, name what's missing. Loud failure is recoverable. Someone sees the error, understands the blast radius, and fixes it. Quiet failure — the stale number that looks fresh — is the expensive kind, because nobody knows to fix anything until a decision has already been made on bad data, often in a meeting that mattered.
The reframe that helps: freshness is data, too. A number without a visible "as of" timestamp is only half a number. Make staleness visible and most of the desire to paper over it disappears, because now the consumer can decide for themselves whether yesterday's data is good enough for today's decision. That's their call to make, not the pipeline's call to make silently on their behalf.
So when someone asks for resilience, I try to ask back: resilient how? Do you want it to keep going, or do you want it to tell you the truth? Those are different systems. The first feels better in a demo. The second is the one you can actually trust on the day it matters.