Your job runner reports on one thing: did the job finish without throwing. It has no view into whether the business operation the job exists to perform actually happened. Those are two different questions, and most stacks only ever answer the first one — so a job can complete, report success, and accomplish nothing, and every dashboard stays green.
What we found
A background job enriched contact records by calling a third-party enrichment provider — the one that supplied the vast majority of every batch. In the logs, that provider had been failing for a while. In the run history, the job was green.
The upstream failures were right there in the logs, every time the job ran:
The reason is in the job's own error handling. The enrichment call was wrapped in fallback logic: when the primary provider failed, the code caught the error, pulled what it could from lower-tier sources — often little, sometimes nothing — logged the failure, and let the job finish. From the job runner's point of view, that's a clean run. It completed. It didn't throw. It reported success.
The question a job runner answers is "did this run without erroring." The question that matters is "did the operation actually produce what it was supposed to." When fallback logic turns an upstream failure into a clean completion, those two answers point in opposite directions — and only the reassuring one is on the dashboard.
Why nothing caught it
None of the existing monitoring paths was looking at these signals together. The job runner saw a completed job. The APM saw no thrown exception. The provider's 400s and 402s sat in the application logs, correctly recorded, but disconnected from the job's outcome. Each system was right about its own slice. Nobody was holding two slices at once — and the finding only exists in the space between them. APM watches for thrown errors and latency; log tools store the lines. Neither asks whether the outcome matched the intent — and that question only exists across layers, holding the job runner's structural "it ran" against the payload's semantic "it produced almost nothing."
400 / 402 responses from the enrichment provider, sitting in the application logsThe key wasn't detecting another error. It was connecting the provider failure to the business outcome of the job.
In the same analysis, outbound email was silently failing too — and this one carries sharper stakes. The email provider was returning a "domain not found" configuration error, so transactional messages simply weren't being delivered. Transactional email is the channel that carries password resets, receipts, and confirmations — the messages a user is actively waiting on — and it was failing with nothing erroring loudly enough to page anyone. Same shape as the enrichment job: a real operation failing under a status that read as fine.
"The job succeeded" and "the work got done" are not the same statement. Fallback logic exists to keep a failure from crashing the run — but the same catch that prevents the crash also erases the signal. Success becomes a report about the code path, not about reality.
What this means for you
Silent success is one of the most expensive failure modes there is, because nothing ever asks you to look. A job that crashes gets fixed by Tuesday. A job that succeeds while doing nothing can run for months — enriching no records, sending no email, moving no data — and the first time anyone notices is when a downstream number is inexplicably wrong.
The signals to catch it were already there: the upstream errors, the empty outputs, and the jobs that kept reporting success. Nothing was missing. The signals just weren't connected.
That's the gap CodeMinder is built to close: connecting what happened across systems to determine whether the work actually got done — and what deserves attention when it didn't.
- Audit the
try/catchblocks around your third-party SaaS calls — a caught error that lets the job finish is a silent failure waiting to happen. - Alert on batch yield — records written, profiles returned, emails delivered — not just job execution exit codes.
- Watch key-rotation and billing logs for
HTTP 400/402responses from providers; they're the earliest tell that output is about to quietly drop.