The alarm fired three times. Nothing got in.

What happened — three times

A CloudWatch alarm on CodeMinder's own production WAF, cm-prod-waf-high-blocks, fired three times in four weeks: August 5, August 8, and September 2. Every firing was the same class of event — a single cloud-hosted machine running a credential-harvesting wordlist against paths that don't exist on our API. Every firing reported high blocks. Not one said whether it mattered.

• Three different source IPs (all Google Cloud) — so blocking any single one wouldn't have stopped the next.
• Three different wordlists — AWS-credential-focused, then multi-cloud plus instance metadata, then PHP/CI-CD paths.
• Three alarms. Zero incidents. Every one needed a manual investigation to prove we were safe.

The triage tax

The first time, proving it was harmless took roughly six hours — overnight, ad-hoc, from scratch. Not because the answer was complex, but because the evidence was scattered across four log sources and three AWS services, and you have to know exactly how they fit. The first pass even burned time on dead ends: hunting for load-balancer access logs in an account that doesn't have that log type, chasing a coincidence between the block count and an unrelated customer issue, and querying database tables that belonged to a customer's system instead of ours.

By the third firing, the same investigation took five minutes — because someone had turned that first painful pass into a written runbook and a script.

Sit with that progression: six hours, then fifteen minutes, then five. The task never changed. What changed is that a senior engineer absorbed the cost once, by hand, and compiled the knowledge into something repeatable. The alarm didn't get smarter. A person did the work the alarm couldn't.

The threshold is unreliable, too

The blocks-over-threshold, in order, were 170, 151, 103 — the margin over the limit collapsing against a baseline of essentially zero blocked traffic:

So the alarm is simultaneously too noisy (any burst of background internet noise pages a human) and going blind (the same noise is about to slip under the line). A static threshold is the wrong instrument for this.

Then we fed it to our own pipeline

This is the exact problem CodeMinder is built for — an alert that says "something was blocked" without saying whether it mattered, and a multi-source manual hunt to find out. So we ran the September 2 event through CodeMinder itself.

The point wasn't reducing 565 records to four. It was turning them into decision-ready findings. The pipeline read the event as automated reconnaissance from a single source and named the campaign shapes:

Four labeled, prioritized findings — instead of 500+ undifferentiated blocked-request log lines and a night of manual queries.

The reality gap

Both were looking at the same event. Only one answered the question you actually have when you're paged — what is this, and do I need to care? That meaning doesn't live in any single log line. It lives in the relationship between hundreds of them — same source, same window, wordlist shapes — the things that turn hundreds of blocked requests into an understanding of what happened.

What this means for you

A threshold alarm is a smoke detector: it tells you smoke exists, not whether there's a fire or someone burned toast. That's fine — until every burst of background internet noise pages an engineer who then has to prove, by hand, that it was harmless. Two questions get collapsed into one:

• Did our controls hold? Your alarm answers this crudely, by counting.
• What was it, and did any of it matter? Your existing tools hold pieces of the answer — but a human still has to correlate the evidence across them.

The first question is cheap. The second is where senior engineering hours disappear — and it's the one you're actually asking when the pager goes off.