You can have perfect logs and still miss every attack, because logs only matter if something is watching them and telling a human when it counts. The hard part of alerting isn't generating alerts, it's generating ones people still trust at 3am. Cross roughly 20% false positives and the team stops reading them, at which point your monitoring is worse than nothing, because it costs effort and catches the real one too late. So the rule that governs everything here: every alert that pages someone should demand an action.
What's worth watching
Three layers, because attacks show up differently in each:
Application. Spikes in 401s and 403s (credential stuffing), spikes in 500s (someone exercising an exploit), unusual response volume (data exfiltration), logins from a new geography.
Infrastructure. Outbound traffic to unfamiliar IPs, unexpected processes, an IAM role being assumed from somewhere it never is.
Business logic. A surge in gift-card redemptions, mass account creation, a spike in password resets. These are invisible at the network layer and obvious at the business layer.
Match severity to response
An alert's severity is a promise about how fast someone reacts, so define it that way:
| Sev | Response | Example |
|---|---|---|
| P1 | Immediate | Active breach, RCE |
| P2 | Within the hour | Credential stuffing |
| P3 | Within the day | An unusual login |
| P4 | Next triage | A single failed login |
Make alerts actionable
Every alert should carry enough context to start investigating without going back to the raw logs, plus a link to the runbook and to the relevant logs. An alert that just says "anomaly detected" sends someone digging before they've even confirmed it's real.
Cut the noise relentlessly
Noise is the thing that kills a monitoring programme, so attack it directly:
- Deduplicate. Fifty identical alerts become one alert with a count of fifty.
- Correlate. A single 404 is nothing. Five hundred distinct 404s against
/admin/*from one IP is reconnaissance. - Baseline. Compare against normal as a percentage, not a raw count, so a busy Tuesday doesn't page anyone.
Write the runbook
Every P1 and P2 should link to a runbook that answers four questions: what does this alert mean, how do I investigate it, how do I contain it, and who do I contact? The runbook is what lets the person who didn't write the detection still handle it at 3am.
Tune it forever
Alerting is never finished. Track the outcome of every alert, review weekly, tune the thresholds, and retire rules that have gone stale. After any incident, ask the most useful question in this whole discipline: what alert would have caught this sooner? Then go build it.