It's 3am and an alert just fired. Is this a breach or a noisy bug? The teams that handle that moment well aren't calmer or smarter, they've just decided most of it in advance. The difference between a 15-minute incident and a 15-hour one is almost entirely preparation, so most of this lesson is about the work you do before anything happens.
Before the incident
Define severity levels now, while nobody's panicking:
| Sev | Definition | Example |
|---|---|---|
| 1 | Active breach, data exfiltration | Attacker has a shell |
| 2 | Confirmed exploit, limited impact | SQLi exploited, WAF now blocking |
| 3 | Suspicious, unconfirmed | Unusual outbound traffic |
| 4 | Vulnerability disclosed, no exploitation | A new CVE |
Write a lightweight playbook. It only needs to answer the questions you won't want to be figuring out live: who's on call, where you communicate, who's allowed to make a containment decision, and where the break-glass credentials are.
Set up break-glass access. Pre-approved emergency access, tested quarterly so you know it works before you need it.
During the incident
The first 15 minutes
- Acknowledge (about 2 minutes). Ack the alert, open a channel, post a one-line summary of what you know.
- Assemble (about 3 minutes). Page the on-call for the affected service, security, and an incident commander.
- Contain (about 10 minutes). Stop the bleeding. Root cause is a later problem; right now you're limiting damage.
What containment looks like depends on what you're facing:
| Scenario | Containment |
|---|---|
| Credential stuffing | Block the IPs, enable CAPTCHA, force MFA |
| Compromised account | Revoke sessions, rotate credentials |
| Data exfiltration | Isolate the service |
| Malicious dependency | Roll back |
Who does what
Clear roles stop everyone from investigating the same thing while nobody communicates:
- Incident Commander. Owns the timeline, the comms, and the decisions. Does not get into the weeds investigating.
- Tech Lead. Investigates and executes the containment.
- Comms Lead. Keeps stakeholders updated.
On a SEV-3 or SEV-4, one person wears all three hats. That's fine.
Don't destroy the evidence
Before you start cleaning up, preserve what happened: snapshot the affected volumes, export the logs, save any captures. The instinct to fix fast can wipe the very evidence you'll need to understand the breach and prove its scope.
After the incident
Run a blameless post-incident review within 48 hours, while memories are fresh: the timeline, what went well, what to improve, and action items each with an owner. Then actually track those actions. Put them in the backlog, tag them incident-followup, and review them in planning, otherwise the same incident comes back.
Practise on a calm day
Run a 45-minute tabletop once a quarter: walk through a hypothetical, rotating the scenario each time across application attacks, infrastructure compromise, supply chain, and insider threat. A few common ones worth rehearsing:
- Compromised dependency. Which services use it? Roll back or patch, then check the build logs.
- Leaked credential. Revoke immediately, rotate, audit usage, find the source of the leak.
- Ransomware. Isolate, don't pay, restore from backups, preserve forensics, involve legal if customer data is in scope.
The whole thing fits in one line: prepare before, contain first and investigate second during, and review blamelessly after. The preparation is what buys you the speed.