Incident Response for Software Teams: Name a Commander in 15 Minutes

Incident Response for Software Teams: Name a Commander in 15 Minutes

An effective production incident response restores service, stops user harm, preserves evidence, and starts a blameless learning cycle. Right now, that means declaring the incident, naming an Incident Commander and a Communications Lead, moving the conversation to a workspace that does not depend on production, and keeping a living timeline. This approach follows Google SRE and NIST guidance, and Atlassian’s public reviews.

Key Takeaways

Effective incident response restores service quickly, minimizes harm, and emphasizes blameless learning by defining roles, communication, and structured postmortem analysis.

Table of Contents

• What should you do in the first 10 to 15 minutes?
• Who owns what: the Incident Commander, Ops Lead, and Communications Lead
• How do you decide when to declare, and how do you triage?
• What belongs in runbooks and playbooks?
• How should you communicate during an incident?
• How do blameless postmortems turn incidents into fixes?
• How do you practice and build the right tooling?
• How do you keep on-call teams ready?
• How does incident response fit into the development lifecycle?
• Which metrics tell you whether response is improving?
• How do you handle a security incident on a software team?
• How should incident response work with customer support?
• How should you classify severity and set priorities?
• How does incident response scale across team sizes?
• How we help teams get ready for the next incident
• FAQ
• Sources

What should you do in the first 10 to 15 minutes?

The opening moments of an incident can shape whether it stays manageable or turns into an all-hands scramble. Work through these steps in order, and resist the urge to start fixing before someone owns the response.

1. Declare the incident when your criteria are met, for example, when another team is needed or customers can see the outage.
2. Name an Incident Commander and a Communications Lead, out loud, in the channel.
3. Move the conversation to a chat room, document, and tracker that do not depend on the failing service.
4. Triage by symptom: what are users experiencing, and which SLO is burning? Prioritize mitigation that cuts user impact over finding the root cause.
5. Capture evidence (logs, graphs, recent deploys) in a living incident document, then send the first stakeholder update with a time for the next one.

Pro Tip: If you are unsure whether to declare, declare. Closing a small incident early is easier than managing a late escalation.

Who owns what: the Incident Commander, Ops Lead, and Communications Lead

Confusion kills more incident timelines than bad code does. Google’s SRE book organizes response around the “three Cs”: Coordinate, Communicate, and Control. Each C maps to a role, and each role has one job.

• Incident Commander: holds the high-level state of the incident, delegates work, and removes roadblocks. The commander does not dive into the code. The moment the commander starts debugging, nobody is watching the whole board.
• Operations Lead: directs technical mitigation and is the only person who changes production. One hand on the controls keeps changes from colliding.
• Communications Lead: sends stakeholder updates and handles incoming questions so responders stay focused.

Assign these roles by knowledge and context, not by the org chart. A senior director who has never touched the failing service should not command its incident just because of title. Delegation matters more than reporting lines here.

Handoffs need to be explicit. When the commander changes, say so verbally and announce it to everyone in the channel, so nobody is guessing who is in charge. If the incident splits into separate problems, spin up sub-incidents with their own leads rather than stuffing everything into one thread.

How do you decide when to declare, and how do you triage?

Declare on objective criteria, not on gut feel or on who is willing to raise the alarm. Google SRE advises setting those criteria in advance and leaning toward declaring early, because closing a minor incident quickly is cheaper than untangling a late escalation. Good triggers include a customer-visible outage or a need for another team.

Alert on symptoms, not internals. Google SRE guidance recommends alerts aligned to SLOs, because they map directly to what users feel. A CPU spike may be nothing; a burning error budget never is. If you need a deeper look at making this signal reliable, our guide to application observability covers it.

Once declared, run a quick triage rubric:

• Blast radius: how many users or systems are affected?
• Affected functionality: is checkout broken, or is a settings page slow?
• Customer segment: are paying enterprise accounts hit first?
• Time to fix: is there a known rollback, or are you in unknown territory?

Escalate when the rubric says the problem is bigger than the people in the room. Pull in additional teams early, and open a dedicated restoration stream when recovery work (not diagnosis) becomes the long pole.

What belongs in runbooks and playbooks?

Treat them as two different tools. A runbook is a procedure for a known, routine task, such as restarting a worker pool. A playbook is scenario-based: it guides mitigation and investigation when a specific class of failure shows up. NIST’s community profile calls for documented procedures and playbooks as core parts of incident response planning.

A usable playbook has these elements:

• Symptoms that identify the scenario, so responders know they are in the right document
• Quick checks that help confirm or rule out the cause
• Mitigation steps, including how to roll back
• Verification steps that prove the fix worked
• Named contacts and owners

Pre-tested matters as much as written. Atlassian’s post-incident review of its April 2022 outage stresses documented, pre-tested runbooks and playbooks, audited regularly and updated after every incident. Keep them in version control so changes are reviewed and dated. For a concrete template, this support runbook design guide offers structure you can adapt, and automation can collect logs and run diagnostics so responders arrive to evidence, not an empty page.

Pro Tip: Run each playbook against a staging failure at least once. A step that reads well but fails in practice is worse than no step at all.

How should you communicate during an incident?

Internal communication is often harder than the technical fix. Atlassian’s review notes that pre-built communications templates keep stakeholder inquiries from pulling responders away from mitigation. One person, the Communications Lead, owns that stream and filters questions before they reach the people doing the work.

Match cadence to audience:

• Responders: continuous, inside the incident channel
• Executives: a short update at agreed intervals, plus any change in severity
• Support teams: early and often, so they can brief customers
• Public: a status page entry as soon as customers are affected, in plain language

Use a compact template every time: what happened, scope and impact, mitigation steps taken, your confidence or uncertainty, and the time of the next update. Admitting uncertainty builds trust faster than false precision.

Pre-built templates earn their keep here, as the same Atlassian review shows: internal communication during a major incident can be harder than the technical fix itself. For more on keeping stakeholders aligned before things break, see our piece on stakeholder management and the cost of misalignment.

How do blameless postmortems turn incidents into fixes?

Write a postmortem whenever an incident meets your declared criteria, and write it so nobody is afraid to contribute. Google’s postmortem culture chapter argues that the most important shift is from asking “who” to asking “what.” Blameless writing focuses on how systems and processes allowed the failure, which is what makes people report honestly.

A solid postmortem process includes:

• A timeline built from the living incident document
• Contributing factors described as system and process conditions, not personal mistakes
• Action items with one owner, a priority, a tracking ticket, and a closure target
• Review by other teams, since the same weakness often hides elsewhere
• Aggregation across incidents to expose systemic trends

Then feed the action items into the same backlog as feature work. A postmortem that produces a document but no shipped change is theater.

How do you practice and build the right tooling?

Response skills fade without use. Run tabletop exercises and live drills, and rotate the commander and communications roles so more people gain experience before a real outage tests them. Google SRE lists regular practice, including simulated incidents, as part of preparation.

• Keep incident documents, chat channels, and trackers independent of the production system, so they stay reachable during an outage.
• Automate evidence collection and safe, routine mitigations to free responders for diagnosis.
• For large incidents, plan parallel restoration and validation streams with a staffing roster.

Google’s workbook describes large-scale incidents that needed dedicated restoration streams and automation for batching restores. Safe mitigations also pair well with deployment practices; our guide to zero downtime deployments covers patterns that make rollbacks painless.

Pro Tip: Store your incident workspace credentials and links somewhere that does not require the failing system to open them.

How do you keep on-call teams ready?

On-call readiness is built before the pager goes off. Preparation, in Google’s incident management guide, means reliable alerting, well-defined on-call processes, playbooks, and regular practice. A responder who has never seen a real incident is learning the process and the outage at the same moment, which is a poor time to learn anything.

Start with the basics. Every on-call engineer should know how to declare an incident, who the escalation contacts are, and where the playbooks live. Shadow rotations help: new engineers sit with an experienced responder for a few shifts before carrying the pager alone. Keep rotations sustainable, because tired responders make worse decisions, and a rotation that burns people out will eventually lose them.

Then practice. Tabletop sessions walk a team through a scenario on paper. A scenario-based exercise can take this further: a facilitator plays the failing system and reveals symptoms as the on-call engineer works through the problem aloud. Nothing is at risk, and the whole team sees how a colleague reasons under pressure. For incidents that need many people at once, practice a war room too: a shared channel or call with the commander in charge, rather than parallel threads.

Treat every real incident as training material. Replaying the timeline in a team meeting turns one team’s bad night into the whole team’s experience.

How does incident response fit into the development lifecycle?

Incident response is not a separate discipline that starts when code ships. It feeds back into planning, building, testing, and releasing. The strongest teams treat each incident as a requirements document written by production, which is why DevSecOps practices pull operations and security concerns earlier in the lifecycle rather than bolting them on afterward.

In practice, that means a few habits:

• Postmortem action items go into the same backlog as features and get prioritized against them.
• New services ship with alerts, dashboards, and a draft playbook, not as a follow-up task.
• Release changes are traceable, so responders can ask “what shipped recently?” within seconds. Our overview of release management explains how disciplined shipping reduces the surface area for incidents.
• CI/CD pipelines include automated rollback paths that the response team has actually tested.
• Security checks in the pipeline catch issues before they become incidents.

The AWS Well-Architected guidance supports the same loop: document processes centrally, use runbooks for known problems, and file recurring problems into a tracking system so systemic issues get prioritized and fixed. The point is closing the loop. An incident that teaches the team something should change how the next release gets built, tested, or deployed.

Which metrics tell you whether response is improving?

Pick a small set of numbers and watch their trend, not their absolute value. Useful measures include time to detect, time to acknowledge, time to mitigate, and time to fully resolve. Add incident frequency by severity, and the share of postmortem action items closed on schedule. Each answers a different question: are we noticing problems fast, are we acting fast, and are we actually learning?

Be careful with averages. A marathon outage can hide many quick recoveries, so look at medians and the long tail separately. Also, mitigation time usually matters more than resolution time, because users care when the pain stops, not when the root cause is closed out.

Pair those measures with SLO data. The Google Cloud operational excellence guidance recommends comprehensive monitoring, a knowledge base, and a centralized incident management system to improve resolution times and learning. If your tooling records declaration, role assignment, and update times automatically, most of these metrics fall out for free, without anyone filling out forms mid-crisis.

Finally, watch for gaming. If you reward a low incident count, people stop declaring incidents. Reward fast declaration and completed follow-up instead. A rising incident count with a falling time-to-mitigate is often a sign of a healthier culture, not a sicker system.

How do you handle a security incident on a software team?

A security incident follows the same skeleton as any other: declare, assign roles, communicate, contain, learn. What changes is the care around evidence and disclosure. NIST’s community profile stresses clear policies, defined roles and authorities, and documented playbooks, and notes that responsibilities span many internal and external parties. For a software team, that means deciding in advance who talks to legal, who talks to customers, and who authorizes shutting a system down.

Preserve evidence before you clean up. Snapshot affected hosts, keep logs, and record every action in the incident timeline. Rushing to restore service can destroy exactly the information you need to understand what happened.

Recovery deserves special care when data may be altered. NIST’s data integrity project emphasizes logging, corruption testing, determining the last known good state, and regular backups so you can recover to a trustworthy point. Restoring from a backup you have not validated can reintroduce the problem.

Keep the security incident channel tight. Limit access to people who need it, because details of an active compromise are themselves sensitive. Afterward, run the same blameless review as any other incident, and feed the findings into your build pipeline and access controls.

How should incident response work with customer support?

Support teams hear about incidents from customers before engineers do, and they field the angriest messages during one. Treat them as a primary partner in the response, not an afterthought.

Give support an early, direct line to the Communications Lead. When an incident is declared, support should receive a short brief: what is broken, who is affected, what customers should be told, and when the next update lands. A shared status page, written in plain language, lets support point customers to one source instead of improvising answers.

Information also needs to flow the other direction. Support sees symptom patterns that dashboards miss, such as a spike of tickets from one region or one plan tier. Route those reports into the incident channel through a single liaison, so responders get the signal without a flood of individual messages. That mirrors the Communications Lead’s job of protecting responders from interruptions.

After resolution, support closes the loop with customers. Google’s workbook describes validated contact lists for customers as part of scaling large incident recovery, a reminder that knowing whom to reach, and how, is worth preparing before you need it. Afterward, include support in the postmortem so their view of customer impact shapes the action items.

How should you classify severity and set priorities?

Severity levels give everyone a shared vocabulary, so a single word can trigger the right response. Keep the scale short and define each level by user impact, not by technical cause. A reasonable shape looks like this:

• Critical: a core function is down or data is at risk for many users. Full response, executive notice, public status update.
• High: a major feature is degraded or a significant customer segment is affected. Dedicated team, regular updates.
• Moderate: limited impact or a workaround exists. Handled in working hours with a named owner.
• Low: minor issue, tracked as normal work.

Assign severity using the triage rubric: blast radius, affected functionality, customer segment, and time to fix. Write down who may change the level, and make it easy to raise. Downgrading later is cheap; discovering you under-called an incident is not.

Priority is a separate question from severity. Severity says how bad the impact is; priority says what gets engineering attention first when several things compete. A moderate issue affecting your largest customer may outrank a high-severity issue affecting an internal tool. Write those tie-breakers into your process before the pressure arrives. AWS guidance similarly advises documenting your process centrally so responders are not inventing rules mid-incident.

How does incident response scale across team sizes?

The principles hold across team sizes; the machinery changes. A small team may have one person wearing the commander and communications hats, a shared document, and a short list of playbooks. That is fine. What matters is that roles are named out loud, even when one person holds two of them, and that the commander does not also run the fix.

As teams grow, specialize. Separate the Communications Lead from the Operations Lead, add a scribe to keep the timeline, and build a rotation of trained commanders so the same people are not exhausted by every event. Larger organizations also need sub-incident structure, where a complex outage splits into workstreams with their own leads reporting to one commander.

Project complexity shifts the load too. A monolith with one failure domain needs fewer coordination points than a mesh of services owned by many teams. For the second case, invest in clear service ownership, an up-to-date contact directory, and escalation paths that work during an outage.

At the extreme, Google’s workbook shows that very large incidents call for dedicated restoration streams and automation to scale manual validation. Whatever your size, scale the process to match the failure you can imagine, then practice it. Start with a lightweight version and add structure only where drills expose a gap.

How we help teams get ready for the next incident

Most teams know what good incident response looks like and lack the time to build it. We close that gap through our software consulting and delivery support, working alongside your engineers rather than handing over a binder.

What we can do with you:

• Design runbooks and playbooks for your actual services
• Improve DevOps and CI/CD so rollbacks and diagnostics are dependable
• Run simulated incident drills and tabletop exercises
• Set up a post-incident tracking workflow so action items get shipped
• Provide retained support after the handoff

You leave with documented playbooks, automated diagnostics wired into your tooling, a tracking workflow, and a handoff plan your team owns. If you want to start with an incident readiness workshop, contact us and tell us what keeps your pager busy.

FAQ

What is the first thing to do during a production incident?

Declare the incident and assign an Incident Commander. Naming one owner for coordination stops people from working at cross purposes. Declare early on objective criteria, as Google SRE recommends.

What are the key roles in software incident response?

The three core roles are the Incident Commander, who holds the high-level state, the Operations Lead, who directs technical mitigation, and the Communications Lead, who updates stakeholders. Google SRE assigns them by knowledge and context, not by the org chart.

What is a blameless postmortem?

A blameless postmortem reviews an incident by looking at how systems and processes contributed, not at who made a mistake. Google SRE describes it as a mechanism for learning because people report honestly when they are not afraid of punishment.

Why keep incident tools separate from production?

If the incident tooling runs on the failing system, you lose it exactly when you need it. Google SRE advises keeping documents, trackers, and chat rooms independent of production so they stay accessible during outages.

How often should you test runbooks and playbooks?

Test them on a regular schedule and update them after every incident. Atlassian’s review calls for pre-tested, regularly audited runbooks, because systems change faster than documents do.

Sources

• Google SRE - Incident Management: Key to Restore Operations
• Post-Incident Review on the Atlassian April 2022 outage - Inside Atlassian
• Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile

Recommended

• OWASP Agentic AI Top 10: Security Risks for 2026
• Stop Costly AI Pilots: NIST Mapped AI Readiness Assessment for Leaders
• Prompt Injection Defense: A 2026 Guide for Security Engineers