Incident playbook

This documentation is the initial description of a proactive incident process. It might be considered as a continuous plan or a rapid response plan, the purpose is simply to get the asset or system back into a working state which gives Truesec the possibility to prevent breaches and minimize impact of such. But we should also be mature enough to learn from incidents and become better at avoiding mistakes or issues in our technologies or processes.

This documentation and process is highly influenced by the Incident Management process described by Atlassian .

Overall incident process

The overall incident process is described as follows:

Detect

The incident is raised by anyone in the organization, possibly also through monitoring services and alerting functionality built into the platform. Channels for communicating with development team is any of the following:

• Slack
• Teams
• Email
• Phone
• Physical

The incident is logged by the respondent of the raised incident in our service management system (JIRA) for tracking and documentational purpose.

An overall impact description is put into the JIRA ticket, this is potentially just an indication of business impact. However, it is required to display the importance of incident.

Organize

An Incident Manager (IM) is appointed, this person is responsible for the overall process and has mandate to bring in further resources if needed. The IM is also responsible to ensure that the relevant stakeholders are informed about the situation and status updates. Depending on the type of incident the IM can be a Product Manager, Engineering Manager, or someone from the Incident and Response team.

The IM allocates and prioritizes resources needed for the Task Force (TF) and clearly communicates the roles and responsibilities of each member of the TF. The TF is communicated by the IM so that stakeholders know who works with what, making sure correct focus can be established for the team.

The IM organizes a group chat and/or a teams call with the TF. The IM creates a confluence page to capture the events and who does what, when, how, and why, etc.

Identify

The TF then works to identify the root cause of the problem and scope the work needed to implement, verify and deploy the solution. This scope is communicated to stakeholders.

Implement

Implementation is done by any member of the task force or other requested resource if needed and state of work is consistently updated in the JIRA ticket and communicated by IM to stakeholders. Scope and timeline are revisited when new information evolves, changes in timeline is documented and communicated to stakeholders.

The Version Control System (VCS) should contain a source code replica of production environment, either through separate branch or explicit release tags giving the TF the possibility to immediately create a hotfix branch for the isolated solution.

Verify

Implemented solution needs to be verified with “four-eyes”-principle and verified with QA. If possible, unit- and/or integration-tests should be created to help avoid similar issues in the future.

Deploy

Deployment happens in three stages:

1. Continuous deployment to development environment during implementation phase
2. Prior to release to production, deploy in eventual staging environment for stakeholders with access to test and verify solution.
3. Deploy to production

Postmortem

The IM owns the postmortem, closes the JIRA-ticket, and documents the overall process. The IM distributes the postmortem report to stakeholders. The fix and solution are committed back to the VCS development branch to prevent regression of the issue.