CDC Cyber Defence Center Respond Security Incident Investigation 

Home  Cyber Threat Response  Breach Response  Security Incident Investigation 

Faced with the growing threat of breaches, cyber-attacks and fraud, more organizations are ramping up their efforts to build robust incident response strategies that identify how an investigation would proceed and what data would need to be collected.

Many organizations have a fairly straight-forward security incident response program, where a security incident is identified and the response team takes steps to mitigate the issue and handle recovery. What sets organizations apart is how well they perform a root-cause analysis; that may involve deeper forensics and increased coordination among different groups as part of a detailed investigation, says Greg Thompson, a senior IT security executive at a large Canadian bank.

Having a formal investigation management strategy helps pave the way for understanding why an incident happened and taking appropriate steps to prevent a recurrence. Investigations are "less about the technology and more about the process," Thompson says.

Elements of Incident Response
Responders need to collect all the necessary information to assess the severity of the incident, notify relevant teams and business units, and mitigate risks as soon as possible. All the evidence collected during the course of the investigation has to be available in an organized and centralized fashion for various stakeholders to analyze. While technology, such as incident response software or a security information and event management (SIEM) system, can help responders collect evidence and track communications between various teams, the "analyst's gut instinct" and manual investigation can't be automated, Thompson notes.

An effective investigation strategy also relies on the team and senior executives being able to track what steps have been taken since the last update, who is responsible for each activity, how long each task is expected to take and the expected outcomes of each task, experts say. Each step taken by the response team must be logged as part of an audit trail. This log is particularly important if law enforcement is brought into the investigation or regulators want to see the evidence.

There are often multiple investigations in progress at a given time, and a responder may have a different task for each one, says Dan Houser, a senior security architect for a Fortune 500 global healthcare services firm. This makes a formal investigation process even more important so that the lead investigator knows who to contact for specific requests and who to pull together when analyzing certain incident types.

A well-defined process pre-defines the steps necessary to complete the investigation so that the response team knows what to do next. "Look at your playbook; this is what we did, this is what we have, and this is what we do next," Houser says.

What Can Be Automated
Having a clear idea of the steps required in the investigation will also enable organizations to use technology to automate parts of the process, which will save time and minimize the possibility of making a mistake. Evidence collection and management is a critical step, as well as generating a pre-defined task list and assigning the tasks.

If the system being used to support investigations can take certain actions, such as remotely connecting to the host and taking a snapshot of the system based on an automated alert, or generate a checklist with pre-defined tasks for a particular incident type, responders can deal with low-risk and routine incidents faster and more efficiently, freeing them up to work on more labor-intensive tasks and higher-risk incidents, experts say.

There is no definitive tool or ubiquitous platform for tracking all the information gathered during an investigation or for recording what steps the incident responders have taken, says Chris Triolo, vice president of professional services, enterprise security products at HP Software. As a result, organizations have a tremendous amount of flexibility when it comes to deciding how to manage their investigations.

"Many organizations do rely on Excel spreadsheets or just use a Word template to document every new incident," Triolo notes. But relying on spreadsheets to manage investigations can be time-consuming and downright challenging when there are multiple IT security incidents in progress.

USA Funds, an Indianapolis-based organization that assists students with financial aid for higher education, previously relied on different investigative tools depending on the nature of the incident and tracked incident response efforts with a spreadsheet, says Milan Tesanovich, the non-profit's director of information risk management. As a result of switching to a cloud-based incident response service from Co3 Systems, the response team now spends one-tenth of the time it took previously to manage incidents, Tesanovich claims.

Automating portions of the workflow also increased the team's accountability. With the software, USA Funds can establish "detailed evidence of the extent and timeliness of our compliance and due diligence efforts investigation," Tesanovich says.

For forensics, organizations can take advantage of SIEM platforms for increased visibility within the network infrastructure. Integrating the data with functions such as evidence attachment, internal ticketing, communication, and status tracking ensures teams can access both the workflow rules and data at once, says David Dudley, an incident response team leader at San Jose-based Rook Consulting, which specializes in risk management.

But automated tools aren't always the best option for notifying the next person in the chain. "It's still much faster to pick up a phone and call. The last thing you want is to have the e-mail sit in the Inbox," Houser says. But logging that the call was made on the work ticket or recording the conversation is useful for tracking and auditing the workflow, he adds.

Steps in the Investigation
No matter what tools are ultimately used to assign tasks and track steps taken by the incident response team, organizations need to take the time to define how the investigation would proceed, says Ted Julian of Co3 Systems.

Organizations should identify what threats and risks they face and make plans for how to investigate each scenario, Julian says. But a comprehensive response strategy doesn't just focus on what the organization has experienced but also on what similar organizations are facing, he stresses.

For a financial services organization, "just because you haven't been hit by a DDoS [distributed-denial-of-service] attack in the past year, that doesn't mean you shouldn't be planning for it," Julian warns.

Once an organization identifies the type of incidents it may need to investigate, it needs to develop a workflow defining the steps that would be taken. The actual steps depend on whether the priority for that incident type is to preserve evidence, or resume business functionality, Houser says.

"Getting your playbook straight is important in determining that first step," he adds.

It's also important to identify what kind of data needs to be collected to assess the severity of the incident and determine next steps, Julian says.

In the case of a system intrusion, for example, the organization would need to figure out where the attackers came in and what systems were affected. In the case of a DDoS attack, the team would need to determine the ingress and egress points, type of attack, and volume of the attack. And in the case of a lost laptop at a healthcare provider, the responders must identify what information was stored on the device to determine what data breach notification laws and regulatory requirements apply.

Organizations can take advantage of guidelines for incident triage, computer forensic handling, and other aspects of the investigation, including the SANS Institute's comprehensive end-to-end checklist on what types of technical data needs to be collected for different types of incidents, plus two resources from the National Institute of Standards and Technology: the Guide to Integrating Forensic Techniques into Incident Response and the Computer Security Incident Handling Guide.

These guidelines provide excellent benchmarks to determine if the response team has the experience or skills to collect the necessary types of data, says Brian Evans, a principal at Kansas City-based Tom Walsh Consulting, which specializes in healthcare information security. Too many healthcare organizations designate personnel to lead an incident response team who lack previous investigative experience, Evans contends. Training the teams in how to use best practices improves decision-making, he adds. And that training should cover threat analysis, forensic techniques, chain-of-custody process and understanding the legal standards.

Houser sums it up this way: A successful incident investigation requires a team with the right credentials, experience and skills, supported by technology to manage the data and the workflow metadata.