Alternate title: So You’ve Found Yourself in a Security Incident

I’ve attended, commanded and, yes, caused, many security incidents in my career. This is not an appeal to authority but, rather, an appeal to experience. Often we don’t have time to talk or teach about security incidents, so this document collects various tips and tricks I’ve encountered throughout the years. Please note that I’m not a lawyer, and do not dispense legal advice, so my focus will remain on information security and I may have to decline to answer your question.

Tips and Tricks

  • Keep Calm. The first thing to do when you realize a security incident has occurred is to keep a cool head. Irrational (i.e. emotional) reactions to these things are natural, but they also cloud your judgement and can cause you to make mistakes.
    • Real-Life Example: I was watching someone engaged in an incident. She was clearly fighting for focus – she caused the incident – as she tried to type in the correct commands to fix the problem. I saw this and pulled her away after getting approval from the incident captain. She begrudgingly left her laptop. I spoke with her. She didn’t want to admit she was stressed, or that she was making mistakes. I told her that I pulled her off because she was making mistakes, making the incident worse. I told her that taking a break to find some calm will help her. We both stood and waited for (what she thought was) an agonizing five long minutes before she was noticeably calmer. I released her back to the incident and she was able to type quickly and clearly – no typos – without any mistakes.
  • Physical Safety First. Make sure you are physically safe. No information or amount of money is so important that it overrides your physical safety. Whether you are a remote company or have physical offices, physical security always takes precedence. Staying calm and rational in a tense situation will help you see priority. But just so we’re clear:
    • If you are driving when you get the alert, pull over to a safe place and take the call.
    • If you are being mugged and asked for your bank card and PIN, your laptop and password or whatever, give the attacker what they want. Then find safety. Then call for help.
    • If you are being asked by a country’s border security to give up your laptop and password, do so. Then find safety. Then call for help.
    • If you are speaking to someone that is in need of help, ensure they are physically safe first and then work everything else.
  • Lock Down Communications. This is the cause of the most arguments in companies. This means using private Slack channels for communication, converting public incident channels to private, using a Google Meet or Zoom bridge for information flow, or even using Google Docs instead of Notion to document the incident. If they aren’t the first responders (and they usually aren’t), the Security team will take control of the situation and lock down communications to protect employees, the company, and its customers. It is not about being malicious, elite, or anti-productive. It is about ensuring only the correct set of people know about the situation.
    • Companies want transparency, but security incidents are one of the few exceptions to this rule. During an investigation we make throw around people’s names and it can look like accusations, all of which are backed by little real evidence at the time. It can be an awful experience on both sides of the table. But, especially in the case of insider threats, the person doing the attack could be listening to the security response and change their tactics. That’s why security teams, police forces, and military lock down areas and control the flow of information.
    • Even a private Slack channel filled with 200 employees is better than a public channel where 500+ employees can view it, link to messages, etc.
  • Find or Be the Incident Commander/Captain. If you are the first person to discover the incident, you are the captain. If you can’t find the captain, you are the captain. That may seem scary, but let’s alleviate it by setting expectations:
    • You do not need to be a domain expert.
    • You do not need to be the captain forever.
    • You do not need to be the scribe.
    • You must find the right set of people to help in the incident, without telling the entire company.
    • You must find someone to take on the captain role if/when you can no longer do it.
    • You must coordinate communications between people doing the investigation. You are the switchboard operator.
    • You must update all parties on a regular interval (min: 10 mins, max: 1 hour).
  • Finish vital important work first. If you are in the middle of something vital (unrelated to the security incident), please either finish it, find a good stopping point or delegate the work or the incident handling to someone else where appropriate. I’ll leave it to you to determine how “vital” is defined, but leaving this work in a half-finished state is a recipe for failure.
  • Keep a timeline. Knowing what happened and when it happened, what actions were taken and when, are critical to the security incident. It helps investigators (be it police, lawyers, or just the security team) keep track of whether the correct response was followed, but also whether there are areas that were missed. I’ve been part of incidents where I’ve missed temporary fixes we made to infrastructure, which then cause problems days later when we forgot to remove them. I’ve also been part of incidents where we’ve made mistakes and a correct timeline showed that we made the right decision based on the information we had at that time. Here’s some basic guidance on writing a timeline:
    • Your first timeline won’t be pretty; it can and will be polished later.
    • Use a Google Doc, and just start entering events in a list. Don’t get fancy.
    • Prefix each event with the timestamp.
    • If the incident crosses the date boundary, separate the list into different days and use the date as the section title.
  • Know When to Escalate. If you are working on an engineering incident and then a thought occurs to you – however small it is – that what you’re seeing may be a security incident, escalate. Even if you could be wrong, escalate. Even if it is 2am, escalate. Even if it is Xmas, escalate. Security can come in, assess the situation and, if needed, convert it to a security incident. Or they can say, “nope all is good” and leave without being bothered. It’s what we are here for and we’d rather be a part of a false positive than a false negative.
  • Use UTC for all timestamps in documentation. First, it is easier because almost nobody lives or thinks in UTC, so it forces them to convert to their local timezone. Second, it doesn’t bias the documentation to a specific person’s (usually the author’s) timezone. Third, it does not change when daylight savings time happens.
    • If you want to specify multiple timezones, be prepared to list use all the timezones in the company. This makes for a verbose document. I do not recommend this.
    • Using local timezones (e.g. Pacific, Mountain) leads to someone entering their own local timezone (e.g. Eastern) and confusing everyone.
  • Take breaks. Security incidents can be exhausting and part of that exhaustion is the adrenaline rush that happens over a long period of time. When you find your mind or body starting to drift, find a good stopping point, delegate work to others, and take a 10 minute break. Go for a walk. Go get sun on your face (unless you live in YVR, then go get overcast on your face). Then return to the incident feeling a bit more refreshed.
  • Eat. Never forget to eat. The adrenaline produced by a security incident can make you not feel hunger (it’s a primitive thing, look it up!), but your brain lives on glucose and needs that energy to function. Your work probably uses your brain, so…yeah.
  • Focus on Facts. Security incidents are fraught with accusations because the investigation happens in real time. You will naturally want to say " John Doe did it" but resist the urge. Instead, focus on things you objectively know due to evidence you’ve collected, and then mark anything you don’t have evidence for as a suspicion (e.g. “ John Doe’s username was seen logging into the server at this time”). It is not your job to accuse someone, it is your job to collect the evidence that makes it easy to argue that the suspect did something. Remember, we are all remote, so even if someone’s laptop made the connection to a critical server, you still don’t know if it was truly them at the keyboard… that is, unless you captured their fingerprint as an MFA factor, and even then, were they awake when it happened? That’s why we stay focused on the facts. It also helps protect the company from defamation claims from the person being accused.
  • Make an Incident Ticket. Always ensure that a single ticket exists for each incident. You will be referring to this ticket often when making changes/fixes/etc. And Security will keep asking for a unique identifier to refer to this incident; saying “INCIDENT-428” is much nicer and anonymous than the “We lost all our customer’s money incident” .
    • While the tool doesn’t specifically matter, I recommend Jira so long as the right access rights can be granted to individual issues (tickets).
    • My other recommendation is to use a single Jira project named “Incidents” and store them all there.
    • If you can’t use a single Jira project, ensure an issue type of “Security Incident” exists.
  • Gather evidence. A key part to any security incident is the investigation. It can happen in any phase. Be sure to take screenshots ( Shift-Command-3 on MacOS) of anything you see that’s out of the ordinary. Save logs and other vital information for analysis later. Anything that helps to prove facts vs suspicions. As an auditor once told me, “if it’s not written down, it didn’t happen.”
  • Store Evidence in GDrive. Evidence files can get large, so they cannot be stored as Jira attachments. Instead, specify a specific Google Shared Drive, open only to the CSIRT team. The top-level folders in the drive are named after the Jira tickets.
  • Screenshots are great evidence. Full screen screenshots (Shift-Command-3) are great for collecting evidence because they include the clock (a timestamp when the screenshot occurred) and the image of whatever you were looking at that seemed like evidence.
  • Gather evidence with hashes. When gathering evidence like log files, make sure you capture the SHA-1 hash of the file. We use hashes in evidence collection to show that the file has not changed between the time the evidence was collected to when it is used by police, lawyers, auditors or other people that care about the integrity of the data (chain of custody). To do this on MacOS, find the evidence file, open Terminal.app and type openssl dgst -sha1 <name of file>. Put the hash and the filename in a comment in the Jira ticket.
    • Try this now on any file in your system. It’s pretty cool.
    • If, for whatever reason, you can’t do this yourself, find someone who can and then note in the Jira ticket that you gave the authority over to the named person.
  • Practice. This sounds corny and cliché but practicing how you respond to incidents helps prepare you for future incidents. However, security incidents are both rare and usually on an eyes-only basis. To get around this, if you have the opportunity to be part of a security incident, volunteer. Be a scribe. Be a captain. Often the best way to learn is to watch.

If reading this list left you with any questions, please don’t hesitate to ask.