What Exactly is Threat Hunting - and Why Does it Matter?

Naturally, we all want to detect every threat to our network as soon as it manifests itself. That's why we spend a ton of money every year on tools that detect things automatically.

But what do we do when automatic detection isn't enough? Perhaps there's a new attack that doesn't yet have a detection signature, or maybe the threat you're after can't really be found using traditional detection methods. Are the tools you use less effective than you assumed they were? Do they struggle to keep up with the evolution of attacker techniques?

These scenarios are great examples of where threat hunting comes into play. With hunting, you send your most experienced analysts into the unknown, searching for threats that the machines failed to find. You send your most experienced because, to be successful, the hunter is going to need to know how to coax data out of your toolset. They're also going to need an intimate knowledge of different types of malware, exploits and network protocols to navigate that vast heap of data consisting of logs, metadata and PCAP.

The problem with sending our best and brightest is that we just don't have enough of them. There's a projected cybersecurity job shortage of 3.5 million by 2021— and that's all security personnel, not the uber-experts. I would venture the candidate pool for those jobs is even smaller.

Hope is not lost, however. Though there is debate over the various attributes that make up threat hunting, every SOC has the potential to engage in some level of hunting – and taking action is what's truly important.

What is Threat Hunting?

Ever since I began my career in cybersecurity in a dark room lit only by stinging light of computer monitors, I have heard many disagreements over the definition of hunting. I've been in and witnessed interviews where the interviewer scoffs or strongly disagrees with what the interviewee has defined as hunting. Many people add all kinds of qualifiers in an attempt to define hunting; some of which make me scratch my head, wondering why we're so caught up in semantics while there's a network to defend.

When I began working in my first Security Operations Center (SOC), my team and I never had a conversation about the definition of hunting. We had to write something on the timesheet and at least someof the things we were doing fell under the accepted industry umbrella of hunting, so rather than write, "I spent 30 minutes searching for IoCs, 30 minutes hunting and 30 minutes experimenting with new rules/parsers," everything got looped under "hunting" and we called it a day.

The first time the hunting definitions and procedures I had grown accustomed to were put into question was when I left my first job and began interviewing for another. When one interviewer asked what I do to hunt for malicious activity on a network, he scoffed at my response of pulling open-source indicators of compromise (IoCs) from places like malware-traffic-analysis.net, zeustracker, etc., responding, "Well that's not really hunting, is it?" I then continued to list other indicators, like POSTs without referrers, HTTP traffic to IPs and other noisy starting points that I had used in the past to begin an open-ended investigation. When I asked if the interviewer considered those activities to be "hunting" the response was "yes."

So, according to this person, searching for IoCs is not hunting; searching for things that are sometimes indicative of malicious activity but will require you to sift through benign traffic is.

I suppose I can understand the logic. With the former, you're searching for something that is known-bad, so any hit should indicate (assuming your IoCs are from a good source and have been vetted appropriately) malicious activity. Some IoCs will, due to the lack of context, require you to do some more digging into the event to see if it is in fact malicious, but this isn't any different than vetting alerts from your IDS. The latter, sifting through the results of less specific queries, does seem to follow the analogy that people were going for by calling the activity hunting.

It Doesn't Matter What You Call It, Just Do It

I mostly subscribe to the definition that hunting is "anything you proactively investigate that your current detection methods don't catch." However, the important point here is that it doesn't really matter what you call it. What matters is that you have capabilities for discovering and remediating risk. In short, it doesn't matter what you're calling it, just that you're doing it.

It's also important to note that hunting is, or at least should be, a continuous improvement process:

If you discover some method that produces results, make it repeatable and add it to your normal detection methods.
If you find yourself repeating the same workflow and it produces results without a lot of false positives, then automate it if possible.
During a hunt, if you find that there is SMB file access of a certain size, followed by an upload over TLS using PowerShell that ends up being exfil, then have your tools inform you every time this happens.

Of course, the feasibility or difficulty of doing something like this will depend on the tools you have at your disposal. The kind of hunting you do in your SOC will depend on many factors, including your acceptable risk and what your security stack and staff looks like. Currently, only the most experienced and sophisticated analysts can partake in threat hunting given it often hinges on tribal knowledge and tricks. If the tools could however take some of that complexity away, then perhaps more teams and individuals could hunt. Choose whatever definition makes sense to you, as long as you're covering your assets.

About the author: Troy Kent is a Threat Researcher at Awake Security. He has spent his career in SOCs as multiple Tiers of Analyst and an Investigator; working ticket queues, hunting for security incidents, rapidly prototyping new ideas into existence, working terrible hours and questioning career decisions.