Becoming a Threat Hunter - What do you recommend?

Hello guys!

What trainings/learn options can you recommend to somebody who decided to become a professional Threat hunter?
What practical skills are crucial and necessary for hunters?
Will be appreciated for any advices, especially from seasoned Threat hunters:)
My background: 5 years experience in AppSec and network Pentest domains.

Find more posts tagged with

Comments

NotHackingYou

Threat hunter here - it sounds like you are already learning what you need. The crux of threat hunting is putting together all the pieces you have learned so far. Determine how an attacker could move through your network and then determine how you could detect that. Rinse, wash, repeat.

tedjames

eLearnSecurity has a Threat Hunter Professional course and certification.

u1tras

Thanks for advice NotHackingYou, but it's a little bit general. Could you clarify what theory and practical skills do I need? For instance, learn A,B,C then practice E, F, D.

u1tras

I have heard about ELS course, but would it be enough to pass to become a good hunter. I haven't seen any reviews here about this course from seasoned IR or hunters.

u1tras

And what about detection, forensics, TI etc. Do I need to be good at it as a threat hunter?

NotHackingYou

Hunting is all about detection. What you learn when you forensically examine a machine should feed your threat hunting. It's a circle. Hunting is designed to find bad things. Examining the bad things fuels your hunting.

To be more prescriptive, I suggest digging deep into common protocols and how those could be abused and then moving on to a deep understanding of the typical attack vectors. I suspect you already have a detailed knowledge here. To apply this knowledge in an active, aggressive fashion designed to detect and disrupt an attacker in real time is threat hunting. I like to start with examining dropped network traffic, DNS logs, and Windows logs.

mactex

Threat Hunter also (one of my hats). Agree with all NotHackingYou stated, and will add that typically you are trying to detect adversaries on your network that your security tools can not. With that; you need a strong understanding of systems and protocols that are most vulnerable, or where Security has the least visibility in your environment. For most; that is network protocols and Unix/Linux IMO.

LionelTeo

Before I go into the gist of threathunting, there are two 'types' of threat hunting perspective which I had heard people talk about. The first type is the external kind of threat hunting. This threat hunting is actually very close to threat intelligence work (or it actually is), which is to pivot pieces of information regarding an adversary and link up a various campaign to the same threat indicator. This threat hunting involves analysing information such as comparing code sections between malware samples, domain registrar, methods of attack, emails, forensic artefacts and partnership with other organizations to derive the information. While this can be classified as threat intelligence work, I had heard or known of people who will consider this type of work as threat hunting.

The threat hunting which I believe you are looking at is regarding identifying the visibility gaps in your current environment and creating the detection to cover the gaps. There are two roadblocks, understanding where to start and what to look at. Regarding on knowing where to start, the first step is to understand what are the information sources which you would like to ingest into your threat hunt operations. Based on my experience, the information source would usually derive from the following: similar malware samples characteristics, honeypot logs, red team reports, replicated attacks and online research articles.

The second hurdle is to have the necessary skills and environment to be able to look up to 30-90 days of logs in your current environment to check your logs visibility against the sources you had derived your information from. I am not able to give any insight on this as this is largely depended on how the logs visibility in your environment is setup, and the tools you used. You had to be comfortable with developing content detection on the tool you are using, and if necessary to take up courses that are related to the tools you are developing your content from.

Here are some examples to derive the threat hunting information from just to give you an idea.

Research Article - Unicode Domain Phishing - https://www.xudongz.com/blog/2017/idn-phishing/
Research Article - Malware using DNS TXT Records - https://thehackernews.com/2017/03/powershell-dns-malware.html
My Dridex Malware Characteristic Analysis in 2015 - https://www.sans.org/reading-room/whitepapers/detection/learning-dridex-malware-adopting-effective-strategy-36397
Honey Pot Data Captured and Presented at Splunk Conference - https://conf.splunk.com/session/2015/conf2015_MGough_MalwareArchaelogy_SecurityCompliance_FindingAdvnacedAttacksAnd.pdf

To summarize up on the points as an answer to your question, the following skills are useful for threathunting.

Malware Analysis - to identify common characteristic between multiple samples and create detection based on correlated characteristics
Penetration Testing/Offensive Security - to replicate attacks in a research environment and compare the results with the current environment
Setting up and maintaining a Honey Pot (or alternatively find a honeypot researcher that is willing to provide the logs) - to capture attacker traffic and use it to compare current logs visibility
Content Tools (SIEM/Yara/IDS) - To develop the content using the current tools that are available in your environment.
Cyber Threat Intelligence - optional if you are looking correlating adversary information on a periodic basis.

mactex

Or, if a GSE posts in your thread; listen to what they say.

the_Grinch

SEC511 is the course to take if you can get it paid for. All about hunting threats!

u1tras

NotHackingYou wrote: »

Hunting is all about detection. What you learn when you forensically examine a machine should feed your threat hunting. It's a circle. Hunting is designed to find bad things. Examining the bad things fuels your hunting.

To be more prescriptive, I suggest digging deep into common protocols and how those could be abused and then moving on to a deep understanding of the typical attack vectors. I suspect you already have a detailed knowledge here. To apply this knowledge in an active, aggressive fashion designed to detect and disrupt an attacker in real time is threat hunting. I like to start with examining dropped network traffic, DNS logs, and Windows logs.

Thanks for advice NotHackingYou! When we are talking about detection - is it like SOC detection with tools or more handy painstaking job?
I'm trying to understand do I need SOC analyst skills or it's something different.

u1tras

mactex wrote: »

Threat Hunter also (one of my hats). Agree with all NotHackingYou stated, and will add that typically you are trying to detect adversaries on your network that your security tools can not. With that; you need a strong understanding of systems and protocols that are most vulnerable, or where Security has the least visibility in your environment. For most; that is network protocols and Unix/Linux IMO.

Thank you mactrex! But how to learn these things? Among currently avalilable trainings/learn options from where should I start?

u1tras

Thank you very much LionelTeo for your post! I'm starting to understand what is threat hunting about. I definitely interesting in the 2nd 'type' of threat hunting which you mentioned. I'll carefully examine all your links. If you don't mind I'll pick up some quotes from your post and ask more granular questions.
"Based on my experience, the information source would usually derive from the following" - can TI act as a source for threat hunter? I'm trying to understand how is crucial to know TI for threat hunter, at least in the beginning of the path.

"The second hurdle is to have the necessary skills and environment to be able to look up to 30-90 days of logs" - is it SOC analyst's job or something different? I'm kind of Red teamer and can be confused in blue team terms.

"You had to be comfortable with developing content detection on the tool you are using" - is it about creating new signatures for Snort/Bro, YARA rules, knowing what and how to search in ELK/Splunk etc?
And one more question - should I know IR/forensics domains or my task is only to find compromises and let IR/forensics guys to do their job?

u1tras

mactex wrote: »

Or, if a GSE posts in your thread; listen to what they say.

Sure, I will!)

u1tras

the_Grinch wrote: »

SEC511 is the course to take if you can get it paid for. All about hunting threats!

Thanks for advice Grinch!
I've taken SEC511. If I remember correctly it doesn't mention hunting (only TI in short) in a explicit form, but really contains a lot of things for threat hunters as I now understand.

LionelTeo

u1tras wrote: »

Thank you very much LionelTeo for your post! I'm starting to understand what is threat hunting about. I definitely interesting in the 2nd 'type' of threat hunting which you mentioned. I'll carefully examine all your links. If you don't mind I'll pick up some quotes from your post and ask more granular questions.
"Based on my experience, the information source would usually derive from the following" - can TI act as a source for threat hunter? I'm trying to understand how is crucial to know TI for threat hunter, at least in the beginning of the path.

TI can highlight malware/adversary of interest and also help the threat hunting team by obtaining samples/information to the team for further analysis. The malware of interest is largely depending on the interest of the business, it can be a malware related to a nation-state adversary or a malware targeting a specific industry. We can take the Shamoon malware as an example, Shamoon is a malware developed to target and orchestrate a devastating attacking against Iranian oil companies. If your company business is in the oil industry or is based in Iran, then this can be classified as a malware of interest to your company.

https://www.scientificamerican.com/article/following-the-developing-iranian-cyberthreat/
https://securityintelligence.com/the-full-shamoon-how-the-devastating-malware-was-inserted-into-networks/

The TI can also help to analyze multiple similar samples and reports common characteristics between each sample to the threat hunting team. After determining the similar characteristics from various samples from the same malware family, the threat hunter can follow up to check against the logs visibility based on the characteristics and proceed to develop the detection against the malware of interest.

Note that TI also had the added responsibility to correlate samples that may be of different family and identify them to be of the same author using techniques such as analyzing the compilation of the code, metadata, API calls and section of the codes. Getting these characteristics can help them to determine if the malware from different family could have originate from the same adversary through the way the malware code is developed. This added work is usually not required from the Security Operations, it is more important for the threat intelligence for them to correlate various malware samples back to the respective APTs.

u1tras wrote: »

"The second hurdle is to have the necessary skills and environment to be able to look up to 30-90 days of logs" - is it SOC analyst's job or something different? I'm kind of Red teamer and can be confused in blue team terms.

SOC environment and practices may differ for each company. To be able to conduct a successful threat hunt, the threat hunter would have to be able to access historical logs as far back as possible. This is necessary so that they are able to determine if the characteristics they pivot from is a common occurrence seen within the environment. I would recommend 30-90 days but logs storage is depending on the size of the environment you are looking at, smaller environments may only require lesser retention if budget and storage is a concern (usually the logs is archived after a certain period as a common practice so the logs may not be accessible without special procedures).

The threat hunter are encouraged to drill down and investigate further on any unique activity seen on during the threat hunt as this will give them insights to verify if this is a real compromise or a false positive. If it is a false positive, then it helps them in their content creation process to understand how to better tune the content development to avoid capturing the false positives.

u1tras wrote: »

"You had to be comfortable with developing content detection on the tool you are using" - is it about creating new signatures for Snort/Bro, YARA rules, knowing what and how to search in ELK/Splunk etc?

Yes, you are along the lines regarding the products commonly used for content development.

u1tras wrote: »

And one more question - should I know IR/forensics domains or my task is only to find compromises and let IR/forensics guys to do their job?

Not necessary although malware analysis course would briefly touch on memory analysis to analyze malware variants that inject themselves into memory. Knowing from the IR/Forensic perspective is an added bonus, and can help with some aspect such as knowing how the command arguments for some programs, such as svchost.exe usually runs in the memory. Such knowledge is useful if you are looking at malware that runs svchost with a different command argument. You can identify a malware sample running a particular svchost command argument as an anomaly and used them to compare against the current environment.

Your main goal as a threat hunter is not to find a compromise, as there is just that little amount of compromise you can possibly find in your environment if it is properly secured. A threat hunter goal to identify visibility gaps gathered from the threat hunt sources that are incorporated into the threat hunting operation model, then follow up creating the detection to address the visibility gaps.

If you find a compromise during a threat hunting, that is great and you can follow up with the respective team if it is necessary. Even if there isn't a compromise after a threat hunt, your team can still follow up with the content team to put in the necessary detections and improve the overall detection structure of the team. Therefore, your KPI would revolve around better improving the team detection and visibility through the threat hunting model instead of finding an actual compromise.

u1tras

LionelTeo wrote: »

Your main goal as a threat hunter is not to find a compromise, as there is just that little amount of compromise you can possibly find in your environment if it is properly secured. A threat hunter goal to identify visibility gaps gathered from the threat hunt sources that are incorporated into the threat hunting operation model, then follow up creating the detection to address the visibility gaps.

LionelTeo, but how many environments are properly secured?
It's very interesting actually, because I thought that threat hunting is primarly about finding an adversary's signs in an environment. And maybe other tasks, but only after the primary one.
SANS teaches us that:
1. Chances are very high that hidden threats are already in our organization.
2. Our controls alone are insufficient.
3. We don't have to wait but to constantly look for intrusions and catch them in progress.
4. Threat hunters focus their search on adversaries who are already within the networks and systems.
etc etc

So, I'm a little bit confused with it

LionelTeo

u1tras wrote: »

LionelTeo, but how many environments are properly secured?
It's very interesting actually, because I thought that threat hunting is primarly about finding an adversary's signs in an environment. And maybe other tasks, but only after the primary one.
SANS teaches us that:
1. Chances are very high that hidden threats are already in our organization.
2. Our controls alone are insufficient.
3. We don't have to wait but to constantly look for intrusions and catch them in progress.
4. Threat hunters focustheir search on adversaries who are alreadywithin the networks and systems.
etc etc

So, I'm a little bit confused with it

SANS isn't exactly wrong, but it is a matter of perspective that is based on. If there is a significant amount of security posture already in place, chances are your alerting system may have already been capturing a good portion of the activity intended. If that is the situation, what would be the chance of a threat hunt activity actually finding something, and if it doesn't, does that means that the threat hunt is unsuccessful?

There is a slight problem with the lookout for intrusion and catching them in the act ideology, is that the threat hunt only looks at the particular occurrence. If we are looking at threat hunting trying to capture a particular activity in progress, this would inadvertently mean that the threat hunter had to look at the occurrence of activity over and over again until he finds it. What if one person from the threat hunting team only finds the 1-2 security event/incident in that work year, does that means that the rest of the threat hunting team members are not doing their duty correctly for the rest of the year?

Consider that if a threat hunting team responsibility is to identify visibility gaps via threat hunting and create a detection alerts to address these gaps. Although the threat hunter may not have discovered any signs of compromise during the initial hunt. However, what happens if the alert created from a threat hunt activity detects a compromise a few months, or years later. Based on this perspective, does that means that the threat hunt is successful or the detection is successful? Therefore, a good threat hunting team not only attempts to look for signs of an adversary, but continuously helps to identify visibility gaps and improve the security detection of the team. As long as the threat hunting team continue to identify and improve on visibility gaps, one of the detection alerts created as a result of threat hunt would capture a true positive detection successfully.

u1tras

Thanks LionelTeo, I understood you.
I imagine what trainings/tools/skills should I learn for "finding an adversary" part of threat hunting.
Can you recommend any learn options for "identifying visibility gaps" part? Maybe some SANS trainings or other (if it exists at all)?
Seems like ELS THP and SEC511 previosly mentioned here definitely not about it.

NotHackingYou

Agreed with LionelToo on measuring success for a threat hunter. It is difficult and you have to be okay with not really "winning" in the traditional sense. I will also second SEC511, that was a great course! And for the other question - yes, it's like SOC tools but deeper and in wider breadth. You're also looking for things outside of what tools can easily detect.

cyberguypr

Keep in mind that threat hunting is a multidisciplinary practice. A good threat hunter knows a bit about everything and “has been there and done that” in many areas of IT/Infosec. But don’t try to cover the world, that’s just impossible. Since the threat hunter is sort of a unicorn I always lean towards a hunt TEAM. This is what you will find in mature security operations.

From a training perspective, understand that given that multidisciplinary aspect there is no one specific training that is the go-to for threat hunting. For example in the SANS world SEC 508 and FOR 572 promote a threat hunting component. However, it is easy to argue you can grow as a threat hunter with many courses including 511, 542, 560, 578, 599, and many others that escape my mind. Things like stack counting, grouping, clustering, etc. may not pop up in any of those courses specifically so there is a lot of knowledge that will need to be acquired elsewhere. And don’t forget about intangible skills such as problem solving, critical thinking, thinking like an attacker, being able to connect the dots, etc.

How do you address visibility gaps? There's no training for that. Leverage something like the MITRE ATT&CK framework. Since it focuses on post-compromise activities, it lends itself perfectly for hunt activities. If you assess your organization capabilities across different techniques in the matrix, you could easily identify where the gaps lie.

More reading/video for your viewing pleasure:
- https://threathunting.org/
- https://sqrrl.com/media/Your-Practical-Guide-to-Threat-Hunting.pdf
- https://www.sans.org/reading-room/whitepapers/threats/scalable-methods-conducting-cyber-threat-hunt-operations-37090
- https://attack.mitre.org/wiki/Introduction_and_Overview

u1tras

Thanks cyberguy, for advice and useful links!

LionelTeo

u1tras wrote: »

Thanks LionelTeo, I understood you.
I imagine what trainings/tools/skills should I learn for "finding an adversary" part of threat hunting.
Can you recommend any learn options for "identifying visibility gaps" part? Maybe some SANS trainings or other (if it exists at all)?
Seems like ELS THP and SEC511 previosly mentioned here definitely not about it.

There isn't really a training cater to identifying visibility gaps as it is specific to the environment you are threathunting for. For example, if a threat hunt identifies a series of window events seen from a compromised honeypot, and the particular window event ids/logs is missing in your environment, that will be the visibility gap. The visibility gap can be different for different SOC since it has different requirement and interest on how the logs visibility is setup. It is more of a thought process than an actual skill.

Unless you are referring to checking the findings replicated in your lab against your environment, in this case, it is depending on the SIEM that your organization had adopted. It can be Splunk/ELK depending on the organization choice. After obtaining the findings from the lab, the threathunter would compare the findings with the current environment via Splunk/ELK if you are asking about this.

u1tras

Thanks LionelTeo, I think now I know where to move and what to learn.