Review: Snort IDS/IPS + Rule Writing
docrice
Member Posts: 1,706 ■■■■■■■■■■
When it comes to the subject of intrusion detection and analysis, there aren't many choices for training courses (aside from the highly-recommended SANS SEC-503, Intrusion Detection In-Depth) and I can understand the frustration of gaining insight into the topic if you're used to thinking about network security from a firewall management perspective or if you manage an IDS / IPS appliance that only spits outs alerts without relevant contextual data to further aid an investigation.
I currently do intrusion detection as part of my official responsibilities and I'm not new to Snort as a tool. However, I'm
also not the brightest person out there who can thoroughly learn from reading user guides and combing community forums, regardless of how well they're written or how active forum members are in exchanging free information. It can be a mental stumbling block of mine. Sometimes it's just more convenient to have a subject matter expert confirm (or correct) your interpretations of how things work as well as provide field-tested advice. And who better to provide that advice about Snort other than Sourcefire themselves?
https://na8.salesforce.com/sfc/p/80000000dRH9saofjYRQFQSTLUlibLaV0ZcOXMs=
I signed up for this four-day course to help fill some knowledge gaps. It's not cheap, and if you include the cost of the SnortCP certification exam, it adds up to about the same as your typical SANS course. And to be honest, just reading the Snort user guide and experimenting on your own will work just fine for many people. Snort's been around for over a decade and security engineers everywhere use it without taking the Sourcefire class. I could do the same and save my hard-earned cash, but given the very limited free time in my schedule to experiment and learn on my own through trial-and-error, I figured it'd be most efficient to just go through formal training, smooth out my rough edges, and solidify my Snort foundations.
So when you walk into class on the first day they hand you a thick course book, about 500 pages including a few appendixes. The training center where I took this course (also used for general training for other vendor products) has the Sourcefire training system image installed on the students' machines which runs a virtual environment simulating a typically-common scenario with an inside network, a DMZ, and a management network. If you've taken a Sourcefire 3D class for their commercial appliances, it's pretty much the same setup. The instructor for my class was apparently the person who writes the material for the Snort course and he obviously knows his stuff.
Day One started off with an explanation of Snort architecture, some basic examples of evasions and why IDS technology needs to be able to account for them, and general installation of Snort with all the typical dependencies (Barnyard2, PulledPork, etc.). Having already taken SANS 503, the IDS evasion concepts were all review for me. You also have to consider that I took 503 via OnDemand and that was hosted by Mike Poor, a former Sourcefire employee. 503 noted some work by Judy Novak (who also came from Sourcefire) and her work was also mentioned in this class as well. Small world.
For those interested in reading about evasion issues relating to IDS, check out the old-but-still-often-referenced paper by Ptacek and Newsham (Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection):
http://insecure.org/stf/secnet_ids/secnet_ids.html
as well as papers by Judy Novak and Steve Sturges on IP and TCP-based reassembly considerations:
Target-Based Fragmentation Reassembly:
http://www.snort.org/assets/165/target_based_frag.pdf
Target-Based TCP Timestamp Stream Reassembly:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.7912&rep=rep1&type=pdf
While Day One was rather slow for me since I wasn't new to Snort, I did find myself wishing that I had been able to take a course like this when I first started with Snort some years ago. It would've helped me get a jump on things much more quickly. Most of the other students in class seemed to be relatively new to the IDS world and since I've already been through the struggle of trying to get Snort up and running with the various dependencies (and components which aren't developed by Sourcefire), I know first-hand how much of a pain it can be. Having someone walk you through the experience and explaining each step is tremendously beneficial, especially if you have a full-time career going and little time at home to tinker. While a good majority of work is done at the command line, the class uses BASE as the administrative GUI, although other front-ends are mentioned in passing.
By the time Day Two rolled around, we were well into the discussion on Snort preprocessors. There's a lot of detail here and it's one of the areas I've been weak in. The instructor went through each section and described a lot of the settings and why they may or may not be relevant for custom-tweaking for a given environment. Lots of good reference information in the course book as well. Quite a few "Ah-ha!" moments for me. Main lesson learned here: go through the readme files for each of the individual preprocessors as there are tons of good info to pick up and apply to Snort tuning.
Then there was discussion about updating, disabling, enabling, and modifying rules via PulledPork. We also went through a distributed install scenario where a separate server and sensor are configured to relay events to the central console, something that many Snort install write-ups on the Internet don't really get into. In a real-world environment other than very small SMB networks, it's common to have multiple sensors that forward event information back to a central server, rather than having standalone / self-contained IDS islands all running their own management interfaces and databases.
Towards the end of the second day, we started getting into rule writing and syntax. For me, this was the real reason I decided to take the class on my own dime. I already know the basics of this, but it's always good to dive deeper and get the right advice on constructing rules for optimum IDS resource efficiency. Up to this point in the course I've been generally finishing all the labs quickly because I already have grounding in the covered topics, but I could foresee some struggles on the next two days as we got into advanced rule writing.
Day Three was about rule optimization, tuning, inline operation, PCRE (Perl-compatible regular expressions), and performance profiling. I suspect many Snort users can do basic day-to-day management of their IDS deployment, but not everyone understands the pattern matching system that Snort uses or make an effort to optimize rules based on how the engine processes packets. A few custom rules could easily bring down a sensor to its knees if they are blindly written without careful consideration.
I felt the regex walk-through was valuable as it's still a hazy area for me. While computationally expensive to perform, many VRT rules contain these. When evaluating alerts and the trigger-packet in question, understanding regular expressions in the rule is generally required for thorough analysis of the event.
One thing you definitely walk away with is an appreciation of the microsecond-level detail and statistics which Snort provides. Looking at performance measurements allows engineers managing Snort sensors to get real numbers as to which rules have been matched (or not matched) the most frequently and how much time and effort is put in their processing when evaluating streams of packets. This review allows for better tuning of rules. The better a rule is written, the more efficient Snort can run, lessening the chance of dropped packets.
Finally on Day Four, the more hardcore material was covered: byte_test / byte_jump / byte_extract options in Snort rules, flowbits, and packet / protocol analysis case studies to show how the rubber really hits the road in real life. If you're still relatively new to Snort, this is where it really starts opening your eyes as to the power and flexibility of the Snort rule language and the complex art of writing optimized rules that make good use of the fast pattern-matching system. This stuff takes practice.
So in summary, for me the class was worth the dollars I spent, especially since part of my work involves IDS / IPS and I'd better know what I'm doing. While I already had the general knowledge, getting clarification on many of the finer details can really make a difference at work. It would have been even more of a value if I was just starting out with Snort, but if that were the case I might have found the material overwhelming. But given the overall cost of the course, most people would be better off reading the Snort users guide and running their own sensors at home or work (unless you're really constrained on time or your employer is willing to foot the bill). Configuring Snort and tweaking configs and rules can be a complex endeavor and for a good reason - unlike some other vendors, Sourcefire's philosophy is for the engineer to really understand his or her network at a low level and tune sensors accordingly. Intrusion detection as an set-it-and-forget-it appliance toaster is unrealistic in my opinion unless your shop has no resources for analysis or reading reports related to these kinds of events.
Some unstated prerequisites for the course - know your basic TCP/IP. If your protocol analysis and networking Kung-fu is weak, read through Laura Chappell's Wireshark Network Analysis book as a good primer. And while SANS 503 covered Snort to some extent on the fourth day of its class, the Sourcefire class obviously goes much deeper and it's much more hands-on with the product. And best of all, after class you can go home, download the software and all the dependencies, and build up a distributed IDS (or IPS) system for free. If you want official Sourcefire VRT "subscribed" rules that are current (meaning not the "registered" rules which are out of date by a month), it's only $30 a year for home use (businesses pay $500 per sensor, with volume discounting for larger sensor counts). Otherwise you can go with the default rules that come with the particular version of Snort (which will already be out of date by the time you down it) or Emerging Threat rules. ET also has a paid version which provides more coverage than their free offering and I believe the pricing is similar to VRT.
The cost of the course for me includes a chance at the SnortCP exam which I get two attempts for. Since I've already paid for it, I'll probably spend the time working for it (although unfortunately it's not proctored). That said, since it's open book and essentially open terminal, the instructor mentioned the questions aren't necessarily going to be easy since they have an excuse to throw just about any question my way. Sounds like "fun."
I currently do intrusion detection as part of my official responsibilities and I'm not new to Snort as a tool. However, I'm
also not the brightest person out there who can thoroughly learn from reading user guides and combing community forums, regardless of how well they're written or how active forum members are in exchanging free information. It can be a mental stumbling block of mine. Sometimes it's just more convenient to have a subject matter expert confirm (or correct) your interpretations of how things work as well as provide field-tested advice. And who better to provide that advice about Snort other than Sourcefire themselves?
https://na8.salesforce.com/sfc/p/80000000dRH9saofjYRQFQSTLUlibLaV0ZcOXMs=
I signed up for this four-day course to help fill some knowledge gaps. It's not cheap, and if you include the cost of the SnortCP certification exam, it adds up to about the same as your typical SANS course. And to be honest, just reading the Snort user guide and experimenting on your own will work just fine for many people. Snort's been around for over a decade and security engineers everywhere use it without taking the Sourcefire class. I could do the same and save my hard-earned cash, but given the very limited free time in my schedule to experiment and learn on my own through trial-and-error, I figured it'd be most efficient to just go through formal training, smooth out my rough edges, and solidify my Snort foundations.
So when you walk into class on the first day they hand you a thick course book, about 500 pages including a few appendixes. The training center where I took this course (also used for general training for other vendor products) has the Sourcefire training system image installed on the students' machines which runs a virtual environment simulating a typically-common scenario with an inside network, a DMZ, and a management network. If you've taken a Sourcefire 3D class for their commercial appliances, it's pretty much the same setup. The instructor for my class was apparently the person who writes the material for the Snort course and he obviously knows his stuff.
Day One started off with an explanation of Snort architecture, some basic examples of evasions and why IDS technology needs to be able to account for them, and general installation of Snort with all the typical dependencies (Barnyard2, PulledPork, etc.). Having already taken SANS 503, the IDS evasion concepts were all review for me. You also have to consider that I took 503 via OnDemand and that was hosted by Mike Poor, a former Sourcefire employee. 503 noted some work by Judy Novak (who also came from Sourcefire) and her work was also mentioned in this class as well. Small world.
For those interested in reading about evasion issues relating to IDS, check out the old-but-still-often-referenced paper by Ptacek and Newsham (Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection):
http://insecure.org/stf/secnet_ids/secnet_ids.html
as well as papers by Judy Novak and Steve Sturges on IP and TCP-based reassembly considerations:
Target-Based Fragmentation Reassembly:
http://www.snort.org/assets/165/target_based_frag.pdf
Target-Based TCP Timestamp Stream Reassembly:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.7912&rep=rep1&type=pdf
While Day One was rather slow for me since I wasn't new to Snort, I did find myself wishing that I had been able to take a course like this when I first started with Snort some years ago. It would've helped me get a jump on things much more quickly. Most of the other students in class seemed to be relatively new to the IDS world and since I've already been through the struggle of trying to get Snort up and running with the various dependencies (and components which aren't developed by Sourcefire), I know first-hand how much of a pain it can be. Having someone walk you through the experience and explaining each step is tremendously beneficial, especially if you have a full-time career going and little time at home to tinker. While a good majority of work is done at the command line, the class uses BASE as the administrative GUI, although other front-ends are mentioned in passing.
By the time Day Two rolled around, we were well into the discussion on Snort preprocessors. There's a lot of detail here and it's one of the areas I've been weak in. The instructor went through each section and described a lot of the settings and why they may or may not be relevant for custom-tweaking for a given environment. Lots of good reference information in the course book as well. Quite a few "Ah-ha!" moments for me. Main lesson learned here: go through the readme files for each of the individual preprocessors as there are tons of good info to pick up and apply to Snort tuning.
Then there was discussion about updating, disabling, enabling, and modifying rules via PulledPork. We also went through a distributed install scenario where a separate server and sensor are configured to relay events to the central console, something that many Snort install write-ups on the Internet don't really get into. In a real-world environment other than very small SMB networks, it's common to have multiple sensors that forward event information back to a central server, rather than having standalone / self-contained IDS islands all running their own management interfaces and databases.
Towards the end of the second day, we started getting into rule writing and syntax. For me, this was the real reason I decided to take the class on my own dime. I already know the basics of this, but it's always good to dive deeper and get the right advice on constructing rules for optimum IDS resource efficiency. Up to this point in the course I've been generally finishing all the labs quickly because I already have grounding in the covered topics, but I could foresee some struggles on the next two days as we got into advanced rule writing.
Day Three was about rule optimization, tuning, inline operation, PCRE (Perl-compatible regular expressions), and performance profiling. I suspect many Snort users can do basic day-to-day management of their IDS deployment, but not everyone understands the pattern matching system that Snort uses or make an effort to optimize rules based on how the engine processes packets. A few custom rules could easily bring down a sensor to its knees if they are blindly written without careful consideration.
I felt the regex walk-through was valuable as it's still a hazy area for me. While computationally expensive to perform, many VRT rules contain these. When evaluating alerts and the trigger-packet in question, understanding regular expressions in the rule is generally required for thorough analysis of the event.
One thing you definitely walk away with is an appreciation of the microsecond-level detail and statistics which Snort provides. Looking at performance measurements allows engineers managing Snort sensors to get real numbers as to which rules have been matched (or not matched) the most frequently and how much time and effort is put in their processing when evaluating streams of packets. This review allows for better tuning of rules. The better a rule is written, the more efficient Snort can run, lessening the chance of dropped packets.
Finally on Day Four, the more hardcore material was covered: byte_test / byte_jump / byte_extract options in Snort rules, flowbits, and packet / protocol analysis case studies to show how the rubber really hits the road in real life. If you're still relatively new to Snort, this is where it really starts opening your eyes as to the power and flexibility of the Snort rule language and the complex art of writing optimized rules that make good use of the fast pattern-matching system. This stuff takes practice.
So in summary, for me the class was worth the dollars I spent, especially since part of my work involves IDS / IPS and I'd better know what I'm doing. While I already had the general knowledge, getting clarification on many of the finer details can really make a difference at work. It would have been even more of a value if I was just starting out with Snort, but if that were the case I might have found the material overwhelming. But given the overall cost of the course, most people would be better off reading the Snort users guide and running their own sensors at home or work (unless you're really constrained on time or your employer is willing to foot the bill). Configuring Snort and tweaking configs and rules can be a complex endeavor and for a good reason - unlike some other vendors, Sourcefire's philosophy is for the engineer to really understand his or her network at a low level and tune sensors accordingly. Intrusion detection as an set-it-and-forget-it appliance toaster is unrealistic in my opinion unless your shop has no resources for analysis or reading reports related to these kinds of events.
Some unstated prerequisites for the course - know your basic TCP/IP. If your protocol analysis and networking Kung-fu is weak, read through Laura Chappell's Wireshark Network Analysis book as a good primer. And while SANS 503 covered Snort to some extent on the fourth day of its class, the Sourcefire class obviously goes much deeper and it's much more hands-on with the product. And best of all, after class you can go home, download the software and all the dependencies, and build up a distributed IDS (or IPS) system for free. If you want official Sourcefire VRT "subscribed" rules that are current (meaning not the "registered" rules which are out of date by a month), it's only $30 a year for home use (businesses pay $500 per sensor, with volume discounting for larger sensor counts). Otherwise you can go with the default rules that come with the particular version of Snort (which will already be out of date by the time you down it) or Emerging Threat rules. ET also has a paid version which provides more coverage than their free offering and I believe the pricing is similar to VRT.
The cost of the course for me includes a chance at the SnortCP exam which I get two attempts for. Since I've already paid for it, I'll probably spend the time working for it (although unfortunately it's not proctored). That said, since it's open book and essentially open terminal, the instructor mentioned the questions aren't necessarily going to be easy since they have an excuse to throw just about any question my way. Sounds like "fun."
Hopefully-useful stuff I've written: http://kimiushida.com/bitsandpieces/articles/
Comments
-
onesaint
Member Posts: 801
Great review docrice. I called Sourcefire a few years back to see about taking the class when I first started dabbling with Snort. While I don't recall the price, I do recall it being prohibitive.
It always nice reading your humble perspectives and how much you feel investing in your knowledge is a much. Very encouraging.Work in progress: picking up Postgres, elastisearch, redis, Cloudera, & AWS.
Next up: eventually the RHCE and to start blogging again.
Control Protocol; my blog of exam notes and IT randomness -
ipchain
Member Posts: 297
Thanks for the great review and sharing your experience. Like you, I also do intrusion detection / prevention, so I might look into this in the near future. Would you mind sharing what the total cost associated with this course turned out to be? I might include it in next year's budget.Every day hurts, the last one kills. -
the_Grinch
Member Posts: 4,165 ■■■■■■■■■■
Solid review!WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff -
docrice
Member Posts: 1,706 ■■■■■■■■■■
Would you mind sharing what the total cost associated with this course turned out to be? I might include it in next year's budget.
Think SANS pricing. Ouch. Oh the pain, the pain...Hopefully-useful stuff I've written: http://kimiushida.com/bitsandpieces/articles/ -
ipchain
Member Posts: 297
Think SANS pricing. Ouch. Oh the pain, the pain...
I had a feeling you were going to say something along those lines. Good thing I am going to try to get my employer to foot the bill for this one!Every day hurts, the last one kills. -
docrice
Member Posts: 1,706 ■■■■■■■■■■
A quick follow-up - just got through the SnortCP exam. As mentioned in another thread, this is a non-proctored, open-book (and practically open-terminal) exam. That said, I didn't really need to access a running install of Snort to get me through this. However, going through the Sourcefire course as well as having the class material in front of me was an enormous advantage. I think the (free) official Snort User's Manual might have also sufficed, but since this is open-book, the exam will ask a lot of nitpicky questions about some rule syntax, preprocessor configs, etc.. I finished the exam in a little over an hour, although they give you three. My brain generally doesn't function after an hour or two, so if I had crossed the two-hour mark my brain would've blue-screened.
But I'll say that while I passed the exam, there's a lot more to learn through further practice. Snort is a highly-configurable and amazingly complex piece of software. Plenty of knobs to turn and buttons to push. Stuff to salivate over if you're into that sort of thing. And if you're ever in a position to perform product evaluations of vendor hardware, knowing how Snort works internally will provide a strong reference point in asking the sales people hard questions about the real capabilities of their offerings.
I'm sure there are people out there who will want to start tackling Snort without really getting the basics of TCP/IP down. My advice - don't skip the fundamentals. If your only understanding of "networking" is the CCNA, go read RFC 791, 792, 793, and 768 ... or go pick up Wireshark Network Analysis or Nmap Network Scanning and start watching the interface while you create packets with hping.
Intrusion detection / prevention and related analysis work heavily relies on an understanding of abnormalities in protocols and interpretation by target hosts and applications. In my opinion, Cisco training by itself is insufficient to prepare you for using IDS solutions (unless your expectation is to buy an inline appliance to run as a toaster and have it do all the work for you, in which case I don't feel that's true intrusion detection analysis and you'll have no real awareness of what's happening on the wire in your network).
Ok, so another obscure security certification out of the way. Now onto something even more obscure like SecurityTube's Python Scripting (although I'm not planning for the certification in this case).Hopefully-useful stuff I've written: http://kimiushida.com/bitsandpieces/articles/ -
riccardosl
Member Posts: 6 ■■■□□□□□□□
Do you have any suggested book to learn Snort that is really worth it to have? Or just hands-on experience?
