Evaluating IPS appliances

docrice

As a spin-off from the other thread about Sourcefire (http://www.techexams.net/forums/off-topic/69318-sourcefire.html), I'd like to get some generalized input from those who have deployed various IPS systems within their environments.

I'm evaluating several different vendors at the moment, each having price tags attached with bragging rights when it comes to high dollar amounts. I'm a firm believer that what works for some organizations doesn't necessary work for others. However, a few fundamental issues come into scope as I form conclusions and recommendations.

1) Open vs. closed rule set. Coming from a Snort background, I've enjoyed the luxury of responding to an alert and seeing what rule caused the sensor to trigger. Some commercial vendors have completely proprietary systems where you can't determine the signature language to ascertain whether the packet hit a rule because the rule was too wide in scope or not. Snort / Sourcefire obviously also have shared object rules which are binary blobs, and Sourcefire does this to hide vulnerability information if the affected third-party vendor wants it closed for the time being given the potential of it becoming weaponized as a 0-day. Most of their rules are open, however. Which leads me to...

2) Being able to see the packet, both header and payload data. Snort / Sourcefire gives you the specific packet that triggered the alert. Although it doesn't provide additional data (such as the complete stream), it does provide a small slice of evidence to form some initial conclusions. If I don't get this with vendor x, then I'm essentially left with having to completely be reliant on that vendor's ability to have extremely accurate rules because there's no way I can verify it other than relatively-limited surrounding data (similar alerts in the same net, same type of hosts, within a small time window, etc.).

3) I think the IPS vs. IDS debate still rages on, but in short I'm assuming that organizations who newly-deploy IPS systems start with a "safe" (read: default) rule set which should block most of the obviously-bad stuff. I'd guess that most organizations never bother to continuously monitor and tune. If so, that means potentially there are still a lot of things which slip by because in my mind there's no way any vendor can get everything right at that first line of defense. I haven't read all the NSS Labs reports (assuming they're even reliable), but I'm throwing out a number to say out-of-the-box we have an average 80% catch-rate. So, does anyone deploy additional detection-only listening posts with more aggressive tuning? I understand that doing large scale analysis on trillions of packets is expensive and prohibitive, but I'm just not comfortable trusting any vendor's claim of being able to stop it all with their default rules, I don't care how large their research teams are or if they buy vulnerabilities discovered by third-parties.

For those who have deployed Sourcefire vs. others (Check Point, HP TippingPoint, Cisco, McAfee, IBM, Top Layer, Juniper, etc.), how do you feel about these points assuming you had some degree of bandwidth to perform things like log analysis and alert reviews? I'm trying to give the other vendors a fair shake and want to keep an open mind to new ways of doing things. But like all things expensive, there's a lot of marketing fluff that I need to weed through.

Everyone

Any IDS or IPS is only as good as its operator.

IMHO you should use IDS and IPS together.

Using one vendor for your network level, and another for your host level is another way. Although outside of the DoD, I haven't seen anyone use a HIPS.

The hospital I was working at used Top Layer. The former IT Director was really into security, so it was kind of a toy for him. It got neglected after he left. I don't know if anyone else even had access to it.

I've used McAfee's products, they were actually pretty good.

There's one other that I've used, but I can't think of the brand.

I look at these the same way I look at my spam/anti-virus gateways. Every vendor has pretty much the same claim. "We block 99% of the bad stuff!"... and they all pretty much do. So it comes down to who you like working with the best, and which one has the best interface or additional features that you like.

the_Grinch

Sorry to hijack the thread, but has anyone read the job postings at Sourcefire? They sound awesome:

Work Conditions

Works closely with software reverse engineers and research analysts to quickly develop detection content for all our core applications.
Moderate to high levels of stress may occur at times.
Fast paced and rapidly changing environment.
Extremely talented and experienced team members and mentors.
No special physical requirements.

Constant internal training, drinking games (love it!), and heated discussions.

Another one (that I applied for, but never got a call) said you must have seen True Lies 4 times and be able to quote it! Sounds like a cool place to work!

wastedtime

I have some experience with network IDS systems and just wanted to add too what you said.

From point one, I prefer open and modifiable rule sets as it may be more difficult to ascertain what the rule alerts on. Alerts that don't give you an idea of why it triggered on a piece of information will give you little to go on to determine if it hit on a false positive or not.

With regards to point two, I recommend a network traffic capturing device like Solera or even a home brewed device from a *nix based box. If you have a device like that sitting off of the same tap as your IDS you could get a lot of information from it. Along with this something like flow records or Bro's connection log records would be good to help determine your traffic.

As far as point three, those are two completely different things in my book. They may be similar in some respects but the actions they perform are different. When you are talking about an IPS, you really need to consider the impact to the network where the IDS is passive. Typically an IPS is where you will get that 80% that you mentioned. The upper 20% can be harder to get even with all of this.

When stuff like Morto is hitting the news, it really shows how pour security is in some places. If that isn't getting caught then what about the more advanced threats.

docrice

I'm planning on deploying a set of passive monitors off the same taps that the IPS will see from, but with more aggressive rule sets. I'd also like to have flow records and full content captures as well, but the latter will have to be on a rolling basis as storing all that payload data would be extremely cost-prohibitive. Most likely won't be able to afford a nice solution like Solera. As long as I can get full PCAPs within an investigation response window, then that probably suffices. I believe something like Security Onion gives me a starting point.

However, I'm not totally against closed boxes when discussing IPS. While I don't feel all that comfortable about it since it limits what I can debug from an event, if the vendor solution presents other advantages I might be okay with it. But if I didn't have the freedom to set up network "security cameras" in conjunction with cameras which actually shoot the intruder, the choice of an IPS definitely becomes much more critical for (my) analysis requirements.