Load testing and bench marking

Bl8ckr0uter

I'd like to hear how the some of the more experience network engineers (and network security engineers) go about testing network gear. Span ports from current network? Mock deployments? Packet generators? Test Lans? How do you do it?

Everyone

Test? F' it, we'll do it live! I'll write it, and we'll do it live!

docrice

I think this really depends on what you're trying to accomplish. Do you want to see how a device handles a load of TCP traffic? UDP? Odd protocols? Weird payloads? Non-RFC complaint traffic? Malicious traffic? Digital bombs? Glowing packets with lightsaber energy in the payload?

Interesting you bring this up because I will need to actually do this fairly soon. One tool that I've heard of:

http://www.inversepath.com/ftester.html

Bl8ckr0uter

docrice wrote: »

I think this really depends on what you're trying to accomplish. Do you want to see how a device handles a load of TCP traffic? UDP? Odd protocols? Weird payloads? Non-RFC complaint traffic? Malicious traffic? Digital bombs? Glowing packets with lightsaber energy in the payload?

Interesting you bring this up because I will need to actually do this fairly soon. One tool that I've heard of:

Inverse Path - Research

Have you looked at Ostinato or Hping to test firewall/ids rules/signatures?

Ostinato (Traffic Generator) Demo
Hping - Active Network Security Tool

I am thinking these would be good tools as well.

docrice

Both of those are packet crafters, and I've used them for miscellaneous tests. Ostinato is fine if you need a GUI, but I like Hping better overall. These are just a couple of many tools one can use to test devices, but as I said, it depends what exactly you're trying to do. If you want to really get into custom crafting, Scapy would seem to be a better solution.

One could also replay a bunch of PCAPs with "hacker-injected stuff" via tcpreplay through an IPS to see what it picks up. Then you also have tools like Metasploit, etc.. So the tools used is highly dependent on the device type and what kind of performances issues are being investigated.

Bl8ckr0uter

I've been meaning to play with scapy for like 2 months now. I am rebuilding my laptop and installing scapy and hping are on my to do list. Would you say that they aren't the best for testing throughput and the like?

docrice

I'm no expert at any of these tools, but Hping and Scapy seem best suited for creating specific packets in specific ways. If you want raw throughput, there might be simpler tools.

TTCP
Index of /ttcp/

Iperf
Iperf | Download Iperf software for free at SourceForge.net

I'm sure there are tons others out there.

Bl8ckr0uter

Good deal. +Rep. I'll have to take a look at those.

TBH when I was studying for my CCNA (and CCNA:S) I have always wanted to find accurate tools to measure throughput and TCP performance. These should help a ton. Thanks!

Did you do your IDS study btw?

docrice

I'm still in an evaluation phase of certain parts of our security program. While performance (in terms of throughput and catch-rate) is important, another area that's also important is the existing staff's ability to utilize the management interface, integration with reporting tools, etc., which need to be factored in. I don't hear a lot of talk about these when I hear of IDS / IPS as most people tend to focus more on pure detection, understandably.

Bl8ckr0uter

That's really interesting that you say that. Do you feel that you need more indepth monitoring than say, syslog, netflow and dpi on the firewalls can provide?

docrice

In a more ideal world, while I think having logs and NetFlow data help tremendously, nothing beats having actual payload evidence as well. If there's an incident with a machine involved in some kind of questionable activity that gets caught by a sensor, producing a report that shows Joe from Marketing is doing something against policy requires justifying that finger-pointing. Logs are as only as good as the device's ability to produce detailed information about the event. NetFlow shows the who-to-who and on what ports at what times, which is a second plane of correlating evidence. A filtering device's ability to detect (and optionally block) is heavily dependent on the vendor's technology to properly understand applications and protocols to do the right thing. We know how that story goes.

We still have to live with false positives / negatives being commonplace. At the end of the day, what I really want is payload information and all the traffic stream(s) which support what the logs say. At a host level, I'm sure you could get into keystroke and memory monitoring, but that's probably pushing it for most organizations. Nevertheless, if you could get data from end-user systems (or through corresponding central logging) that said process x launched at 17:48pm PT by user account y and be able to tie it together with other log systems along with real packet traces from the wire, that really would help weigh in on conclusions without having to speculate (or be perceived as being biased).

While having fancy equipment and an army of open source tools is great, the analyst ultimately has to rely on data. If there's not enough reliable data, it's hard to drill down to the root cause in a time-efficient way. Part of that work experience is the management interface which drives the investigation.

Bl8ckr0uter

I was reading an article on InfoSec Island that basically said that SIEM is dead and that people are going away from the model of monitoring. I know many people never look at packets at all and would rather rely on netflow and stats.

docrice

Which article is it? I'd be interested in reading that viewpoint, but I've been taught otherwise. NetFlow and stats are great, but to generalize events based simply on those sources of data would grossly neglect visibility into what's really happening on the wire. Otherwise you're trusting stats to show you the needle in the haystack. The devil's always in the details, especially when you're talking about malware, covert channels, insider leaks, etc..

Bl8ckr0uter

https://www.infosecisland.com/blogview/16427-SIEM-An-Epitaph-Blog-Post.html

-Thats the article about SIEM. I'll have to find the thing about packets.

docrice

I think this is another one of those things where a solution (such as SIEMs) were marketed as the answer to visibility. While I agree that SIEMs haven't fulfilled this, at the same time the fundamental problem is floods of data and making sense of it. Part of the shortcoming is technology, but it's also people, policy, and process. IT environments are incredibly complex systems with a zillion moving parts (hardware, OS, processes in memory, timestamps, flows, traffic, events of interest based on anomalies and signatures, etc.). To expect something like a SIEM to perform magic I think is unrealistic.

I think another underlying problem is that most IT shops simply don't have control over their networks. They're always playing catch-up, lack a real understanding of what should be on the wire and in what form, never establish baselines due to other priorities and deadlines, etc.. We rely much more on blacklisting than whitelisting. That's a losing war, but that's the reality for now.

Most shops probably don't have the resources to figure out their environment well, let alone monitor it effectively. We get in the habit of relying too much on vendor sauce to determine events.

These days modern information security requires far more than just log and event data - it requires the ability to collect and analyze ALL network security data, in real-time.

SIEM leaves a legacy on which information security can build. There will be a collection - all security data is welcomed.

I think this nails it on the head. A SIEM is only as good as the data fed to it. Logs aren't everything. Real-time is great, but so many things can happen in a minute. Reaction time will be based on how well you're prepared to handle virtually unlimited kinds of events. It's one thing to have high-res video cameras placed in every corner of every building, but if you only have one security guard watching the entire campus, you're in trouble.

After taking TCP/IP Weapons School 3.0, I was pretty much convinced that an effective security program requires good data. Logs by themselves aren't the answer, because you have the dependency of proper auditing levels to produce the right kind of data, and other points of evidence collection to support assertions. You also need someone who can make sense of it all, and that's not what a SIEM can do. Except maybe IBM's Watson, who knows.

http://www.youtube.com/watch?v=UM4ZrsOjmNQ

Zentraedi

Was recently looking at this load generator to do some tests:

XM12 - 12 slot high density, high performance chassis :: Chassis :: Ixia - Enabling a Converged World

Bl8ckr0uter

Wow that looks expensive. I'll take two please!

Turgon

Iperf. But there are many tools out there. Suggest you start by defining what your requirements of the device in question actually are before putting plans for a testing rig together. Once you know what to test, the testing resources needed to deliver that become more obvious. If you are deploying a lot of gear and a new topology, build it in a lab environment. A reference model like that is always a good idea and keep it off your production network.