High Speed IDS Traffic Splitting With Stream Capable Cards and Daemonlogger

I am pontificating upon the problem of doing IDS on high speed (10 gigabit or greater) networks without massive amounts of traffic being missed by the IDS due to the bottleneck of a single core being used for analysis. More specifically, a single stream of traffic is only being able to be analyzed by a single process that only is able to consume the resources of a single core of a multicore CPU. Now I know Snort 3.0 is on the way and has a multithreading model, but reading Mr. Roesch’s blog on this specific subject does not look like the performance boost is going to be that great due to how multicore Intel CPU’s operate. This is more of a theoretical idea (since I haven’t had an opporutinity to do it in real life yet) and there are probably some kinks in what I have diagramed below, but let us just suspend disbelief and pretend this motherboard exists and is capable of such a feat:

So, what we are doing is is taking a 10gigabit network tap connection and running it into a Napatech Network Adapter. The reason for Napatech over other competitor products (such as Endace DAG cards) is that currently the Napatech cards allow you to split the traffic into 32 streams as opposed to Endaces 4 streams. Endace Z-series cards (like those used in the Ninja Z-Boxes) can team together to produce 8 possible streams currently, however this is still only 25% of the capabilities of Napatech’s cards for about double the cost. I have been told by US based resellers that Endace should have new cards to market that should support 32 streams by about the end of the year.

These cards are somewhat expensive (3-6k dollars a pop) but they do something that is extremely useful. They hash the headers of the packets as they come in and move them to a specific stream of data, think of the stream as a virtual network adapter. What is very cool about this is that a TCP session will always hash to the same stream so you have stateful distribution of the streams. So, if you had 100mbit/s of traffic and a single quad core processor in your IDS server, you could split the traffic into four streams run four instances of Snort and attach one to each stream. This means each core of your CPU is only monitoring ~25mbit/s of traffic at any given time and it should be able to keep up with that without dropping any packets quite easily.

Now, if we use a tool such as Daemonlogger in conjuction with this, we can do some interesting stuff. Daemonlogger can act as a “softtap” in that it can copy what it sniffs on one interface and replay it on another (much like a regeneration tap). So, in theory, we can attach a Daemonlogger process to a stream of traffic and have it replay the traffic out another interface so that that stream can be passed to another box with more processors/cores available for individual Snort instances. Effectively, this allows us to construct a traffic splitter out of one commodity server, a few commodity quad head NIC’s, a piece of GPLv2 licensed software and a single specialized network card.

If you started out with a 10gigabit tap (remember that is full duplex so it is 10g transmit and 10g receive for a combined 20gigabit) and were able to use all 32 streams and deliver them to other servers, you would only have to monitor up to 640mbit/s. This maximum number obviously relies upon 100% utilization of RX and TX simultaniously on the link you are monitoring. You could also use more stream capable cards in the servers that Daemonlogger is passing the traffic to and have them break the streams apart even further to pass down to other servers. This cascading of traffic could be done over and over (hardware budget permitting) until you have streams of traffic made into sizes managable enough to be processed by the processing power of a single core of your CPU that is consumed by your IDS. Good idea? I hope so. Now all we need to do is get some 10g in this place for me to play with. On second though, I’ll stick to my slower and more manageable speeds. But if some else does something like this, I would love to read about it.