We have finished up our first round of testing against the modified version of the Sguil client (we have modified the 0.7.0 CVS version). Using the alert information displayed in the Sguil client we create a query and feed it into the NetWitness API through a vbscript which calls explorer.exe and passes it a NetWitness URL. When you install NetWitness Investigator, it registers the nw://<url> as a protocol within the OS. This URL is the API/method by which you can use alerting from other products to find specific sessions, ip’s or timeframes of traffic to review in any combination.
To do this, we first modified the Xscript section of sguil.tk and removed the transcript and wireshark options as we are now relying upon NetWitness for pcap capture instead of daemonlogger/sancp/tcpdump etc:
You can see that we are calling the command NetWitnessEvent and passing it a value of from or to. The reason for this is that events that are triggered list the source and destination IP address for the particular packet that caused the alert. However, NetWitness is session aware, so you may need to query using the source address as the destination and vice versa. This is calling the NetWitnessEvent function that we have added to lib/extdata.tcl:
proc NetWitnessEvent { direction } {
global ACTIVE_EVENT SERVERHOST XSCRIPT_SERVER_PORT DEBUG CUR_SEL_PANE XSCRIPTDATARCVD
global socketWinName SESSION_STATE WIRESHARK_STORE_DIR WIRESHARK_PATH
if {!$ACTIVE_EVENT} {return}
set selectedIndex [$CUR_SEL_PANE(name) curselection]
set sidcidList [split [$CUR_SEL_PANE(name) getcells $selectedIndex,alertID] .]
set cnxID [lindex $sidcidList 1]
set sensorID [lindex $sidcidList 0]
set proto [$CUR_SEL_PANE(name) getcells $selectedIndex,ipproto]
set srcIP [$CUR_SEL_PANE(name) getcells $selectedIndex,srcip]
set srcPort [$CUR_SEL_PANE(name) getcells $selectedIndex,srcport]
set dstIP [$CUR_SEL_PANE(name) getcells $selectedIndex,dstip]
set dstPort [$CUR_SEL_PANE(name) getcells $selectedIndex,dstport]
if { $CUR_SEL_PANE(format) == "SSN" } {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,starttime]
} else {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,date]
}
set future [clock scan "2 minute" -base [clock scan $timestamp -gmt 1]]
set past [clock scan "-2 minute" -base [clock scan $timestamp -gmt 1]]
set future [clock format $future -format {%Y-%m-%d+%H:%M:%S} -gmt 1]
set past [clock format $past -format {%Y-%m-%d+%H:%M:%S} -gmt 1]
set future [regsub -all -expanded {[\:]} $future {%3A}]
set past [regsub -all -expanded {[\:]} $past {%3A}]
if { $proto == "6" } {
if { $direction == "from" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=TCP+%7C%7C+$srcIP%3A$srcPort+-%3E+$dstIP%3A$dstPort&time=$past+to+$future&view=session&where=ip.src%3D$srcIP+%26%26+tcp.srcport%3D$srcPort+%26%26+ip.dst%3D$dstIP+%26%26+tcp.dstport%3D$dstPort"
}
if { $direction == "to" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=TCP+%7C%7C+$dstIP%3A$dstPort+-%3E+$srcIP%3A$srcPort&time=$past+to+$future&view=session&where=ip.src%3D$dstIP+%26%26+tcp.srcport%3D$dstPort+%26%26+ip.dst%3D$srcIP+%26%26+tcp.dstport%3D$srcPort"
}
}
if { $proto == "17" } {
if { $direction == "from" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=UDP+%7C%7C+$srcIP%3A$srcPort+-%3E+$dstIP+%3A+$dstPort&time=$past+to+$future&view=session&where=ip.src%3D$srcIP+%26%26+udp.srcport%3D$srcPort+%26%26+ip.dst%3D$dstIP+%26%26+udp.dstport%3D$dstPort"
}
if { $direction == "to" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=UDP+%7C%7C+$dstIP%3A$dstPort+-%3E+$srcIP+%3A+$srcPort&time=$past+to+$future&view=session&where=ip.src%3D$dstIP+%26%26+udp.srcport%3D$dstPort+%26%26+ip.dst%3D$srcIP+%26%26+udp.dstport%3D$srcPort"
}
}
if { $proto == "1" } {
if { $direction == "from" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ICMP+%7C%7C+$srcIP+-%3E+$dstIP&time=$past+to+$future&view=session&where=ip.src%3D$srcIP+%26%26+ip.dst%3D$dstIP+%26%26+ip.proto%3D1"
}
if { $direction == "to" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ICMP+%7C%7C+$dstIP+-%3E+$srcIP&time=$past+to+$future&view=session&where=ip.src%3D$dstIP+%26%26+ip.dst%3D$srcIP+%26%26+ip.proto%3D1"
}
}
}
This function will create different queries based upon protocol type (TCP/UDP/ICMP only currently) and use the source/destination address and source/destination port. It will look for sessions that match those specific values and then automatically open them in NetWitness Investigator:
To replicate the SANCP session type queries, we again modify sguil.tk but this time we modify the IPQuery Menu section:
You can see we are calling a proc/function called NetWitness and are passing it a variable for which address(es) we are interested in (and if they are source or destination addresses) along with some predefined time periods. You have much better flexibility and control if you actually create these queries within NetWitness directly, but just being able to right click makes for greater ease of use for analysts. This is calling the NetWitnessEvent function that we have added to lib/extdata.tcl:
proc NetWitness { direction hours } {
global ACTIVE_EVENT SERVERHOST XSCRIPT_SERVER_PORT DEBUG CUR_SEL_PANE XSCRIPTDATARCVD
global socketWinName SESSION_STATE WIRESHARK_STORE_DIR WIRESHARK_PATH
if {!$ACTIVE_EVENT} {return}
set selectedIndex [$CUR_SEL_PANE(name) curselection]
set sidcidList [split [$CUR_SEL_PANE(name) getcells $selectedIndex,alertID] .]
set cnxID [lindex $sidcidList 1]
set sensorID [lindex $sidcidList 0]
set proto [$CUR_SEL_PANE(name) getcells $selectedIndex,ipproto]
set srcIP [$CUR_SEL_PANE(name) getcells $selectedIndex,srcip]
set srcPort [$CUR_SEL_PANE(name) getcells $selectedIndex,srcport]
set dstIP [$CUR_SEL_PANE(name) getcells $selectedIndex,dstip]
set dstPort [$CUR_SEL_PANE(name) getcells $selectedIndex,dstport]
if { $CUR_SEL_PANE(format) == "SSN" } {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,starttime]
} else {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,date]
}
if {$hours == 1} {
set future [clock scan "30 minute" -base [clock scan $timestamp -gmt 1]]
set past [clock scan "-30 minute" -base [clock scan $timestamp -gmt 1]]
} else {
set hours [expr $hours / 2]
set future [clock scan "$hours hour" -base [clock scan $timestamp -gmt 1]]
set past [clock scan "-$hours hour" -base [clock scan $timestamp -gmt 1]]
}
set future [clock format $future -format {%Y-%m-%d+%H:%M:%S} -gmt 1]
set past [clock format $past -format {%Y-%m-%d+%H:%M:%S} -gmt 1]
set future [regsub -all -expanded {[\:]} $future {%3A}]
set past [regsub -all -expanded {[\:]} $past {%3A}]
if { $direction == "Src" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ip.src%3D$srcIP&time=$past+to+$future&where=ip.src%3D$srcIP"
}
if { $direction == "SrcAsDst" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ip.dst%3D$srcIP&time=$past+to+$future&where=ip.dst%3D$srcIP"
}
if { $direction == "Dst" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ip.dst%3D$dstIP&time=$past+to+$future&where=ip.dst%3D$dstIP"
}
if { $direction == "DstAsSrc" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=ip.src%3D$dstIP&time=$past+to+$future&where=ip.src%3D$dstIP"
}
if { $direction == "SrcToDst" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=$srcIP+-%3E+$dstIP&time=$past+to+$future&where=ip.src%3D$srcIP+%26%26+ip.dst%3D$dstIP"
}
if { $direction == "DstToSrc" } {
exec wscript c:/users/analyst/nw.vbs "nw://broker/?name=$dstIP+-%3E+$srcIP&time=$past+to+$future&where=ip.src%3D$dstIP+%26%26+ip.dst%3D$srcIP"
}
}
Now we can right click on IP’s within Sguil and use the alert data to perform these SANCP queries into NetWitness as shown below:
You may have noticed that the nw://<url>’s are being passed to a visual basic script entitled nw.vbs within the analyst accounts home directory. We had some issues with executing long length commands from within TCL and ran into 8.3 filename limitations as well. The vbscript is very simple and uses the run method to execute explorer.exe while passing it the URL we have formed to perform the query in NetWitness Investigator. If NetWitness Investigator is not running, it will open up and prompt you for your authentication credentials. Additionally, if is already open it will just create a tab in the investigator and display you the sessions/reports. The contents of the nw.vbs file are as follows, it may look weird butyou have to escape quotes with quotes when you do vb scripting so it looks like you have gone quote crazy:
Well the good folks over at SecurixLive.com have already fixed this and a few other little things and rolled it into their latest release of Barnyard2 v1.8-beta1. Go and get it!
We have identified an issue with Barnyard2 version 1.7 build 255 that causes duplicate entries to be created in the Sguil database. The issue was that the entries in the unified2 output from Snort will be read into two seperate lists (AlertList and LogList) as the unified2 log file is processed. Elsewhere in the Barnyard2 code, all the relavent information is stored into a buffer for creating the Sguil database entry for the alert (summary of event) and the log (offending packet) prior to firing off the output logging. The debugging output shows you that the SPECIAL style output is going to be used for the event/packet combo that was pulled out of the unified2 output file:
So the debugging shows us only firing off once with SPECIAL style (Packet+Event). But if we look at the output presented to STDOUT, we see that the same alert inserted twice into the Sguil server (we have omitted packet contents for brevity):
So this lead us to inspect src/spooler.c which lead us to src/plugbase.c CallOutputPlugins() function. While reviewing it, we notice that if the OUTPUT_TYPE__SPECIAL (which Sguil is) is set, then it will process idx-func() once for the entry in the AlertList and once for the LogList:
So if we remove the processing the second time around for the LogList, we are able to stop the double entries from being created in the database. Below is what the patch (which you can download here) looks like. Obviously, you can just delete the lines with the single minus sign in front of them that are located within the CallOutputPlugins() function of src/plugbase.c:
To apply the patch, move it to the src/ directory within the Barnyard2 v 1.7 distribution directory and perform the following then compile/install as normal:
#patch plugbase.c < plugbase.c.patch
Additional things we discovered during the troubleshooting process were how to enable the debugging within Barnyard2. You must first compile it with the --enable-debug option, then prior to running it you must set the environment variable BARNYARD2_DEBUG to one of the numbers listed in src/debug.h (if you want everything, and that IS a lot, just do export BARNYARD2_DEBUG=4294967295).
We will be deploying NetWitness soon and we have been looking for how to leverage it for the packet capture portion of our new centralized Sguil deployment instead of sancp or daemonlogger. We have come up with a way, all be it a bit hackish, of modifying the Sguil client to allow you to view the pcap/session data from within NetWitness Investigator.
First, we modified client/sguil.tk and added the following line under the following line underneath the section of code notated by the comment # Xscript Menu:
This will provide us with a NetWitness menu option where you normally see your Wireshark and Get Transcript options within the Sguil client:
Now that we have that, we need code that will do something when this option is selected. For that we need to add the following to the end of the client/lib/extdata.tcl file:
proc NetWitness { } {
global ACTIVE_EVENT SERVERHOST XSCRIPT_SERVER_PORT DEBUG CUR_SEL_PANE XSCRIPTDATARCVD
global socketWinName SESSION_STATE WIRESHARK_STORE_DIR WIRESHARK_PATH
if {!$ACTIVE_EVENT} {return}
set selectedIndex [$CUR_SEL_PANE(name) curselection]
set sidcidList [split [$CUR_SEL_PANE(name) getcells $selectedIndex,alertID] .]
set cnxID [lindex $sidcidList 1]
set sensorID [lindex $sidcidList 0]
set proto [$CUR_SEL_PANE(name) getcells $selectedIndex,ipproto]
set srcIP [$CUR_SEL_PANE(name) getcells $selectedIndex,srcip]
set srcPort [$CUR_SEL_PANE(name) getcells $selectedIndex,srcport]
set dstIP [$CUR_SEL_PANE(name) getcells $selectedIndex,dstip]
set dstPort [$CUR_SEL_PANE(name) getcells $selectedIndex,dstport]
if { $CUR_SEL_PANE(format) == "SSN" } {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,starttime]
} else {
set timestamp [$CUR_SEL_PANE(name) getcells $selectedIndex,date]
}
exec wscript c:/users/user/test.vbs "nw://test?collection=test&time=All+Data&more-states=&more-all-states=&name=$srcIP+%3E+$dstIP+%3A+$dstPort&where=ip.src%3D$srcIP+%26%26+ip.dst%3D$dstIP+%26%26+tcp.dstport%3D$dstPort&view=session"
}
Now you will notice at the second to last line we are executing a vbscript called test.vbs in c:\users\user\. The contents of that file are as follows (and yes, all those quotes are necessary as you escape a quote with another quote when writing vbscript):
When this gets executed, it will have the NetWitness url (nw://<url>) with the source ip, destination ip and destination port, passed to the Windows shell (explorer.exe). We will be going back and adding time into the mix as well once our actual NetWitness deployment is up and running. We are currently just testing and demonstrating for proof of concept using the free version of NetWitness Investigator 9:
Now, this definately does not seem like the most direct or correct way to do this. However, we discovered some odd behavior that lead us down this path. If you attempt to pass the URL directly to the NwInvestigator.exe binary, it will crash it. If you attempt to pass the URL directly to explorer.exe from within the TCL script, it only opens up explorer but it does not open up NetWitness Investigator. I believe it has something to do with the quoted arguments and how they are passed, but I could not fix it as I know little of TCL and even less about how it works on Windows.
We really do love Sguil, but the client and server lack a few desirable things. As far as I can tell, there is only one SQL connection shared between the server and all the clients connecting to it. Obviously if someone runs a SANCP query that is a little over the top, until it comes back everyone else cannot query the database. This would make sense as the Sguil server requires the non-threaded version of TCL to work. LAMP based applications don’t run into this single connection for all users bottleneck as Apache and MySQL are multithreaded and each request will create its own (or multiple) database connection for the POST/GET and then close it on completion. I already knew about SQuerT, but it lacks a few things, such as an authentication mechanism. There also is a web client directly associated with the Sguil project, but it was never fully completed and most of the application is a mock up. Lastly, the current Sguil clients will not be that easy to integrate with our upcoming NetWitness deployment. The combination of which, should be quite awesome.
So I have decided to start developing, what I am calling for now, jSguil. It will be along the lines of SQuerT in that it is not to be a function for function replacement for the standard TCL Sguil client. This basically gives me an excuse to actually buckle down and learn JSON/jQuery/webtwodotoh. It will be written in PHP and utilize jQuery/jqgrid. Development so far has been slow and painful as I continiously learn how to actually write PHP/JavaScript (sort of). It end up with me having to redo the entire thing I just spent two hours on due to discovering how to do something in a much better manner. Below is a screenshot of just the SANCP queries you can run. You can sort the results of each column by clicking on it. You may also notice the data is paginated and is currently allowing you to view 50/200/500 records at a time. Every time you sort or get a new page, another query is executed for only the number of records you have chosen to display. This keeps the response times quite fast, even on large resultsets.
If you have ever tried to use the Sguil client’s reverse DNS under the IP Resoluation tab and noticed that it caused the application to be unresponsive, here is the reason why. Tcl uses TCP for DNS by default. So if your DNS server does not allow TCP DNS, the client just sits there endlessly attempting to create the TCP socket connection everytime it tries to do a gehostbyaddress(). You can review the configuration by doing the following inside of tclsh:
If you want the client to do UDP DNS queries, you have to ensure you have the tcludp package installed. With Ubuntu, the package is named libudp-tcl. So the following should get you Ubuntu guys where you need to go:
There is quite a lot of documentation provided with the Napatech cards if you are a customer, but the default configs provided aren’t what you want to use to hit the ground running for IDS setups. To configure the card to split the traffic up into 8 streams by hashing the headers, create a /opt/napatech/config/custom.cfg file that contains the following. We of course are assuming you have installed the necessary hardware/software and compiled basic tools like tcpdump against the Napatech provided libpcap:
#=NTPL=#
DeleteFilter=All
HashMode=Hash5TupleSorted
SetupPacketFeedEngine[TimeStampFormat=PCAP;DescriptorType=PCAP;MaxLatency=1000;SegmentSize=4096;Numfeeds=8]
PacketFeedCreate[NumSegments=16;Feed=(0..7)]
Capture[Priority=0;Feed=(0..7)] = ALL
Just something thrown together quickly for use with our Ubuntu 8.04 LTS systems to initialize the card and configure it on boot. You must set CONFIG to be your config file or else the script will just exit. This init script assumes you have only one card in the box and it is ntxc0. Don’t forget to create your symlinks S##napatech and K##napatech into your /etc/rc*.d/ directories. Also don’t forget to ensure that your IDS processes are starting up after this gets loaded or else the ntxc0:# streams won’t be available to monitor. The “status” command pulls the log entries from the card. If running that status command yeilds >>> Error: No adapters could be found., then the driver is not loaded.
#!/bin/sh -e
#### BEGIN INT INFO
# Provides: Napatech
# Required-Start: mountkernfs $local_fs
# Required-Stop: $local_fs
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Loads Napatech network card
# Description: Napatech kernel driver and configuration
### END INIT INFO
#
# Author: Eoin Miller (eoin[dot]miller[at]trojanedbinaries[dot]com)
#
set -e
Had a meeting today and someone clued me into the existance of VSS Monitoring makes taps that use a hashing algorithm to distribute the load of 10G taps across several monitoring ports. This method ensures that all the packets of a TCP session is distributed to the same monitoring port. This is much like what stream capable cards from vendors such as Endace and Napatech perform. This definately caught my attention as this is the first tap I have seen with this level of functionality. In my previous post I wrote about using a single stream capable card that uses a hashing algorithm to be combined with Daemonlogger and quite a few quad head network cards to create a “traffic splitter”. Well it appears VSS Monitoring has actually already created a tap to do just this. I don’t know about the pricing of the devices (standard tap, a stream capable card and commodity server/nics is probably going to be cheaper) but stripped down specialized hardware can have its advantages. Here is a podcast they did outlining the functionality of the taps:
Unfortunately you have to dig into this level to see that this is the functionality and even in the podcast they keep going over filtering. They go over what load balancing is on their site, but they do not have a list of which of their taps support it. As best I can tell, any 10G capable distributed tap they sell has this functionality.
Took a little time today to create a 128×128 vector based logo for Sguil. Below is a PNG version for you to preview it on the web:
You can download the vector based SVG type icon here! Or for you Windows guys, here is a Vista/Windows 7 icon. We delivered a copy of it over to the Sguil maintainers and they may add it in to the CVS, so check in the tree and you may already have it.
We use Daemonlogger as a pcap logger of choice with the Sguil implementation with the securixlive NSMnow projects management scripts. The NSMnow scripts were stopping and restarting the packet capturing every 15 minutes, which causes a great deal of problems when you are trying to investigate and you are missing portions of TCP streams. So we swapped out to using Daemonlogger and would have it stop and start every night after creating sub directories for each day in YYYY-MM-DD format. This patch I wrote stops that from needing to happen. You can download the patch here.
There is still some functionality left to be desired. I plan to update Daemonlogger more so that it reindexes existing files in the dated subdirectories when it starts up and this way it can continue to autoprune the oldest existing file within the logging directory instead of just the files created during the latest time the program was run.
