Pcap Parser

Written by WilliamMetcalf

parsep-extend-range.pl Your friendly neighborhood PCAP parser

Part of my job is to watch our organizations IDS/IPS sensors and respond to alerts. If you know who I am, I don't think that it will come as a surprise to you that we use snort to monitor our little corner of the metaverse.

A problem I was continuously running into when investigating alarms generated with snort as with almost any IDS/IPS is that often times all you have to work with is payload from the single packet that triggered the alarm and maybe logs on a server. I realize that if you configure a snort rule to do so, you can tag other interesting traffic from a would-be attacker after an alarm has fired. Sguil provides a means for extracting an entire session based on the packet that generated the alarm via log_packets.sh. You can also use sguil to extract individual sessions not generated by an alert if you are logging with sancp.

While this is fine it lacked some features that I really desired such as the output generated by honeysnap , afterglow, argus , etc. The other issue that I ran into is that even though I was performing full packet capture, tools such as ethereal and tcpdump don't support wildcard 's so if you have directory with 100 1 gig pcaps and you want to extract all traffic to/from an attacker out of these pcaps you are shit out of luck unless you pass your bpf to tcpdump one pcap at a time. The first iteration of parsep was to search every pcap that matched a given a wild card for traffic from a specified ip address /netmask, save the data to a temp file and then put it all back together into one uber-pcap using mergecap for analysis. This was fine if you only have to parse a couple of gigs of pcaps, but obviously does not scale well to a couple hundred gigs of pcaps. The latest iteration that I'm releasing optionally uses session data from argus based on time to determine in what pcaps traffic from our attacker resides. So instead of having to parse 200 gigs of pcaps you may only have to parse 1 gig of session data and based on it's output parse only the 10 1 gig pcaps that actually contain data from the attacker. Hmmm that is a really horrible description, how about we just get to some examples.

Usage: parsep-extend-range.pl argus pcap

Example: perl parsep-extend-range.pl 127.0.0.1 32 argus 0 /var/log/sessiondata/argusfile.* pcap 0 /var/log/fullcap/daemonlogger.*

./parsep-extend-range.pl 127.0.0.1 32 argus 2 /var/log/sessiondata/argusoutput.* pcap 0 /var/log/fullcap/daemonlogger.*

removed /var/log/sessiondata/argusoutput.ra.5 from argus search list removed /var/log/sessiondata/argusoutput.ra.4 from argus search list removed /var/log/sessiondata/argusoutput.ra.3 from argus search list removed /var/log/sessiondata/argusoutput.ra.2 from argus search list removed 4 argus files out of 6 so that only last 2 remain argusfile here is /var/log/sessiondata/argusoutput.ra.1 /var/log/sessiondata/argusoutput.ra making dir to store data not removing any files from pcap filemask array finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra.1 finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra file list before dup removal putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190648607 into new file array revised file list /var/log/fullcap/daemonlogger.pcap.1190640570 /var/log/fullcap/daemonlogger.pcap.1190647914 /var/log/fullcap/daemonlogger.pcap.1190648607 searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190640570 reading from file /var/log/fullcap/daemonlogger.pcap.1190640570, link-type EN10MB? (Ethernet) searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190647914 reading from file /var/log/fullcap/daemonlogger.pcap.1190647914, link-type EN10MB? (Ethernet) searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190648607 reading from file /var/log/fullcap/daemonlogger.pcap.1190648607, link-type EN10MB? (Ethernet) merging pcaps generating connection graph using afterglow reading from file 127.0.0.11190681713/127.0.0.1.pcap, link-type EN10MB? (Ethernet) No property file specified, using default settings. Not a color: generating argus file from merged pcap outputing session data to text file exporting tcpflowdata generating honeysnap data creating tarball 127.0.0.11190681713.tgz removing temp data we got you now sucka

In the example above we are storing about five day's of session data with argus and 200 gigs of pcaps using daemonlogger. I know that it is dumb but the script requires that full pcaps have a filename of something.something.date for example daemonlogger.pcap.1190647914 The script I use for daemonlogger is included below

#!/bin/sh . /etc/init.d/functions case "$1" in start) echo -n "Starting daemonlogger: " /usr/local/bin/daemonlogger -r -d -i br0 -l /var/log/fullcap/ -m 200 -S 1515 touch /var/lock/daemonlogger sleep 3 echo ;; stop) echo -n "Stopping daemonlogger: " killproc daemonlogger rm -f /var/lock/daemonlogger echo ;; restart) $0 stop $0 start ;; status) status daemonlogger ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 esac

exit 0

The script I use for argus is below along with the logrotate file since argus doesn't have any sort of built-in ringbuffer functionality like daemonlogger or tshark. Because of the logrotate file causes our older session data to be added first to the array we flip the array to cause new session data to be at the front of the array.

#!/bin/sh . /etc/init.d/functions case "$1" in start) echo -n "Starting argus: " /usr/local/sbin/argus -d -p -c -J -w /var/log/sessiondata/argusoutput.ra -i br0 touch /var/lock/argus sleep 3 echo ;; stop) echo -n "Stopping argus: " kill `cat /var/run/argus.pid` echo ;; restart) $0 stop $0 start ;; status) status argus ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 esac

exit 0

Logrotate file

/var/log/sessiondata/argusoutput.ra { rotate 5 missingok nocompress daily postrotate /etc/init.d/argusd restart >/dev/null 2&>1 endscript }

I hope somebody finds this useful ;-), If not, oh well it is useful to me. If you look at the script you can see how easy it is to add support for almost any tool that you want to run the merged pcap through. If you don't understand the importance of collecting full content captures along with session data I suggest that you pickup one or all of Richard Bejtlich's books on NSM.

Matt Jonkman of bleedingthreats.net was kind enough to host the script for me. He is truly a king among men ;-)....


Topic attachments
I Attachment Action Size Date Who Comment
Unknown file formatbz2 parsep-extend-range.pl.bz2 manage 3.4 K 2007-10-08 - 12:40 WilliamMetcalf It is a multiple PCAP file parser
Edit | Attach | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2007-10-08 - MattJonkman
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © Emerging Threats