I have updated my PcapParser to support more options and have included a web interface. You can download the latest version here. The web interface uses a php extension that you must install that verifies bpf syntax that is passed as userinput. I'm using pfring so if you are not look at the README in the bpfcompile subdirectory for instructions. The perl script also now requires Net::Pcap and Mail::Sendmail. The updated version also has a configuration file that usually lives at /etc/pcapp/pcapp.conf.

All of the options can also be passed as command line options. Anything passed via command line overrides what is in the config file.

The pcap parser will work with or without the web interface. The conf file has to modified to fit your environment.

If you are using the web interface you must also modify the processpcap2_conf.php to supply the directories where your argus and pcap files are stored. These should be the same as your pcapdir and argusdir in your pcapp.conf file

Sample command line usage...

In this example we are going to use all argus files to extract sessiondata about our attacker and then use that to determine which out of all of our pcap files traffic resides in. The traffic is then merged into a single pcap and then tcpflow,chaosreader,afterglow and honeysnap are run against the pcap. The files are then md5sum'd and the output of these runs are put into a tar.bz file with a web index.

/usr/bin/parsep4.pl -ip="192.168.1.1" -netmask="32" -argusnum=0 -pcapnum=0 -dotcpflow=yes -domd5deep=yes -dochaosreader=yes -doafterglow=yes -dohoneysnap=yes

This is the same as above although now we are using a bpf to see all tcp traffic that is not 80,443,20, or 21, and we are only looking through the last 1 argus file i.e. today's traffic.

/usr/bin/parsep4.pl -bpf="tcp and not port 80 and not port 443 and not port 21 and not port 20" -argusnum=1 -pcapnum=0 -dotcpflow=yes -domd5deep=yes -dochaosreader=yes -doafterglow=yes -dohoneysnap=yes

Pcap Parser

Written by WilliamMetcalf

parsep-extend-range.pl Your friendly neighborhood PCAP parser

Part of my job is to watch our organizations IDS/IPS sensors and respond to alerts. If you know who I am, I don't think that it will come as a surprise to you that we use snort to monitor our little corner of the metaverse.

A problem I was continuously running into when investigating alarms generated with snort as with almost any IDS/IPS is that often times all you have to work with is payload from the single packet that triggered the alarm and maybe logs on a server. I realize that if you configure a snort rule to do so, you can tag other interesting traffic from a would-be attacker after an alarm has fired. Sguil provides a means for extracting an entire session based on the packet that generated the alarm via log_packets.sh. You can also use sguil to extract individual sessions not generated by an alert if you are logging with sancp.

While this is fine it lacked some features that I really desired such as the output generated by honeysnap , afterglow, argus , etc. The other issue that I ran into is that even though I was performing full packet capture, tools such as ethereal and tcpdump don't support wildcard 's so if you have directory with 100 1 gig pcaps and you want to extract all traffic to/from an attacker out of these pcaps you are shit out of luck unless you pass your bpf to tcpdump one pcap at a time. The first iteration of parsep was to search every pcap that matched a given a wild card for traffic from a specified ip address /netmask, save the data to a temp file and then put it all back together into one uber-pcap using mergecap for analysis. This was fine if you only have to parse a couple of gigs of pcaps, but obviously does not scale well to a couple hundred gigs of pcaps. The latest iteration that I'm releasing optionally uses session data from argus based on time to determine in what pcaps traffic from our attacker resides. So instead of having to parse 200 gigs of pcaps you may only have to parse 1 gig of session data and based on it's output parse only the 10 1 gig pcaps that actually contain data from the attacker. Hmmm that is a really horrible description, how about we just get to some examples.

Usage: parsep-extend-range.pl argus pcap

Example: perl parsep-extend-range.pl 127.0.0.1 32 argus 0 /var/log/sessiondata/argusfile.* pcap 0 /var/log/fullcap/daemonlogger.*

./parsep-extend-range.pl 127.0.0.1 32 argus 2 /var/log/sessiondata/argusoutput.* pcap 0 /var/log/fullcap/daemonlogger.*

removed /var/log/sessiondata/argusoutput.ra.5 from argus search list removed /var/log/sessiondata/argusoutput.ra.4 from argus search list removed /var/log/sessiondata/argusoutput.ra.3 from argus search list removed /var/log/sessiondata/argusoutput.ra.2 from argus search list removed 4 argus files out of 6 so that only last 2 remain argusfile here is /var/log/sessiondata/argusoutput.ra.1 /var/log/sessiondata/argusoutput.ra making dir to store data not removing any files from pcap filemask array finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra.1 finding proper pcaps from argus session data in file /var/log/sessiondata/argusoutput.ra file list before dup removal putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190640570 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190647914 into new file array putting file /var/log/fullcap/daemonlogger.pcap.1190648607 into new file array revised file list /var/log/fullcap/daemonlogger.pcap.1190640570 /var/log/fullcap/daemonlogger.pcap.1190647914 /var/log/fullcap/daemonlogger.pcap.1190648607 searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190640570 reading from file /var/log/fullcap/daemonlogger.pcap.1190640570, link-type EN10MB? (Ethernet) searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190647914 reading from file /var/log/fullcap/daemonlogger.pcap.1190647914, link-type EN10MB? (Ethernet) searching for 127.0.0.1/32 in file /var/log/fullcap/daemonlogger.pcap.1190648607 reading from file /var/log/fullcap/daemonlogger.pcap.1190648607, link-type EN10MB? (Ethernet) merging pcaps generating connection graph using afterglow reading from file 127.0.0.11190681713/127.0.0.1.pcap, link-type EN10MB? (Ethernet) No property file specified, using default settings. Not a color: generating argus file from merged pcap outputing session data to text file exporting tcpflowdata generating honeysnap data creating tarball 127.0.0.11190681713.tgz removing temp data we got you now sucka

In the example above we are storing about five day's of session data with argus and 200 gigs of pcaps using daemonlogger. I know that it is dumb but the script requires that full pcaps have a filename of something.something.date for example daemonlogger.pcap.1190647914 The script I use for daemonlogger is included below

#!/bin/sh . /etc/init.d/functions case "$1" in start) echo -n "Starting daemonlogger: " /usr/local/bin/daemonlogger -r -d -i br0 -l /var/log/fullcap/ -m 200 -S 1515 touch /var/lock/daemonlogger sleep 3 echo ;; stop) echo -n "Stopping daemonlogger: " killproc daemonlogger rm -f /var/lock/daemonlogger echo ;; restart) $0 stop $0 start ;; status) status daemonlogger ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 esac

exit 0

The script I use for argus is below along with the logrotate file since argus doesn't have any sort of built-in ringbuffer functionality like daemonlogger or tshark. Because of the logrotate file causes our older session data to be added first to the array we flip the array to cause new session data to be at the front of the array.

#!/bin/sh . /etc/init.d/functions case "$1" in start) echo -n "Starting argus: " /usr/local/sbin/argus -d -p -c -J -w /var/log/sessiondata/argusoutput.ra -i br0 touch /var/lock/argus sleep 3 echo ;; stop) echo -n "Stopping argus: " kill `cat /var/run/argus.pid` echo ;; restart) $0 stop $0 start ;; status) status argus ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 esac

exit 0

Logrotate file

/var/log/sessiondata/argusoutput.ra { rotate 5 missingok nocompress daily postrotate /etc/init.d/argusd restart >/dev/null 2&>1 endscript }

I hope somebody finds this useful ;-), If not, oh well it is useful to me. If you look at the script you can see how easy it is to add support for almost any tool that you want to run the merged pcap through. If you don't understand the importance of collecting full content captures along with session data I suggest that you pickup one or all of Richard Bejtlich's books on NSM.

Matt Jonkman of bleedingthreats.net was kind enough to host the script for me. He is truly a king among men ;-)....


  • sentinal.tar.bz2: Automatically generate PCAP parser information based on barnyard csv output
Topic attachments
I Attachment Action Size DateSorted ascending Who Comment
Unknown file formatbz2 parsep-extend-range.pl.bz2 manage 3.4 K 2007-10-08 - 12:40 WilliamMetcalf It is a multiple PCAP file parser
Unknown file formatbz2 pcapp.tar.bz2 manage 330.8 K 2009-02-16 - 22:33 WilliamMetcalf Version 4 of PcapParser with Web Interface
Unknown file formatbz2 pcapp-0.1.tar.bz2 manage 335.4 K 2009-04-08 - 16:07 WilliamMetcalf Small Fixes to Version 4
Unknown file formatbz2 sentinal.tar.bz2 manage 1.6 K 2009-04-08 - 17:48 WilliamMetcalf Automatically generate PCAP parser information based on barnyard csv output
Topic revision: r7 - 2009-04-08 - WilliamMetcalf
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © Emerging Threats