farrokhi/nfdump
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Initial git release 1.6.13

Browse Source
This commit is contained in:
phaag 2015-10-03 14:06:34 +02:00
parent 752de78504
commit 758237b450
147 changed files with 107413 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
AUTHORS Executable file
View File

@ -0,0 +1 @@
Peter Haag peter.haag@switch.ch

32
BSD-license.txt Normal file
View File

@ -0,0 +1,32 @@
The nfdump project is distributed under the BSD license:
Copyright (c) 2014, Peter Haag
Copyright (c) 2009, Peter Haag
Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the author nor the names of its contributors may be
used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

1
COPYING Symbolic link
View File

@ -0,0 +1 @@
BSD-license.txt

438
ChangeLog Executable file
View File

@ -0,0 +1,438 @@
2014-11-16
- Fix v1 extension size bug
- Add htonll check for autoconf
- Fix AddExtensionMap compare bug
- Fix ipfix templare withdraw problems - free all maps correctly
- Add minilzo 2.08 - fixes CVE-2014-4607
- Cleanup some stat code. more needs to be done ..
- Cleanup man pages for -O -n
- Remove SunPro test in configure - no longer supported anyway
2014-06-15 v1.6.12p1
- Add pblock compare functions
- Update extended filter: Allow modification left/right values
2014-02-16 v1.6.12
- Add NAT pool port allocation
- Modify/fix NAT vrf tags. Add egress vrf ID
- Modify common record due to exporter exhaustion. new common record
type 10 adds 4 extra bytes. Reads v1 common record transparently
- Fix sflow potential crash
2013-11-13 v1.6.11
- Add ASA/NSEL 9.x protcol changes
- Make it llvm compilable
2013-08-12 v1.6.10p1
- Fix -t +/- n timeslot option
- Fix bug in nfanon - stat record update.
- Fix bug in netflow v5 mudule: extension map size wrong.
- Fix bug nfexport: In some cases could result in wrong flow counter.
- Fix nftrack - could coredump in some cases.
2013-05-16 v1.6.10
- Fix SPARC compile/optimise bug
- Add output packet/bytes counter to global stat - importatnt for NSEL flows ASA > 8.5
- Add NSEL filter options xnet
- Modify extension descriptor code for nfdump1.7. Still use 1.6 extension map layout for compatibility
- Add prototype for nfpcapd - pcap -> nfdump collector. Converts traffoc directly to nfdump files.
- Fix bug in ipfix module: uninitialised variable
- Cleanup syslog/LogError calls
- Fix minor non critical bugs and compile issues
2013-03-02 v1.6.9
- Fix some bugs in beta 1.6.9 NSEL code
- Fix bug statistics update with aggreagted flow records
- Fix sflow bug sfcapd stores wrong (ghost) dump by past samples in same sflow datagram
2012-12-31
- Add time received in csv output
- ICMP should handled better now - somewhat
- Implement ASA NSEL records
- Add definitions in nffile and nx for ASA NSEL extensions
2012-11-09 v1.6.8p1
- Add dynamic source directory tree for multiple exporters
- Fix exporter bug: 'too many exporters' with large time windows
- Fix uninitialised exporter sysid in default sampler record - v9
- Fix v9/ipfix cache initialisation with no templates > 1 in same packet
2012-10-26 v1.6.8
- Add ip list option for 'next ip' in filter syntax
- Accept v9 sampler_id in 2bytes
- Fix IPFIX mac address bug - did not get collected
- Add IPFIX packet/octet TotalCount fields 85/86
- Add received timestamp to sflow collector
- Fix long flow duration calculation - 32bit overflow
- Fix v9 sampling ID: allow 2 byte ID
- Add IPFIX options as rfc5101 section-6.2
- Add exporter records for sflow collector
- Fix bug for MAC address printing %idmc and %odmc.
- Add received time stamp extension
- Add recursive format parser. Allows to extend predefined formats.
- Change flow record sorting to heapsort. remove limit 1000
- Merge -m option to -O tstart. -m now depricated.
- Add -O tend. Print order according to tend of flows ascending
- Apply -O print order for printing flow cache. Applies to -A
2012-07-31 v1.6.7-tc-1
- Special version for TC
- Print exporter and sampling records with nfdump -E
- Added exporter and sampling records to file.
2012-07-30 v1.6.7
- Prepare for file catalog in current file format.
- Fix bug in ReadBlock when reading flow from stdin pipe
- Add new more flexible translation engine for v9
- Add nprobe client/server delay fields
- Prepare for NSEL merging
- Fix memory corruption with double -A flags
- Fix bug in nfreader with compat15 mode files
2012-03-12 v1.6.6
- Minor IPFIX bug.
- IPFIX implement template withdraw
- For IPFIX, check packet sequence per template and observation domain
- Fix time window, when no flows collected or no flows matched
while processing
- Fixed typos
- Fix seg fault bug - test for EMPTY_LIST was missing at several places.
2012-02-19 v1.6.6b1
- Fix bps/pps. make it uint64_t, as bps/pps > 4Gb/s overflows.
- In record raw print mode: decode ICMP instead of src/dst ports
- sflow use announced exporter IP instead of sending IP for router ID
- sflow: Ignore extra fill bytes. Do not complain.
- sflow: fix packet length issue.
- Add IPFIX protokoll support
2011-12-31 v1.6.5
- Fix 64bit bug when using byte/packet limits
- for v5 and sampling use 64bit counters to prevent overflow for large sampled flows.
- Fixed Ident printig bug
2011-07-11 v1.6.4
- some code restructuring - prepare for IPFIX module
- Add netflow v1 module. Some routers still use that
- Add %sn, %dn output tags for src/dst networks
- Fix buffer length check in v5.
- Fix export bug: include last flow cache bucket, when exporting
- number in all filter expressions accept hex values
- fix an sflow colletor bug. Missing extension maps in rotated files
- implement extended statistics. Currently ports and bpp distribution
vectors can be collected automatically be nfcapd. Still experimental
2011-02-26 v1.6.3p1
- Fix timebug fix :(, make it a compile time option
- fix v7 sequence errors
2011-02-15
- Zero out unused fields after aggregation
2011-02-05
- Fix SysUptime 32bit overflow in v5 header
- Add fix for strange first/last swap reported by some users.
2011-01-09 v1.6.3
- Fix extension size bug
- Move IP anonymisation to separate binary nfanon
- Fix initialise bug of -o fmt: and not available fields
2010-09-09 v1.6.2
- released
- fixes some sflow bugs in sfcapd
2010-04-28 v1.6.1p0
- Update flow tools converter to build with Google-Code version 0.68.5
- Fix sflow bugs
2010-03-05 v1.6.1
- Fix bug in man page for -t
- Test sampler infos before using them ( nfcapd startup )
- Add sampling tags #34, #35 used by JunOS
- nfexpire: Fix empty .nfsat, when setting limits on an empty directory
- Fix coredump for -B -m (-w) combination
- Optimise some extension map code
2009-12-28 stable v1.6
- Few bug fixes in release candidates rc1, rc2 and rc3
2009-11-16 snapshot-1.6b-20091116
- Update sflow collector with new tags
- Add router IP extension
- Add router ID (engine type/ID) extension
2009-09-30 snapshot-1.6b-20090930
- snapshot bugfix release
2009-11-0801 snapshot-1.6b-20090806
- Add srcmask and dstmask aggregation
- Add csv output mode. -o csv
- Fix some bugs of previous beta
- Add bidirectional aggregation of flows ( -b, -B )
- Add possibility to save aggregated flows into file ( -w )
Note: This results in a behaviour change for -w in combination
with aggragation )
- Extend -N ( do not scale numbers ) to all text output not just summary
- Make extension handling more robust for some moody IOSes.
- Remove header lines of -s stat, when using -q ( quiet )
Note: This results in a behaviour change for -N
- Remove -S option from nfdump ( legacy 1.4 compatibility )
- Make use of log (syslog) functions for nfprofile.
- Move log functions to util.c
2009-06-19 snapshot-1.6b-20090717
- Flow-tools converter updated - supports more common elements.
- Sflow collector updated. Supports more common elements.
- Add sampling to nfdump. Sampling is automatically recognised
in v5 undocumented header fields and in v9 option templates.
see nfcapd.1(1)
- Add @include option for filter to include more filter files.
- Add flexible aggregation comparable to Flexible Netflow (FNF)
- All new tags can be selected in -o fmt:... see nfdump(1)
- topN stat for all new tags is implemented
- Integrate developer code to read from pcap files into stable
- Update filter syntax for new tags
- Added more v9 tags for netflow v9.
The detailed tags are listed in nfcapd(1)
Adding new tags also extended the binary file format with
data block format 2, which is extension based. File format
for version <= 1.5.* ( Data block format 1 ) is read
transparently. Data block 2 are skipped by nfdump 1.5.7.
32bit but AS and interface numbers are supported.
- Add flexible storage option for nfcapd. To save disk space, the
data extensions to be stored in the data file are user selectable.
- Added option for multiple netflow stream to same port.
-n <Ident,IP,base_directory>
Example: -n router1,192.168.100.1,/var/nfdump/router1
So multiple -n options may be given at the command line
Old style syntax still works for compatibility, ( -I .. -l ... )
but then only one source is supported.
- Move to automake for building nfdump
- Switch scaling factor ( k, M, G ) from 1024 to 1000.
- Make nfdump fully 64bit compliant. ( 8bit data alignments and access )
2009-04-17 stable 1.5.8
- Fix daylight summer time bug, when guessing sub dirs. file access ( -M, -r )
- Bug fixes for 64bits CPUs
2008-02-22 stable-1,5.7
- Add icmp type/code decoding
- Add proper icmp v9 decoding
- Fix memory leaks in -e auto expire mode in nfcapd.
- Fix somee potential dead locks with file locking, when expiring
- Fix multicast bug in nfreplay
- Add hostname lookup for IP addresses in filter.
2007-10-15 stable-1.5.6
- Fix odd CISCO behaviour for ICMP type/code in src port.
- Add fast LZO1X-1 compression option (-z) for output file.
- Add lists for port in syntax -> port in [ 135 137 445]
- Add lists for AS syntax -> as in [ 1024 1025 ]
- Bug fix in filter for syntax 'src as and dst as'
2007-08-24 stable-1.5.5
- Fix nfprofile bug, nfprofile crashes when last opts line is not valid for
some reason.
- Fix potential hand for nfexpire, on empty flow directories.
2007-08-08 snapshot-20070808
- Idents may contain '-' in name.
- Fixed install bugs in Makefile.in and configure.in
- Installs now cleanly on Solaris
- Handle 4byte interface numbers in v9. Quick fix: 4bytes reduced to 2bytes.
- Fix aggregation bug in statistics.
- ftok(3) C library call replaced by more reliable own implementation.
Did result in error messages like "Another collector is already running"
- Fix minor bugs iin file range selction -R.
- Add recursive behaviour for -R <directory>
- New option -i can canche Ident descriptor in data files.
2007-03-12 snapshot-20070312
- Bug fix release of 20070306
2007-03-06 snapshot-20070306
- Fix bug in flist.c. Resulted in a coredump when using sub dirs and -R . ( all files )
- Fix minor bug in nfcapd.c.
- Extend nfprofile for alerting system of nfsen - special version of profiles
- Extend nfprofile for shadow profiles.
2007-08-10 snapshot-20070110
- Fix some compiler warnings, when compiled on a 64bit LINUX
- Fixes an sflow bug: IP address was printed in wrong direction. ( lower bits first )
- Add new IP addr taging option -T for easy parsing for nfsen lookups
- Add new IP list for massive address filtering:
syntax: ip in [ 12345 23456 3456 ....]
- Change nfprofile for channel based profiling. This breaks with old nfprofile
functionality.
- Remove space from ICMP type/code when followed by an IP address
2006-07-21 snapshot-20060809
- Make nfexpire ready for profile expiration
- Fix bug in nfrpofile. sub dir hierarchy not handled correctly.
2006-07-21 snapshot-20060721
- Add -N option for plain number output in summary line
2006-07-21 snapshot-20060721
- Do recursive file selection when a directory is given by -R
2006-06-14 snapshot-20060621
- Add srcas/dstas/proto aggregation.
Note: This changes the default aggregation behaviour, but gives more flexibility
- Add tos to element statistics list
2006-06-14 snapshot-20060614
- Add additional stat line at the end of output
- Add new binary nfexpire. Manages data expiry on time and/or size based limits
Includes new bookkeeping records in nfcapd. See nfexpire(1)
- Add ICMP type/code decoding in flow listing instead of dst port
- Add packet repeater in nfcapd/sfcapd. In addition, incoming UDP packets can
be directly forwarded to another IP address/Port. See new option -R
- Add sub directory hierarchies: Files can be stored into various sub dir levels
based on different time formats. see new option -S
- Some minor bug fixes.
- Code cleanup in nfcapd. better daemonize code and communication with launcher.
2006-04-xx v.1.5.1
Fix bug in nfdump.c: Writing anonymized flows to file did not work corretly
stdin input format now compatible with file format, therefore
'nfdump < file' works again as it did in nfdump 1.4.
Fix bug in nfcapd.c: Error handling not correct when receiving a non
recognized netflow packet. Resulted in an endless loop
2006-03-27 snapshot 1.5-20060327
Make all element statistics -s transport layer protocol
independant by default. Add :p to stat name ( e.g. srcip:p ) to
enable transport layer dependant statistics on request.
2006-03-20 snapshot 1.5-20060320
Fix bug in filter engine: 'not flags xyz' produces wrong results
when more than a single flag is specified.
Minor man page fixes.
2006-03-06 v1.5
Fix bug nfcapd. Laucher signaled too early. File not yet properly
closed.
2006-02-14 v1.5-beta-5
Add srcas, dstas, input and output interfaces in aggregated
output.
Fix IPv6 bug in filter: accept 1234:: address.
rename nfcapd.curent tmp file to nfcapd.curren.<pid>. Poorly
configured nfcapd processes may mess up themselves otherwise.
2006-02-02 v1.5-beta-4
Fix netflow v5 dPkts <-> dOctets collector bug.
Update pipe format to include more information
Allow AS number 0 in filter syntax.
Add some more boundary checking - netflow exporters aren't bug free either - sigh ..
2006-01-11 v1.5-beta-3
Fix isnumber incompatibility in grammar.y
Add 'if' statistics
2006-01-10 v1.5-beta-2
nf_common.c Fix bug in format parser.
Extended 'proto <protocol>' syntax to support all protocols
Change time format in summary line to ISO format
2005-12-20 v1.5-beta-1
*.* A lot of internal changes, not mentioned here. :(
nfdump Add subnet aggregation for option -A
A new syntax e.g. srcip4/24, dstip6/64 is supported for subnet wise aggregation.
example: traffic of a whole subnet -A srcip4/24 -s srcip/bytes
nfdump Add more stat element option. -s <stat> now supports:
srcip, dstip, ip, srcport, dstport, port, srcas, dstas, as, inif, outif, proto
nfdump Add -z. Suppress writing flows to data files. Only stat information is written.
nfprofile Used only be nfsen for upcoming shadow profiles. If you don't understand this
simply ignore it.
nfdump Add -q option to suppress header as well as stat information at the bottom
nfprofile for easier post processing with external programms.
nf_common.c Output format processsing rewritting for more flexibility. Besides standard
nfdump.c output formats line, long extended etc., user defined output formats are now
possible and can even be compiled into nfdump for easy access. See -o fmt:<format>
and nfdump.c around line 100.
*.* Integrate netflow v9 into nfdump. Only a subset of v9 is stored into
the data files, basically everything needed for nfdump to work as it did before.
This also includes IPv6 support for any nfdump options. CryptoPAN extended
to work with IPv6. IPv6 condensed output format for better readability.
Output formats available in long and condensed mode: e.g. line -> line6
extended -> extended6
*.* Replace binary data file format. Old format not flexible enough for
upcoming netflow v9/sflow data. *.stat files are gone. The same
information is now available under nfdump -I
New format about 5% larger in size, but faster for reading and writing.
speed gain eaten up by more complex processing - sigh ..
compat14 mode enables transparent reading of old style format.
nffile.[ch] now handles all data file stuff.
nfreplay Multicast enabled:
Add -j <join group>. Joins the specified multicast group ( v4 or v6 )
sending flows to this group.
nfreplay IPv6 enabled:
Add option -4 and -6 to force a specific protocol, otherwise
protocol is automatically selected according the hostname to send flows to.
Add -K key, to send data anonymized, using CryptoPAn
nfcapd Multicast enabled:
Add -j <join group>. Joins the specified multicast group ( v4 or v6 )
for listening.
nfcapd IPv6 enabled:
Add option -4 and -6 for IPv4 and IPv6. By default, listen on IPv4.
Option -b <host/IP> to bind for a specific host/IP address automatically
selects appropriate protocol.
nfnet.c All functions to setup network sockets for listening/sending are
put into this file.
2005-08-22 v1.4
- nfreplay: Bug fix sending flows.
- nfdump: Add CryptoPAn code to anonymize IP addresses. New option -K
- nfdump: Change time format in output to ISO 8601 compatible: e.g. 1981-04-05 14:30:30.100
- nfdump: Add scaling factor k,m,g to number in filter syntax: e.g. bytes > 1m
- nfdump: Create new output format extended with additional fields pps, bps and bpp
- nfdump: Rename output format extended to raw
- nfdump: More than one single flow element statistic ( -s ) is now possible
- nfdump: Add user defined sort order in flow element statistic
- nfdump: Flow element statistic can be ordered by more than one order in the same run
- nfdump: Add pps, bps and bpp fields in flow element statistics
- nfdump: Add more symbolic protocols ESP, AH, GRP and RVSP to filter syntax
- nfdump: Add duration, pps, bps and bpp to filter syntax
- nfdump: Make nfdump miliseconds aware. Older versions skipped msecs.
Binary nfdump file format changed due to this.
output formats changed, due to this.
- nfdump: Add interface in/out if <num> syntax to filter
- nfcapd: Add flow_sequence check. Reports missing flows now.
- nfcapd: Report statistics to syslog LOG_INFO when data file is rotated.
- ft2nfdump: Add ft2nfdump to read netflow data from flow-tools
2005-04-21 v1.3
- Add option -A for more flexible aggregation.
- Correct spelling errors :(
2005-03-04 v1.2.1
Bug fix release
- nfcapd: launcher subprocess may hang on Linux 2.6.x kernels.
Cleaned up interrupt handling.
- nfcapd: fix include order of socket.h and types.h in order to
compile cleanly under FreeBSD 4.x
- nfcapd: clean up syslog logging.
- nfdump: Multiple sources ( -M ) and sort flows ( -m ) with
-c <limit> did not list the correct flows.
- nfprofile: Profiling with multiple sources may produce incorrect
profiles.
2004-12-20 v1.2
- nfcapd handles transparent v5 and v7 flows. v7 gets converted into v5
- nfcapd can execute any command at the end of interval. New option -x
- nfdump Extended filter syntax for flags, to, bytes and packets
- Rearrange output formats in nfdump: new switch -o, remove switch -E
output formats: 'line', 'long', 'extended' and 'pipe'
- More flexible statistic handling in nfdump: cleanup ugly -s -s -s
syntax. Replaced by -s <stat> option. New statistics for Port and AS.
2004-09-20 v 1.1
First public Version.

133
CreateSubHierarchy.pl Executable file
View File

@ -0,0 +1,133 @@
#!/usr/bin/perl
#
#
# Sample script to clean old data.
# Run this script each hour to cleanup old files to make room for
# new data. When max_size_spool is reached the oldest files are
# deleted down to high_water.
#
# Copyright (c) 2004, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# * Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
# * Neither the name of SWITCH nor the names of its contributors may be
# used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# $Author: peter $
#
# $Id: CreateSubHierarchy.pl 77 2006-06-14 14:52:25Z peter $
#
# $LastChangedRevision: 77 $
#
use strict;
use warnings;
use POSIX qw(strftime);
use Time::Local;
use Getopt::Std;
our(
$opt_l, # Data directory
$opt_S, # Sub hierarchy format. Correspondes to -S to nfcapd. See nfcapd(1)
);
getopts('l:S:');
my $subdir_format;
my @subdir_formats = (
"",
"%Y/%m/%d",
"%Y/%m/%d/%H",
"%Y/%W/%u",
"%Y/%W/%u/%H",
"%Y/%j",
"%Y/%j/%H",
"%F",
"%F/%H"
);
sub usage {
print "$0 [options]\n",
" -l datadir Data directory\n",
" -S <num> Sub hierarchy format. Correspondes to -S to nfcapd. See nfcapd(1)\n",
"\n";
exit(0);
}
sub ISO2UNIX {
my $isotime = shift;
$isotime =~ s/\-//g; # allow '-' to structur time string
# 2004 02 13 12 45 /
my $sec = 0;
my ( $year, $mon, $mday, $hour, $min ) = $isotime =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})/;
$mon--;
# round down to nearest 5 min slot
my $diff = $min % 5;
if ( $diff ) {
$min -= $diff;
}
my $unixtime = Time::Local::timelocal($sec,$min,$hour,$mday,$mon,$year);
return $unixtime;
} # End of ISO2UNIX
if ( !defined $opt_l || !defined $opt_S ) {
usage();
}
my $data_dir = $opt_l;
if ( !defined $subdir_formats[$opt_S] ) {
die "Unknown format number $opt_S";
}
$subdir_format = $subdir_formats[$opt_S];
opendir DIR, "$data_dir" || die "Can't open current directory: $!\n";
$| = 1;
print "Reorganizing data files ... ";
while ( my $entry = readdir DIR ) {
next if $entry =~ /^\./;
next unless -f "$data_dir/$entry";
next unless $entry =~ /nfcapd\.(\d{12})$/;
my $date = $1;
my $unix_time = ISO2UNIX($date);
my $sub_path = strftime $subdir_format, localtime($unix_time);
if ( !-d "$data_dir/$sub_path" ) {
print "Need to create '$data_dir/$sub_path'\n";
my @dirlist = split '\/', $sub_path;
my $all_dirs = undef;
foreach my $dir ( @dirlist ) {
$all_dirs = defined $all_dirs ? "$all_dirs/$dir" : $dir;
if ( !-d "$data_dir/$all_dirs" ) {
mkdir "$data_dir/$all_dirs" || die "Can't create subdir '$data_dir/$all_dirs'\n";
}
}
}
rename "$data_dir/$entry", "$data_dir/$sub_path/$entry" || die "Can't move file: $!\n";
}
print "done.\n";

370
INSTALL Normal file
View File

@ -0,0 +1,370 @@
Installation Instructions
*************************
Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation,
Inc.
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved. This file is offered as-is,
without warranty of any kind.
Basic Installation
==================
Briefly, the shell command `./configure && make && make install'
should configure, build, and install this package. The following
more-detailed instructions are generic; see the `README' file for
instructions specific to this package. Some packages provide this
`INSTALL' file but do not implement all of the features documented
below. The lack of an optional feature in a given package is not
necessarily a bug. More recommendations for GNU packages can be found
in *note Makefile Conventions: (standards)Makefile Conventions.
The `configure' shell script attempts to guess correct values for
various system-dependent variables used during compilation. It uses
those values to create a `Makefile' in each directory of the package.
It may also create one or more `.h' files containing system-dependent
definitions. Finally, it creates a shell script `config.status' that
you can run in the future to recreate the current configuration, and a
file `config.log' containing compiler output (useful mainly for
debugging `configure').
It can also use an optional file (typically called `config.cache'
and enabled with `--cache-file=config.cache' or simply `-C') that saves
the results of its tests to speed up reconfiguring. Caching is
disabled by default to prevent problems with accidental use of stale
cache files.
If you need to do unusual things to compile the package, please try
to figure out how `configure' could check whether to do them, and mail
diffs or instructions to the address given in the `README' so they can
be considered for the next release. If you are using the cache, and at
some point `config.cache' contains results you don't want to keep, you
may remove or edit it.
The file `configure.ac' (or `configure.in') is used to create
`configure' by a program called `autoconf'. You need `configure.ac' if
you want to change it or regenerate `configure' using a newer version
of `autoconf'.
The simplest way to compile this package is:
1. `cd' to the directory containing the package's source code and type
`./configure' to configure the package for your system.
Running `configure' might take a while. While running, it prints
some messages telling which features it is checking for.
2. Type `make' to compile the package.
3. Optionally, type `make check' to run any self-tests that come with
the package, generally using the just-built uninstalled binaries.
4. Type `make install' to install the programs and any data files and
documentation. When installing into a prefix owned by root, it is
recommended that the package be configured and built as a regular
user, and only the `make install' phase executed with root
privileges.
5. Optionally, type `make installcheck' to repeat any self-tests, but
this time using the binaries in their final installed location.
This target does not install anything. Running this target as a
regular user, particularly if the prior `make install' required
root privileges, verifies that the installation completed
correctly.
6. You can remove the program binaries and object files from the
source code directory by typing `make clean'. To also remove the
files that `configure' created (so you can compile the package for
a different kind of computer), type `make distclean'. There is
also a `make maintainer-clean' target, but that is intended mainly
for the package's developers. If you use it, you may have to get
all sorts of other programs in order to regenerate files that came
with the distribution.
7. Often, you can also type `make uninstall' to remove the installed
files again. In practice, not all packages have tested that
uninstallation works correctly, even though it is required by the
GNU Coding Standards.
8. Some packages, particularly those that use Automake, provide `make
distcheck', which can by used by developers to test that all other
targets like `make install' and `make uninstall' work correctly.
This target is generally not run by end users.
Compilers and Options
=====================
Some systems require unusual options for compilation or linking that
the `configure' script does not know about. Run `./configure --help'
for details on some of the pertinent environment variables.
You can give `configure' initial values for configuration parameters
by setting variables in the command line or in the environment. Here
is an example:
./configure CC=c99 CFLAGS=-g LIBS=-lposix
*Note Defining Variables::, for more details.
Compiling For Multiple Architectures
====================================
You can compile the package for more than one kind of computer at the
same time, by placing the object files for each architecture in their
own directory. To do this, you can use GNU `make'. `cd' to the
directory where you want the object files and executables to go and run
the `configure' script. `configure' automatically checks for the
source code in the directory that `configure' is in and in `..'. This
is known as a "VPATH" build.
With a non-GNU `make', it is safer to compile the package for one
architecture at a time in the source code directory. After you have
installed the package for one architecture, use `make distclean' before
reconfiguring for another architecture.
On MacOS X 10.5 and later systems, you can create libraries and
executables that work on multiple system types--known as "fat" or
"universal" binaries--by specifying multiple `-arch' options to the
compiler but only a single `-arch' option to the preprocessor. Like
this:
./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CPP="gcc -E" CXXCPP="g++ -E"
This is not guaranteed to produce working output in all cases, you
may have to build one architecture at a time and combine the results
using the `lipo' tool if you have problems.
Installation Names
==================
By default, `make install' installs the package's commands under
`/usr/local/bin', include files under `/usr/local/include', etc. You
can specify an installation prefix other than `/usr/local' by giving
`configure' the option `--prefix=PREFIX', where PREFIX must be an
absolute file name.
You can specify separate installation prefixes for
architecture-specific files and architecture-independent files. If you
pass the option `--exec-prefix=PREFIX' to `configure', the package uses
PREFIX as the prefix for installing programs and libraries.
Documentation and other data files still use the regular prefix.
In addition, if you use an unusual directory layout you can give
options like `--bindir=DIR' to specify different values for particular
kinds of files. Run `configure --help' for a list of the directories
you can set and what kinds of files go in them. In general, the
default for these options is expressed in terms of `${prefix}', so that
specifying just `--prefix' will affect all of the other directory
specifications that were not explicitly provided.
The most portable way to affect installation locations is to pass the
correct locations to `configure'; however, many packages provide one or
both of the following shortcuts of passing variable assignments to the
`make install' command line to change installation locations without
having to reconfigure or recompile.
The first method involves providing an override variable for each
affected directory. For example, `make install
prefix=/alternate/directory' will choose an alternate location for all
directory configuration variables that were expressed in terms of
`${prefix}'. Any directories that were specified during `configure',
but not in terms of `${prefix}', must each be overridden at install
time for the entire installation to be relocated. The approach of
makefile variable overrides for each directory variable is required by
the GNU Coding Standards, and ideally causes no recompilation.
However, some platforms have known limitations with the semantics of
shared libraries that end up requiring recompilation when using this
method, particularly noticeable in packages that use GNU Libtool.
The second method involves providing the `DESTDIR' variable. For
example, `make install DESTDIR=/alternate/directory' will prepend
`/alternate/directory' before all installation names. The approach of
`DESTDIR' overrides is not required by the GNU Coding Standards, and
does not work on platforms that have drive letters. On the other hand,
it does better at avoiding recompilation issues, and works well even
when some directory options were not specified in terms of `${prefix}'
at `configure' time.
Optional Features
=================
If the package supports it, you can cause programs to be installed
with an extra prefix or suffix on their names by giving `configure' the
option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
Some packages pay attention to `--enable-FEATURE' options to
`configure', where FEATURE indicates an optional part of the package.
They may also pay attention to `--with-PACKAGE' options, where PACKAGE
is something like `gnu-as' or `x' (for the X Window System). The
`README' should mention any `--enable-' and `--with-' options that the
package recognizes.
For packages that use the X Window System, `configure' can usually
find the X include and library files automatically, but if it doesn't,
you can use the `configure' options `--x-includes=DIR' and
`--x-libraries=DIR' to specify their locations.
Some packages offer the ability to configure how verbose the
execution of `make' will be. For these packages, running `./configure
--enable-silent-rules' sets the default to minimal output, which can be
overridden with `make V=1'; while running `./configure
--disable-silent-rules' sets the default to verbose, which can be
overridden with `make V=0'.
Particular systems
==================
On HP-UX, the default C compiler is not ANSI C compatible. If GNU
CC is not installed, it is recommended to use the following options in
order to use an ANSI C compiler:
./configure CC="cc -Ae -D_XOPEN_SOURCE=500"
and if that doesn't work, install pre-built binaries of GCC for HP-UX.
HP-UX `make' updates targets which have the same time stamps as
their prerequisites, which makes it generally unusable when shipped
generated files such as `configure' are involved. Use GNU `make'
instead.
On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot
parse its `<wchar.h>' header file. The option `-nodtk' can be used as
a workaround. If GNU CC is not installed, it is therefore recommended
to try
./configure CC="cc"
and if that doesn't work, try
./configure CC="cc -nodtk"
On Solaris, don't put `/usr/ucb' early in your `PATH'. This
directory contains several dysfunctional programs; working variants of
these programs are available in `/usr/bin'. So, if you need `/usr/ucb'
in your `PATH', put it _after_ `/usr/bin'.
On Haiku, software installed for all users goes in `/boot/common',
not `/usr/local'. It is recommended to use the following options:
./configure --prefix=/boot/common
Specifying the System Type
==========================
There may be some features `configure' cannot figure out
automatically, but needs to determine by the type of machine the package
will run on. Usually, assuming the package is built to be run on the
_same_ architectures, `configure' can figure that out, but if it prints
a message saying it cannot guess the machine type, give it the
`--build=TYPE' option. TYPE can either be a short name for the system
type, such as `sun4', or a canonical name which has the form:
CPU-COMPANY-SYSTEM
where SYSTEM can have one of these forms:
OS
KERNEL-OS
See the file `config.sub' for the possible values of each field. If
`config.sub' isn't included in this package, then this package doesn't
need to know the machine type.
If you are _building_ compiler tools for cross-compiling, you should
use the option `--target=TYPE' to select the type of system they will
produce code for.
If you want to _use_ a cross compiler, that generates code for a
platform different from the build platform, you should specify the
"host" platform (i.e., that on which the generated programs will
eventually be run) with `--host=TYPE'.
Sharing Defaults
================
If you want to set default values for `configure' scripts to share,
you can create a site shell script called `config.site' that gives
default values for variables like `CC', `cache_file', and `prefix'.
`configure' looks for `PREFIX/share/config.site' if it exists, then
`PREFIX/etc/config.site' if it exists. Or, you can set the
`CONFIG_SITE' environment variable to the location of the site script.
A warning: not all `configure' scripts look for a site script.
Defining Variables
==================
Variables not defined in a site shell script can be set in the
environment passed to `configure'. However, some packages may run
configure again during the build, and the customized values of these
variables may be lost. In order to avoid this problem, you should set
them in the `configure' command line, using `VAR=value'. For example:
./configure CC=/usr/local2/bin/gcc
causes the specified `gcc' to be used as the C compiler (unless it is
overridden in the site shell script).
Unfortunately, this technique does not work for `CONFIG_SHELL' due to
an Autoconf limitation. Until the limitation is lifted, you can use
this workaround:
CONFIG_SHELL=/bin/bash ./configure CONFIG_SHELL=/bin/bash
`configure' Invocation
======================
`configure' recognizes the following options to control how it
operates.
`--help'
`-h'
Print a summary of all of the options to `configure', and exit.
`--help=short'
`--help=recursive'
Print a summary of the options unique to this package's
`configure', and exit. The `short' variant lists options used
only in the top level, while the `recursive' variant lists options
also present in any nested packages.
`--version'
`-V'
Print the version of Autoconf used to generate the `configure'
script, and exit.
`--cache-file=FILE'
Enable the cache: use and save the results of the tests in FILE,
traditionally `config.cache'. FILE defaults to `/dev/null' to
disable caching.
`--config-cache'
`-C'
Alias for `--cache-file=config.cache'.
`--quiet'
`--silent'
`-q'
Do not print messages saying which checks are being made. To
suppress all normal output, redirect it to `/dev/null' (any error
messages will still be shown).
`--srcdir=DIR'
Look for the package's source code in directory DIR. Usually
`configure' can determine that directory automatically.
`--prefix=DIR'
Use DIR as the installation prefix. *note Installation Names::
for more details, including other options available for fine-tuning
the installation locations.
`--no-create'
`-n'
Run the configure checks, but stop before creating any output
files.
`configure' also accepts some other, not widely useful, options. Run
`configure --help' for more details.

4
Makefile.am Executable file
View File

@ -0,0 +1,4 @@
SUBDIRS = . bin man
EXTRA_dist = CreateSubHierarchy.pl

765
Makefile.in Normal file
View File

@ -0,0 +1,765 @@
# Makefile.in generated by automake 1.14.1 from Makefile.am.
# @configure_input@
# Copyright (C) 1994-2013 Free Software Foundation, Inc.
# This Makefile.in is free software; the Free Software Foundation
# gives unlimited permission to copy and/or distribute it,
# with or without modifications, as long as this notice is preserved.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.
@SET_MAKE@
VPATH = @srcdir@
am__is_gnu_make = test -n '$(MAKEFILE_LIST)' && test -n '$(MAKELEVEL)'
am__make_running_with_option = \
case $${target_option-} in \
?) ;; \
*) echo "am__make_running_with_option: internal error: invalid" \
"target option '$${target_option-}' specified" >&2; \
exit 1;; \
esac; \
has_opt=no; \
sane_makeflags=$$MAKEFLAGS; \
if $(am__is_gnu_make); then \
sane_makeflags=$$MFLAGS; \
else \
case $$MAKEFLAGS in \
*\\[\ \ ]*) \
bs=\\; \
sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \
| sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \
esac; \
fi; \
skip_next=no; \
strip_trailopt () \
{ \
flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \
}; \
for flg in $$sane_makeflags; do \
test $$skip_next = yes && { skip_next=no; continue; }; \
case $$flg in \
*=*|--*) continue;; \
-*I) strip_trailopt 'I'; skip_next=yes;; \
-*I?*) strip_trailopt 'I';; \
-*O) strip_trailopt 'O'; skip_next=yes;; \
-*O?*) strip_trailopt 'O';; \
-*l) strip_trailopt 'l'; skip_next=yes;; \
-*l?*) strip_trailopt 'l';; \
-[dEDm]) skip_next=yes;; \
-[JT]) skip_next=yes;; \
esac; \
case $$flg in \
*$$target_option*) has_opt=yes; break;; \
esac; \
done; \
test $$has_opt = yes
am__make_dryrun = (target_option=n; $(am__make_running_with_option))
am__make_keepgoing = (target_option=k; $(am__make_running_with_option))
pkgdatadir = $(datadir)/@PACKAGE@
pkgincludedir = $(includedir)/@PACKAGE@
pkglibdir = $(libdir)/@PACKAGE@
pkglibexecdir = $(libexecdir)/@PACKAGE@
am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd
install_sh_DATA = $(install_sh) -c -m 644
install_sh_PROGRAM = $(install_sh) -c
install_sh_SCRIPT = $(install_sh) -c
INSTALL_HEADER = $(INSTALL_DATA)
transform = $(program_transform_name)
NORMAL_INSTALL = :
PRE_INSTALL = :
POST_INSTALL = :
NORMAL_UNINSTALL = :
PRE_UNINSTALL = :
POST_UNINSTALL = :
subdir = .
DIST_COMMON = INSTALL NEWS README AUTHORS ChangeLog \
$(srcdir)/Makefile.in $(srcdir)/Makefile.am \
$(top_srcdir)/configure $(am__configure_deps) \
$(srcdir)/config.h.in COPYING compile depcomp install-sh \
missing ylwrap
ACLOCAL_M4 = $(top_srcdir)/aclocal.m4
am__aclocal_m4_deps = $(top_srcdir)/configure.ac
am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \
$(ACLOCAL_M4)
am__CONFIG_DISTCLEAN_FILES = config.status config.cache config.log \
configure.lineno config.status.lineno
mkinstalldirs = $(install_sh) -d
CONFIG_HEADER = config.h
CONFIG_CLEAN_FILES =
CONFIG_CLEAN_VPATH_FILES =
AM_V_P = $(am__v_P_@AM_V@)
am__v_P_ = $(am__v_P_@AM_DEFAULT_V@)
am__v_P_0 = false
am__v_P_1 = :
AM_V_GEN = $(am__v_GEN_@AM_V@)
am__v_GEN_ = $(am__v_GEN_@AM_DEFAULT_V@)
am__v_GEN_0 = @echo " GEN " $@;
am__v_GEN_1 =
AM_V_at = $(am__v_at_@AM_V@)
am__v_at_ = $(am__v_at_@AM_DEFAULT_V@)
am__v_at_0 = @
am__v_at_1 =
SOURCES =
DIST_SOURCES =
RECURSIVE_TARGETS = all-recursive check-recursive cscopelist-recursive \
ctags-recursive dvi-recursive html-recursive info-recursive \
install-data-recursive install-dvi-recursive \
install-exec-recursive install-html-recursive \
install-info-recursive install-pdf-recursive \
install-ps-recursive install-recursive installcheck-recursive \
installdirs-recursive pdf-recursive ps-recursive \
tags-recursive uninstall-recursive
am__can_run_installinfo = \
case $$AM_UPDATE_INFO_DIR in \
n|no|NO) false;; \
*) (install-info --version) >/dev/null 2>&1;; \
esac
RECURSIVE_CLEAN_TARGETS = mostlyclean-recursive clean-recursive \
distclean-recursive maintainer-clean-recursive
am__recursive_targets = \
$(RECURSIVE_TARGETS) \
$(RECURSIVE_CLEAN_TARGETS) \
$(am__extra_recursive_targets)
AM_RECURSIVE_TARGETS = $(am__recursive_targets:-recursive=) TAGS CTAGS \
cscope distdir dist dist-all distcheck
am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) \
$(LISP)config.h.in
# Read a list of newline-separated strings from the standard input,
# and print each of them once, without duplicates. Input order is
# *not* preserved.
am__uniquify_input = $(AWK) '\
BEGIN { nonempty = 0; } \
{ items[$$0] = 1; nonempty = 1; } \
END { if (nonempty) { for (i in items) print i; }; } \
'
# Make sure the list of sources is unique. This is necessary because,
# e.g., the same source file might be shared among _SOURCES variables
# for different programs/libraries.
am__define_uniq_tagged_files = \
list='$(am__tagged_files)'; \
unique=`for i in $$list; do \
if test -f "$$i"; then echo $$i; else echo $(srcdir)/$$i; fi; \
done | $(am__uniquify_input)`
ETAGS = etags
CTAGS = ctags
CSCOPE = cscope
DIST_SUBDIRS = $(SUBDIRS)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
distdir = $(PACKAGE)-$(VERSION)
top_distdir = $(distdir)
am__remove_distdir = \
if test -d "$(distdir)"; then \
find "$(distdir)" -type d ! -perm -200 -exec chmod u+w {} ';' \
&& rm -rf "$(distdir)" \
|| { sleep 5 && rm -rf "$(distdir)"; }; \
else :; fi
am__post_remove_distdir = $(am__remove_distdir)
am__relativize = \
dir0=`pwd`; \
sed_first='s,^\([^/]*\)/.*$$,\1,'; \
sed_rest='s,^[^/]*/*,,'; \
sed_last='s,^.*/\([^/]*\)$$,\1,'; \
sed_butlast='s,/*[^/]*$$,,'; \
while test -n "$$dir1"; do \
first=`echo "$$dir1" | sed -e "$$sed_first"`; \
if test "$$first" != "."; then \
if test "$$first" = ".."; then \
dir2=`echo "$$dir0" | sed -e "$$sed_last"`/"$$dir2"; \
dir0=`echo "$$dir0" | sed -e "$$sed_butlast"`; \
else \
first2=`echo "$$dir2" | sed -e "$$sed_first"`; \
if test "$$first2" = "$$first"; then \
dir2=`echo "$$dir2" | sed -e "$$sed_rest"`; \
else \
dir2="../$$dir2"; \
fi; \
dir0="$$dir0"/"$$first"; \
fi; \
fi; \
dir1=`echo "$$dir1" | sed -e "$$sed_rest"`; \
done; \
reldir="$$dir2"
DIST_ARCHIVES = $(distdir).tar.gz
GZIP_ENV = --best
DIST_TARGETS = dist-gzip
distuninstallcheck_listfiles = find . -type f -print
am__distuninstallcheck_listfiles = $(distuninstallcheck_listfiles) \
| sed 's|^\./|$(prefix)/|' | grep -v '$(infodir)/dir$$'
distcleancheck_listfiles = find . -type f -print
ACLOCAL = @ACLOCAL@
AMTAR = @AMTAR@
AM_DEFAULT_VERBOSITY = @AM_DEFAULT_VERBOSITY@
AUTOCONF = @AUTOCONF@
AUTOHEADER = @AUTOHEADER@
AUTOMAKE = @AUTOMAKE@
AWK = @AWK@
CC = @CC@
CCDEPMODE = @CCDEPMODE@
CFLAGS = @CFLAGS@
CPP = @CPP@
CPPFLAGS = @CPPFLAGS@
CYGPATH_W = @CYGPATH_W@
DEFS = @DEFS@
DEPDIR = @DEPDIR@
ECHO_C = @ECHO_C@
ECHO_N = @ECHO_N@
ECHO_T = @ECHO_T@
EGREP = @EGREP@
EXEEXT = @EXEEXT@
FTS_OBJ = @FTS_OBJ@
FT_INCLUDES = @FT_INCLUDES@
FT_LDFLAGS = @FT_LDFLAGS@
GREP = @GREP@
INSTALL = @INSTALL@
INSTALL_DATA = @INSTALL_DATA@
INSTALL_PROGRAM = @INSTALL_PROGRAM@
INSTALL_SCRIPT = @INSTALL_SCRIPT@
INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@
LDFLAGS = @LDFLAGS@
LEX = @LEX@
LEXLIB = @LEXLIB@
LEX_OUTPUT_ROOT = @LEX_OUTPUT_ROOT@
LFLAGS = @LFLAGS@
LIBOBJS = @LIBOBJS@
LIBS = @LIBS@
LTLIBOBJS = @LTLIBOBJS@
MAKEINFO = @MAKEINFO@
MKDIR_P = @MKDIR_P@
OBJEXT = @OBJEXT@
PACKAGE = @PACKAGE@
PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_STRING = @PACKAGE_STRING@
PACKAGE_TARNAME = @PACKAGE_TARNAME@
PACKAGE_URL = @PACKAGE_URL@
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
RRD_LIBS = @RRD_LIBS@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
VERSION = @VERSION@
YACC = @YACC@
YFLAGS = @YFLAGS@
abs_builddir = @abs_builddir@
abs_srcdir = @abs_srcdir@
abs_top_builddir = @abs_top_builddir@
abs_top_srcdir = @abs_top_srcdir@
ac_ct_CC = @ac_ct_CC@
am__include = @am__include@
am__leading_dot = @am__leading_dot@
am__quote = @am__quote@
am__tar = @am__tar@
am__untar = @am__untar@
bindir = @bindir@
build_alias = @build_alias@
builddir = @builddir@
datadir = @datadir@
datarootdir = @datarootdir@
docdir = @docdir@
dvidir = @dvidir@
exec_prefix = @exec_prefix@
host_alias = @host_alias@
htmldir = @htmldir@
includedir = @includedir@
infodir = @infodir@
install_sh = @install_sh@
libdir = @libdir@
libexecdir = @libexecdir@
localedir = @localedir@
localstatedir = @localstatedir@
mandir = @mandir@
mkdir_p = @mkdir_p@
oldincludedir = @oldincludedir@
pdfdir = @pdfdir@
prefix = @prefix@
program_transform_name = @program_transform_name@
psdir = @psdir@
sbindir = @sbindir@
sharedstatedir = @sharedstatedir@
srcdir = @srcdir@
sysconfdir = @sysconfdir@
target_alias = @target_alias@
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
SUBDIRS = . bin man
EXTRA_dist = CreateSubHierarchy.pl
all: config.h
$(MAKE) $(AM_MAKEFLAGS) all-recursive
.SUFFIXES:
am--refresh: Makefile
@:
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
*$$dep*) \
echo ' cd $(srcdir) && $(AUTOMAKE) --gnu'; \
$(am__cd) $(srcdir) && $(AUTOMAKE) --gnu \
&& exit 0; \
exit 1;; \
esac; \
done; \
echo ' cd $(top_srcdir) && $(AUTOMAKE) --gnu Makefile'; \
$(am__cd) $(top_srcdir) && \
$(AUTOMAKE) --gnu Makefile
.PRECIOUS: Makefile
Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status
@case '$?' in \
*config.status*) \
echo ' $(SHELL) ./config.status'; \
$(SHELL) ./config.status;; \
*) \
echo ' cd $(top_builddir) && $(SHELL) ./config.status $@ $(am__depfiles_maybe)'; \
cd $(top_builddir) && $(SHELL) ./config.status $@ $(am__depfiles_maybe);; \
esac;
$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES)
$(SHELL) ./config.status --recheck
$(top_srcdir)/configure: $(am__configure_deps)
$(am__cd) $(srcdir) && $(AUTOCONF)
$(ACLOCAL_M4): $(am__aclocal_m4_deps)
$(am__cd) $(srcdir) && $(ACLOCAL) $(ACLOCAL_AMFLAGS)
$(am__aclocal_m4_deps):
config.h: stamp-h1
@test -f $@ || rm -f stamp-h1
@test -f $@ || $(MAKE) $(AM_MAKEFLAGS) stamp-h1
stamp-h1: $(srcdir)/config.h.in $(top_builddir)/config.status
@rm -f stamp-h1
cd $(top_builddir) && $(SHELL) ./config.status config.h
$(srcdir)/config.h.in: $(am__configure_deps)
($(am__cd) $(top_srcdir) && $(AUTOHEADER))
rm -f stamp-h1
touch $@
distclean-hdr:
-rm -f config.h stamp-h1
# This directory's subdirectories are mostly independent; you can cd
# into them and run 'make' without going through this Makefile.
# To change the values of 'make' variables: instead of editing Makefiles,
# (1) if the variable is set in 'config.status', edit 'config.status'
# (which will cause the Makefiles to be regenerated when you run 'make');
# (2) otherwise, pass the desired values on the 'make' command line.
$(am__recursive_targets):
@fail=; \
if $(am__make_keepgoing); then \
failcom='fail=yes'; \
else \
failcom='exit 1'; \
fi; \
dot_seen=no; \
target=`echo $@ | sed s/-recursive//`; \
case "$@" in \
distclean-* | maintainer-clean-*) list='$(DIST_SUBDIRS)' ;; \
*) list='$(SUBDIRS)' ;; \
esac; \
for subdir in $$list; do \
echo "Making $$target in $$subdir"; \
if test "$$subdir" = "."; then \
dot_seen=yes; \
local_target="$$target-am"; \
else \
local_target="$$target"; \
fi; \
($(am__cd) $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$local_target) \
|| eval $$failcom; \
done; \
if test "$$dot_seen" = "no"; then \
$(MAKE) $(AM_MAKEFLAGS) "$$target-am" || exit 1; \
fi; test -z "$$fail"
ID: $(am__tagged_files)
$(am__define_uniq_tagged_files); mkid -fID $$unique
tags: tags-recursive
TAGS: tags
tags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files)
set x; \
here=`pwd`; \
if ($(ETAGS) --etags-include --version) >/dev/null 2>&1; then \
include_option=--etags-include; \
empty_fix=.; \
else \
include_option=--include; \
empty_fix=; \
fi; \
list='$(SUBDIRS)'; for subdir in $$list; do \
if test "$$subdir" = .; then :; else \
test ! -f $$subdir/TAGS || \
set "$$@" "$$include_option=$$here/$$subdir/TAGS"; \
fi; \
done; \
$(am__define_uniq_tagged_files); \
shift; \
if test -z "$(ETAGS_ARGS)$$*$$unique"; then :; else \
test -n "$$unique" || unique=$$empty_fix; \
if test $$# -gt 0; then \
$(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \
"$$@" $$unique; \
else \
$(ETAGS) $(ETAGSFLAGS) $(AM_ETAGSFLAGS) $(ETAGS_ARGS) \
$$unique; \
fi; \
fi
ctags: ctags-recursive
CTAGS: ctags
ctags-am: $(TAGS_DEPENDENCIES) $(am__tagged_files)
$(am__define_uniq_tagged_files); \
test -z "$(CTAGS_ARGS)$$unique" \
|| $(CTAGS) $(CTAGSFLAGS) $(AM_CTAGSFLAGS) $(CTAGS_ARGS) \
$$unique
GTAGS:
here=`$(am__cd) $(top_builddir) && pwd` \
&& $(am__cd) $(top_srcdir) \
&& gtags -i $(GTAGS_ARGS) "$$here"
cscope: cscope.files
test ! -s cscope.files \
|| $(CSCOPE) -b -q $(AM_CSCOPEFLAGS) $(CSCOPEFLAGS) -i cscope.files $(CSCOPE_ARGS)
clean-cscope:
-rm -f cscope.files
cscope.files: clean-cscope cscopelist
cscopelist: cscopelist-recursive
cscopelist-am: $(am__tagged_files)
list='$(am__tagged_files)'; \
case "$(srcdir)" in \
[\\/]* | ?:[\\/]*) sdir="$(srcdir)" ;; \
*) sdir=$(subdir)/$(srcdir) ;; \
esac; \
for i in $$list; do \
if test -f "$$i"; then \
echo "$(subdir)/$$i"; \
else \
echo "$$sdir/$$i"; \
fi; \
done >> $(top_builddir)/cscope.files
distclean-tags:
-rm -f TAGS ID GTAGS GRTAGS GSYMS GPATH tags
-rm -f cscope.out cscope.in.out cscope.po.out cscope.files
distdir: $(DISTFILES)
$(am__remove_distdir)
test -d "$(distdir)" || mkdir "$(distdir)"
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
list='$(DISTFILES)'; \
dist_files=`for file in $$list; do echo $$file; done | \
sed -e "s|^$$srcdirstrip/||;t" \
-e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t"`; \
case $$dist_files in \
*/*) $(MKDIR_P) `echo "$$dist_files" | \
sed '/\//!d;s|^|$(distdir)/|;s,/[^/]*$$,,' | \
sort -u` ;; \
esac; \
for file in $$dist_files; do \
if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \
if test -d $$d/$$file; then \
dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \
if test -d "$(distdir)/$$file"; then \
find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \
fi; \
if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \
cp -fpR $(srcdir)/$$file "$(distdir)$$dir" || exit 1; \
find "$(distdir)/$$file" -type d ! -perm -700 -exec chmod u+rwx {} \;; \
fi; \
cp -fpR $$d/$$file "$(distdir)$$dir" || exit 1; \
else \
test -f "$(distdir)/$$file" \
|| cp -p $$d/$$file "$(distdir)/$$file" \
|| exit 1; \
fi; \
done
@list='$(DIST_SUBDIRS)'; for subdir in $$list; do \
if test "$$subdir" = .; then :; else \
$(am__make_dryrun) \
|| test -d "$(distdir)/$$subdir" \
|| $(MKDIR_P) "$(distdir)/$$subdir" \
|| exit 1; \
dir1=$$subdir; dir2="$(distdir)/$$subdir"; \
$(am__relativize); \
new_distdir=$$reldir; \
dir1=$$subdir; dir2="$(top_distdir)"; \
$(am__relativize); \
new_top_distdir=$$reldir; \
echo " (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) top_distdir="$$new_top_distdir" distdir="$$new_distdir" \\"; \
echo " am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)"; \
($(am__cd) $$subdir && \
$(MAKE) $(AM_MAKEFLAGS) \
top_distdir="$$new_top_distdir" \
distdir="$$new_distdir" \
am__remove_distdir=: \
am__skip_length_check=: \
am__skip_mode_fix=: \
distdir) \
|| exit 1; \
fi; \
done
-test -n "$(am__skip_mode_fix)" \
|| find "$(distdir)" -type d ! -perm -755 \
-exec chmod u+rwx,go+rx {} \; -o \
! -type d ! -perm -444 -links 1 -exec chmod a+r {} \; -o \
! -type d ! -perm -400 -exec chmod a+r {} \; -o \
! -type d ! -perm -444 -exec $(install_sh) -c -m a+r {} {} \; \
|| chmod -R a+r "$(distdir)"
dist-gzip: distdir
tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
$(am__post_remove_distdir)
dist-bzip2: distdir
tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c >$(distdir).tar.bz2
$(am__post_remove_distdir)
dist-lzip: distdir
tardir=$(distdir) && $(am__tar) | lzip -c $${LZIP_OPT--9} >$(distdir).tar.lz
$(am__post_remove_distdir)
dist-xz: distdir
tardir=$(distdir) && $(am__tar) | XZ_OPT=$${XZ_OPT--e} xz -c >$(distdir).tar.xz
$(am__post_remove_distdir)
dist-tarZ: distdir
@echo WARNING: "Support for shar distribution archives is" \
"deprecated." >&2
@echo WARNING: "It will be removed altogether in Automake 2.0" >&2
tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
$(am__post_remove_distdir)
dist-shar: distdir
@echo WARNING: "Support for distribution archives compressed with" \
"legacy program 'compress' is deprecated." >&2
@echo WARNING: "It will be removed altogether in Automake 2.0" >&2
shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz
$(am__post_remove_distdir)
dist-zip: distdir
-rm -f $(distdir).zip
zip -rq $(distdir).zip $(distdir)
$(am__post_remove_distdir)
dist dist-all:
$(MAKE) $(AM_MAKEFLAGS) $(DIST_TARGETS) am__post_remove_distdir='@:'
$(am__post_remove_distdir)
# This target untars the dist file and tries a VPATH configuration. Then
# it guarantees that the distribution is self-contained by making another
# tarfile.
distcheck: dist
case '$(DIST_ARCHIVES)' in \
*.tar.gz*) \
GZIP=$(GZIP_ENV) gzip -dc $(distdir).tar.gz | $(am__untar) ;;\
*.tar.bz2*) \
bzip2 -dc $(distdir).tar.bz2 | $(am__untar) ;;\
*.tar.lz*) \
lzip -dc $(distdir).tar.lz | $(am__untar) ;;\
*.tar.xz*) \
xz -dc $(distdir).tar.xz | $(am__untar) ;;\
*.tar.Z*) \
uncompress -c $(distdir).tar.Z | $(am__untar) ;;\
*.shar.gz*) \
GZIP=$(GZIP_ENV) gzip -dc $(distdir).shar.gz | unshar ;;\
*.zip*) \
unzip $(distdir).zip ;;\
esac
chmod -R a-w $(distdir)
chmod u+w $(distdir)
mkdir $(distdir)/_build $(distdir)/_inst
chmod a-w $(distdir)
test -d $(distdir)/_build || exit 0; \
dc_install_base=`$(am__cd) $(distdir)/_inst && pwd | sed -e 's,^[^:\\/]:[\\/],/,'` \
&& dc_destdir="$${TMPDIR-/tmp}/am-dc-$$$$/" \
&& am__cwd=`pwd` \
&& $(am__cd) $(distdir)/_build \
&& ../configure \
$(AM_DISTCHECK_CONFIGURE_FLAGS) \
$(DISTCHECK_CONFIGURE_FLAGS) \
--srcdir=.. --prefix="$$dc_install_base" \
&& $(MAKE) $(AM_MAKEFLAGS) \
&& $(MAKE) $(AM_MAKEFLAGS) dvi \
&& $(MAKE) $(AM_MAKEFLAGS) check \
&& $(MAKE) $(AM_MAKEFLAGS) install \
&& $(MAKE) $(AM_MAKEFLAGS) installcheck \
&& $(MAKE) $(AM_MAKEFLAGS) uninstall \
&& $(MAKE) $(AM_MAKEFLAGS) distuninstallcheck_dir="$$dc_install_base" \
distuninstallcheck \
&& chmod -R a-w "$$dc_install_base" \
&& ({ \
(cd ../.. && umask 077 && mkdir "$$dc_destdir") \
&& $(MAKE) $(AM_MAKEFLAGS) DESTDIR="$$dc_destdir" install \
&& $(MAKE) $(AM_MAKEFLAGS) DESTDIR="$$dc_destdir" uninstall \
&& $(MAKE) $(AM_MAKEFLAGS) DESTDIR="$$dc_destdir" \
distuninstallcheck_dir="$$dc_destdir" distuninstallcheck; \
} || { rm -rf "$$dc_destdir"; exit 1; }) \
&& rm -rf "$$dc_destdir" \
&& $(MAKE) $(AM_MAKEFLAGS) dist \
&& rm -rf $(DIST_ARCHIVES) \
&& $(MAKE) $(AM_MAKEFLAGS) distcleancheck \
&& cd "$$am__cwd" \
|| exit 1
$(am__post_remove_distdir)
@(echo "$(distdir) archives ready for distribution: "; \
list='$(DIST_ARCHIVES)'; for i in $$list; do echo $$i; done) | \
sed -e 1h -e 1s/./=/g -e 1p -e 1x -e '$$p' -e '$$x'
distuninstallcheck:
@test -n '$(distuninstallcheck_dir)' || { \
echo 'ERROR: trying to run $@ with an empty' \
'$$(distuninstallcheck_dir)' >&2; \
exit 1; \
}; \
$(am__cd) '$(distuninstallcheck_dir)' || { \
echo 'ERROR: cannot chdir into $(distuninstallcheck_dir)' >&2; \
exit 1; \
}; \
test `$(am__distuninstallcheck_listfiles) | wc -l` -eq 0 \
|| { echo "ERROR: files left after uninstall:" ; \
if test -n "$(DESTDIR)"; then \
echo " (check DESTDIR support)"; \
fi ; \
$(distuninstallcheck_listfiles) ; \
exit 1; } >&2
distcleancheck: distclean
@if test '$(srcdir)' = . ; then \
echo "ERROR: distcleancheck can only run from a VPATH build" ; \
exit 1 ; \
fi
@test `$(distcleancheck_listfiles) | wc -l` -eq 0 \
|| { echo "ERROR: files left in build directory after distclean:" ; \
$(distcleancheck_listfiles) ; \
exit 1; } >&2
check-am: all-am
check: check-recursive
all-am: Makefile config.h
installdirs: installdirs-recursive
installdirs-am:
install: install-recursive
install-exec: install-exec-recursive
install-data: install-data-recursive
uninstall: uninstall-recursive
install-am: all-am
@$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am
installcheck: installcheck-recursive
install-strip:
if test -z '$(STRIP)'; then \
$(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \
install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \
install; \
else \
$(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \
install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \
"INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'" install; \
fi
mostlyclean-generic:
clean-generic:
distclean-generic:
-test -z "$(CONFIG_CLEAN_FILES)" || rm -f $(CONFIG_CLEAN_FILES)
-test . = "$(srcdir)" || test -z "$(CONFIG_CLEAN_VPATH_FILES)" || rm -f $(CONFIG_CLEAN_VPATH_FILES)
maintainer-clean-generic:
@echo "This command is intended for maintainers to use"
@echo "it deletes files that may require special tools to rebuild."
clean: clean-recursive
clean-am: clean-generic mostlyclean-am
distclean: distclean-recursive
-rm -f $(am__CONFIG_DISTCLEAN_FILES)
-rm -f Makefile
distclean-am: clean-am distclean-generic distclean-hdr distclean-tags
dvi: dvi-recursive
dvi-am:
html: html-recursive
html-am:
info: info-recursive
info-am:
install-data-am:
install-dvi: install-dvi-recursive
install-dvi-am:
install-exec-am:
install-html: install-html-recursive
install-html-am:
install-info: install-info-recursive
install-info-am:
install-man:
install-pdf: install-pdf-recursive
install-pdf-am:
install-ps: install-ps-recursive
install-ps-am:
installcheck-am:
maintainer-clean: maintainer-clean-recursive
-rm -f $(am__CONFIG_DISTCLEAN_FILES)
-rm -rf $(top_srcdir)/autom4te.cache
-rm -f Makefile
maintainer-clean-am: distclean-am maintainer-clean-generic
mostlyclean: mostlyclean-recursive
mostlyclean-am: mostlyclean-generic
pdf: pdf-recursive
pdf-am:
ps: ps-recursive
ps-am:
uninstall-am:
.MAKE: $(am__recursive_targets) all install-am install-strip
.PHONY: $(am__recursive_targets) CTAGS GTAGS TAGS all all-am \
am--refresh check check-am clean clean-cscope clean-generic \
cscope cscopelist-am ctags ctags-am dist dist-all dist-bzip2 \
dist-gzip dist-lzip dist-shar dist-tarZ dist-xz dist-zip \
distcheck distclean distclean-generic distclean-hdr \
distclean-tags distcleancheck distdir distuninstallcheck dvi \
dvi-am html html-am info info-am install install-am \
install-data install-data-am install-dvi install-dvi-am \
install-exec install-exec-am install-html install-html-am \
install-info install-info-am install-man install-pdf \
install-pdf-am install-ps install-ps-am install-strip \
installcheck installcheck-am installdirs installdirs-am \
maintainer-clean maintainer-clean-generic mostlyclean \
mostlyclean-generic pdf pdf-am ps ps-am tags tags-am uninstall \
uninstall-am
# Tell versions [3.59,3.63) of GNU make to not export all variables.
# Otherwise a system limit (for SysV at least) may be exceeded.
.NOEXPORT:

1
NEWS Symbolic link
View File

@ -0,0 +1 @@
ChangeLog

412
README.md Normal file → Executable file
View File

@ -1,4 +1,412 @@
# nfdump
Netflow processing tools
nfdump is moving here - stay tuned.
Stable Release v1.6.13
See the Changelog file for all changes in release 1.6.10
Notes on NSEL/ASA support
-------------------------
nfdump-1.6.9 includes a new written from scratch implemented NSEL/ASA
module. It's based on the CISCO ASA Spec 8.4:
"Implementation Note for NetFlow Collectors, Version 8.4"
Due to this new implementation, nfdump-1.6.9 is not compatible with old
nfdump-1.5.8-2-NSEL.
To build nfdump, add --enable-nsel to the configure command. By enabling
the ASA/NSEL option, nfdump processes normal flows as well ASA/NSEL records
likewise. nfcapd adds by default all required NSEL extesions equivalent
to '-Tnsel'
Note on NEL support
-------------------
nfdump-1.6.9 includes a new module for decoding the CISCO NEL ( NAT event
logging ) records. It's considered to be experimantal, as no official
documentation can be found. Let me know otherwise.
To build nfdump, add --enable-nel to the configure command. By enabling
the NEL option, nfdump processes normal flows as well NEL records
likewise. nfcapd adds by default all required NEL extesions equivalent
to '-Tnel'
Although it's possibel to enable NSEL und NEL likewise, users could get
confused by nfdump output, as NSEL output format overwrites NEL format.
In that case you need explicitly to define -o nel.
Notes on IPFIX
---------------
nfdump contains an IPFIX module for decoding IPFIX data. It
is considered not yet to be complete and does not yet support full IPFIX.
o Supports basically same feature set of elements as netflow_v9 module
o Only UDP traffic is accepted no SCTP so far
o No sampling support.
o Still more test data needed. If you would like to see more IPFIX
support, please contact me.
General README
--------------
This is a small description, what the nfdump tools do and how they work.
Nfdump is distributed under the BSD license - see BSD-license.txt
The nfdump tools collect and process netflow data on the command line.
They are part of the NFSEN project, which is explained more detailed at
http://www.terena.nl/tech/task-forces/tf-csirt/meeting12/nfsen-Haag.pdf
The Web interface mentioned is not part of nfdump and is available at
http://nfsen.sourceforge.net
nfdump tools overview:
----------------------
nfcapd - netflow collector daemon.
Reads the netflow data from the network and stores the flow records
into files. Automatically rotates files every n minutes. ( typically
every 5 min ) nfcapd reads netflow versions v1, v5, v7 and v9 flows
as well as IPFIX flows transparently. Several netflow streams can be
sent to a single or collector.
nfdump - netflow dump.
Reads the netflow data from the files stored by nfcapd. It's filter
syntax is similar to tcpdump ( pcap like ) but for netflow adapted.
If you like tcpdump you will like nfdump. nfdump displays netflow
data and/or creates top N statistics of flows, bytes, packets. nfdump
has a powerful and flexible flow aggregation including bi-directional
flows. The output format is user selectable and also includes a simple
csv format for post processing.
nfreplay - netflow replay
Reads the netflow data from the files stored by nfcapd and sends it
over the network to another host.
nfexpire - expire old netflow data
Manages data expiration. Sets appropriate limits.
Optional binaries:
nfprofile - netflow profiler. Required by NfSen
Reads the netflow data from the files stored by nfcapd. Filters the
netflow data according to the specified filter sets ( profiles ) and
stores the filtered data into files for later use.
nftrack - Port tracking decoder for NfSen plugin PortTracker.
ft2nfdump - read flow-tools format - Optional tool
ft2nfdump acts as a pipe converter for flow-tools data. It allows
to read any flow-tools data and process and save it in nfdump format.
sfcapd - sflow collector daemon
scfapd collects sflow data and stores it into nfcapd comaptible files.
"sfcapd includes sFlow(TM), freely available from http://www.inmon.com/".
nfreader - Framework for programmers
nfreader is a framework to read nfdump files for any other purpose.
Own C code can be added to process flows. nfreader is not installed
parse_csv.pl - Simple reader, written in Perl.
parse_csv.pl reads nfdump csv output and print the flows to stdout.
This program is intended to be a framework for post processing flows
for any other purpose.
Note for sflow users:
sfcapd and nfcapd can be used concurrently to collect netflow and sflow
data at the same time. Generic command line options apply to both
collectors likewise. Due to lack of availability of sflow devices,
I could not test the correct output of IPv6 records. Users are requested
to send feedback to the list or directly to me. sfcapd's sflow decoding
module is based on InMon's sflowtool code and supports similar fields as
nfcapd does for netflow v9, which is a subset of all available sflow
fields in an sflow record. More fields may be integrated in future
versions of sfcapd.
Compression
-----------
Binary data files can optionally be compressed using the fast LZO1X-1
compression. For more details on this algorithm see,
http://www.oberhumer.com/opensource/lzo. LZO1X-1 is very fast, so
that compression can be used in real time by the collector. LZO1X-1
reduces the file size around 50%. You can check the compression speed
for your system by doing ./nftest <path/to/an/existing/netflow/file>.
Principle of Operation:
-----------------------
The goal of the design is to able to analyze netflow data from
the past as well as to track interesting traffic patterns
continuously. The amount of time back in the past is limited only
by the disk storage available for all the netflow data. The tools
are optimized for speed for efficient filtering. The filter rules
should look familiar to the syntax of tcpdump ( pcap compatible ).
All data is stored to disk, before it gets analyzed. This separates
the process of storing and analyzing the data.
The data is organized in a time-based fashion. Every n minutes
- typically 5 min - nfcapd rotates and renames the output file
with the timestamp nfcapd.YYYYMMddhhmm of the interval e.g.
nfcapd.200907110845 contains data from July 11th 2009 08:45 onward.
Based on a 5min time interval, this results in 288 files per day.
Analyzing the data can be done for a single file, or by concatenating
several files for a single output. The output is either ASCII text
or binary data, when saved into a file, ready to be processed again
with the same tools.
You may have several netflow sources - let's say 'router1' 'router2'
and so on. The data is organized as follows:
/flow_base_dir/router1
/flow_base_dir/router2
which means router1 and router2 are subdirs of the flow_base_dir.
Although several flow sources can be sent to a single collector,
It's recommended to have multiple collector on busy networks for
each source.
Example: Start two collectors on different ports:
nfcapd -w -D -S 2 -B 1024000 -l /flow_base_dir/router1 -p 23456
nfcapd -w -D -S 2 -B 1024000 -l /flow_base_dir/router2 -p 23457
nfcapd can handle multiple flow sources.
All sources can go into a single file or can be split:
All into the same file:
nfcapd -w -D -S 2 -l /flow_base_dir/routers -p 23456
Collected on one port and split per source:
nfcapd -w -D -S 2 -n router1,172.16.17.18,/flow_base_dir/router1 \
-n router2,172.16.17.20,/flow_base_dir/router2 -p 23456
See nfcapd(1) for a detailed explanation of all options.
Security: none of the tools requires root privileges, unless you have
a port < 1024. However, there is no access control mechanism in nfcapd.
It is assumed, that host level security is in place to filter the
proper IP addresses.
See the manual pages or use the -h switch for details on using each of
the programs. For any questions send email to phaag@users.sourceforge.net
Configure your router to export netflow. See the relevant documentation
for your model.
A generic Cisco sample configuration enabling NetFlow on an interface:
ip address 192.168.92.162 255.255.255.224
interface fastethernet 0/0
ip route-cache flow
To tell the router where to send the NetFlow data, enter the following
global configuration command:
ip flow-export 192.168.92.218 9995
ip flow-export version 5
ip flow-cache timeout active 5
This breaks up long-lived flows into 5-minute segments. You can choose
any number of minutes between 1 and 60;
Netflow v9 full export example of a cisco 7200 with sampling enabled:
interface Ethernet1/0
ip address 192.168.92.162 255.255.255.224
duplex half
flow-sampler my-map
!
!
flow-sampler-map my-map
mode random one-out-of 5
!
ip flow-cache timeout inactive 60
ip flow-cache timeout active 1
ip flow-capture fragment-offset
ip flow-capture packet-length
ip flow-capture ttl
ip flow-capture vlan-id
ip flow-capture icmp
ip flow-capture ip-id
ip flow-capture mac-addresses
ip flow-export version 9
ip flow-export template options export-stats
ip flow-export template options sampler
ip flow-export template options timeout-rate 1
ip flow-export template timeout-rate 1
ip flow-export destination 192.168.92.218 9995
See the relevant documentation for a full description of netflow commands
Note: Netflow version v5 and v7 have 32 bit counter values. The number of
packets or bytes may overflow this value, within the flow-cache timeout
on very busy routers. To prevent overflow, you may consider to reduce the
flow-cache timeout to lower values. All nfdump tools use 64 bit counters
internally, which means, all aggregated values are correctly reported.
The binary format of the data files is netflow version independent.
For speed reasons the binary format is machine architecture dependent, and
as such can not be exchanged between little and big endian systems.
Internally nfdump does all processing IP protocol independent, which means
everything works for IPv4 as well as IPv6 addresses.
See the nfdump(1) man page for details.
netflow version 9:
nfcapd supports a large range of netflow v9 tags. Version 1.6 nfdump
supports the following fields. This list can be found in netflow_v9.h
// Flowset record types
#define NF9_IN_BYTES 1
#define NF9_IN_PACKETS 2
#define NF9_FLOWS_AGGR 3
#define NF9_IN_PROTOCOL 4
#define NF9_SRC_TOS 5
#define NF9_TCP_FLAGS 6
#define NF9_L4_SRC_PORT 7
#define NF9_IPV4_SRC_ADDR 8
#define NF9_SRC_MASK 9
#define NF9_INPUT_SNMP 10
#define NF9_L4_DST_PORT 11
#define NF9_IPV4_DST_ADDR 12
#define NF9_DST_MASK 13
#define NF9_OUTPUT_SNMP 14
#define NF9_V4_NEXT_HOP 15
#define NF9_SRC_AS 16
#define NF9_DST_AS 17
#define NF9_BGP_V4_NEXT_HOP 18
#define NF9_LAST_SWITCHED 21
#define NF9_FIRST_SWITCHED 22
#define NF9_OUT_BYTES 23
#define NF9_OUT_PKTS 24
#define NF9_IPV6_SRC_ADDR 27
#define NF9_IPV6_DST_ADDR 28
#define NF9_IPV6_SRC_MASK 29
#define NF9_IPV6_DST_MASK 30
#define NF9_IPV6_FLOW_LABEL 31
#define NF9_ICMP_TYPE 32
#define NF9_SAMPLING_INTERVAL 34
#define NF9_SAMPLING_ALGORITHM 35
#define NF9_ENGINE_TYPE 38
#define NF9_ENGINE_ID 39
#define NF9_FLOW_SAMPLER_ID 48
#define FLOW_SAMPLER_MODE 49
#define NF9_FLOW_SAMPLER_RANDOM_INTERVAL 50
// #define NF9_MIN_TTL 52
// #define NF9_MAX_TTL 53
// #define NF9_IPV4_IDENT 54
#define NF9_DST_TOS 55
#define NF9_IN_SRC_MAC 56
#define NF9_OUT_DST_MAC 57
#define NF9_SRC_VLAN 58
#define NF9_DST_VLAN 59
#define NF9_DIRECTION 61
#define NF9_V6_NEXT_HOP 62
#define NF9_BPG_V6_NEXT_HOP 63
// #define NF9_V6_OPTION_HEADERS 64
#define NF9_MPLS_LABEL_1 70
#define NF9_MPLS_LABEL_2 71
#define NF9_MPLS_LABEL_3 72
#define NF9_MPLS_LABEL_4 73
#define NF9_MPLS_LABEL_5 74
#define NF9_MPLS_LABEL_6 75
#define NF9_MPLS_LABEL_7 76
#define NF9_MPLS_LABEL_8 77
#define NF9_MPLS_LABEL_9 78
#define NF9_MPLS_LABEL_10 79
#define NF9_IN_DST_MAC 80
#define NF9_OUT_SRC_MAC 81
#define NF9_FORWARDING_STATUS 89
#define NF9_BGP_ADJ_NEXT_AS 128
#define NF9_BGP_ADJ_PREV_AS 129
// CISCO ASA NSEL extension - Network Security Event Logging
#define NF_F_FLOW_BYTES 85
#define NF_F_CONN_ID 148
#define NF_F_FLOW_CREATE_TIME_MSEC 152
#define NF_F_ICMP_TYPE 176
#define NF_F_ICMP_CODE 177
#define NF_F_ICMP_TYPE_IPV6 178
#define NF_F_ICMP_CODE_IPV6 179
#define NF_F_FWD_FLOW_DELTA_BYTES 231
#define NF_F_REV_FLOW_DELTA_BYTES 232
#define NF_F_FW_EVENT84 233
#define NF_F_EVENT_TIME_MSEC 323
#define NF_F_INGRESS_ACL_ID 33000
#define NF_F_EGRESS_ACL_ID 33001
#define NF_F_FW_EXT_EVENT 33002
#define NF_F_USERNAME 40000
#define NF_F_XLATE_SRC_ADDR_IPV4 40001
#define NF_F_XLATE_DST_ADDR_IPV4 40002
#define NF_F_XLATE_SRC_PORT 40003
#define NF_F_XLATE_DST_PORT 40004
#define NF_F_FW_EVENT 40005
// Cisco ASR 1000 series NEL extension - Nat Event Logging
#define NF_N_NAT_EVENT 230
#define NF_N_INGRESS_VRFID 234
#define NF_N_NAT_INSIDE_GLOBAL_IPV4 225
#define NF_N_NAT_OUTSIDE_GLOBAL_IPV4 226
#define NF_N_POST_NAPT_SRC_PORT 227
#define NF_N_POST_NAPT_DST_PORT 228
// nprobe latency extensions
#define NF9_NPROBE_CLIENT_NW_DELAY_SEC 57554
#define NF9_NPROBE_CLIENT_NW_DELAY_USEC 57555
#define NF9_NPROBE_SERVER_NW_DELAY_SEC 57556
#define NF9_NPROBE_SERVER_NW_DELAY_USEC 57557
#define NF9_NPROBE_APPL_LATENCY_SEC 57558
#define NF9_NPROBE_APPL_LATENCY_USEC 57559
32 and 64 bit counters are supported for any counters. However, internally
nfdump stores packets and bytes counters always as 64bit counters.
16 and 32 bit AS numbers are supported.
Extensions: nfcapd supports a large number of v9 tags. In order to optimise
disk space and performance, v9 tags are grouped into a number of extensions
which may or may not be stored into the data file. Therefore the v9 templates
configured on the exporter may be tuned with the collector. Only the tags
common to both are stored into the data files. Extensions can be switch
on/off by using the -T option.
Sampling: By default, the sampling rate is set to 1 (unsampled) or to
any given value specified by the -s cmd line option. If sampling information
is found in the netflow stream, it overwrites the default value. Sampling
is automatically recognised when announced in v9 option templates
(tags #48, #49, #50 ) or in the unofficial v5 header hack. Note: Not all
platforms (or IOS versions) support exporting sampling information in
netflow data, even if sampling is configured. The number of bytes/packets
in each netflow record is automatically multiplied by the sampling rate.
The total number of flows is not changed as this is not accurate enough.
(Small flows versus large flows)
nfcapd can listen on IPv6 or IPv4. Furthermore multicast is supported.
Flow-tools compatibility
------------------------
When building with configure option --enable-ftconv, the flow-tools converter
is compiled. Using this converter, any flow-tools created data can be read
and processed and stored by nfdump.
Example:
flow-cat [options] | ft2nfdump | nfdump [options]
See the INSTALL file for installation details.

6
ToDo Normal file
View File

@ -0,0 +1,6 @@
- Add ASA support.
Port CISCO code from nfdump-1.5.7-nsel to v1.6
- Add packeteer code from wm.edu
- Add BGP merge for AS filtering
- Add index for fast IP lookup.
- ... and endless new user ideas ...

1149
aclocal.m4 vendored Normal file
View File

File diff suppressed because it is too large Load Diff

9524
autom4te.cache/output.0 Normal file
View File

File diff suppressed because it is too large Load Diff

9524
autom4te.cache/output.1 Normal file
View File

File diff suppressed because it is too large Load Diff

154
autom4te.cache/requests Normal file
View File

@ -0,0 +1,154 @@
# This file was generated.
# It contains the lists of macros which have been traced.
# It can be safely removed.
@request = (
bless( [
'0',
1,
[
'/opt/local/share/autoconf'
],
[
'/opt/local/share/autoconf/autoconf/autoconf.m4f',
'-',
'/opt/local/share/aclocal-1.14/internal/ac-config-macro-dirs.m4',
'/opt/local/share/aclocal-1.14/amversion.m4',
'/opt/local/share/aclocal-1.14/auxdir.m4',
'/opt/local/share/aclocal-1.14/cond.m4',
'/opt/local/share/aclocal-1.14/depend.m4',
'/opt/local/share/aclocal-1.14/depout.m4',
'/opt/local/share/aclocal-1.14/init.m4',
'/opt/local/share/aclocal-1.14/install-sh.m4',
'/opt/local/share/aclocal-1.14/lead-dot.m4',
'/opt/local/share/aclocal-1.14/make.m4',
'/opt/local/share/aclocal-1.14/missing.m4',
'/opt/local/share/aclocal-1.14/options.m4',
'/opt/local/share/aclocal-1.14/prog-cc-c-o.m4',
'/opt/local/share/aclocal-1.14/runlog.m4',
'/opt/local/share/aclocal-1.14/sanity.m4',
'/opt/local/share/aclocal-1.14/silent.m4',
'/opt/local/share/aclocal-1.14/strip.m4',
'/opt/local/share/aclocal-1.14/substnot.m4',
'/opt/local/share/aclocal-1.14/tar.m4',
'configure.ac'
],
{
'm4_pattern_forbid' => 1,
'AM_SET_CURRENT_AUTOMAKE_VERSION' => 1,
'_AM_SET_OPTION' => 1,
'AC_DEFUN' => 1,
'AM_INIT_AUTOMAKE' => 1,
'AM_AUTOMAKE_VERSION' => 1,
'AM_MISSING_HAS_RUN' => 1,
'AM_SUBST_NOTMAKE' => 1,
'AM_MISSING_PROG' => 1,
'AM_OUTPUT_DEPENDENCY_COMMANDS' => 1,
'AC_DEFUN_ONCE' => 1,
'_AM_CONFIG_MACRO_DIRS' => 1,
'AM_PROG_INSTALL_STRIP' => 1,
'_m4_warn' => 1,
'AM_SANITY_CHECK' => 1,
'AM_SILENT_RULES' => 1,
'include' => 1,
'_AM_PROG_TAR' => 1,
'AM_AUX_DIR_EXPAND' => 1,
'AM_DEP_TRACK' => 1,
'_AM_SET_OPTIONS' => 1,
'_AM_OUTPUT_DEPENDENCY_COMMANDS' => 1,
'AM_RUN_LOG' => 1,
'AC_CONFIG_MACRO_DIR' => 1,
'_AM_IF_OPTION' => 1,
'_AM_SUBST_NOTMAKE' => 1,
'm4_pattern_allow' => 1,
'_AM_AUTOCONF_VERSION' => 1,
'AM_PROG_CC_C_O' => 1,
'_AM_MANGLE_OPTION' => 1,
'AM_CONDITIONAL' => 1,
'AM_SET_LEADING_DOT' => 1,
'AC_CONFIG_MACRO_DIR_TRACE' => 1,
'AM_SET_DEPDIR' => 1,
'_AM_DEPENDENCIES' => 1,
'AM_PROG_INSTALL_SH' => 1,
'm4_include' => 1,
'_AM_PROG_CC_C_O' => 1,
'_AC_AM_CONFIG_HEADER_HOOK' => 1,
'AU_DEFUN' => 1,
'AM_MAKE_INCLUDE' => 1
}
], 'Autom4te::Request' ),
bless( [
'1',
1,
[
'/opt/local/share/autoconf'
],
[
'/opt/local/share/autoconf/autoconf/autoconf.m4f',
'aclocal.m4',
'configure.ac'
],
{
'AM_PROG_F77_C_O' => 1,
'_LT_AC_TAGCONFIG' => 1,
'm4_pattern_forbid' => 1,
'AC_INIT' => 1,
'_AM_COND_IF' => 1,
'AC_CANONICAL_TARGET' => 1,
'AC_SUBST' => 1,
'AC_CONFIG_LIBOBJ_DIR' => 1,
'AM_EXTRA_RECURSIVE_TARGETS' => 1,
'AC_FC_SRCEXT' => 1,
'AC_CANONICAL_HOST' => 1,
'AC_PROG_LIBTOOL' => 1,
'AM_PROG_MKDIR_P' => 1,
'AM_INIT_AUTOMAKE' => 1,
'AM_PATH_GUILE' => 1,
'AC_CONFIG_SUBDIRS' => 1,
'AM_AUTOMAKE_VERSION' => 1,
'LT_CONFIG_LTDL_DIR' => 1,
'AC_REQUIRE_AUX_FILE' => 1,
'AC_CONFIG_LINKS' => 1,
'LT_SUPPORTED_TAG' => 1,
'm4_sinclude' => 1,
'AM_MAINTAINER_MODE' => 1,
'AM_NLS' => 1,
'AC_FC_PP_DEFINE' => 1,
'AM_GNU_GETTEXT_INTL_SUBDIR' => 1,
'AM_MAKEFILE_INCLUDE' => 1,
'_m4_warn' => 1,
'AM_PROG_CXX_C_O' => 1,
'_AM_MAKEFILE_INCLUDE' => 1,
'_AM_COND_ENDIF' => 1,
'AM_ENABLE_MULTILIB' => 1,
'AM_SILENT_RULES' => 1,
'AM_PROG_MOC' => 1,
'AC_CONFIG_FILES' => 1,
'LT_INIT' => 1,
'include' => 1,
'AM_PROG_AR' => 1,
'AM_GNU_GETTEXT' => 1,
'AC_LIBSOURCE' => 1,
'AM_PROG_FC_C_O' => 1,
'AC_CANONICAL_BUILD' => 1,
'AC_FC_FREEFORM' => 1,
'AH_OUTPUT' => 1,
'AC_FC_PP_SRCEXT' => 1,
'_AM_SUBST_NOTMAKE' => 1,
'AC_CONFIG_AUX_DIR' => 1,
'AM_PROG_CC_C_O' => 1,
'm4_pattern_allow' => 1,
'sinclude' => 1,
'AM_XGETTEXT_OPTION' => 1,
'AC_CANONICAL_SYSTEM' => 1,
'AM_CONDITIONAL' => 1,
'AC_CONFIG_HEADERS' => 1,
'AC_DEFINE_TRACE_LITERAL' => 1,
'AM_POT_TOOLS' => 1,
'm4_include' => 1,
'_AM_COND_ELSE' => 1,
'AC_SUBST_TRACE' => 1
}
], 'Autom4te::Request' )
);

1090
autom4te.cache/traces.0 Normal file
View File

File diff suppressed because it is too large Load Diff

909
autom4te.cache/traces.1 Normal file
View File

@ -0,0 +1,909 @@
m4trace:configure.ac:6: -1- AC_INIT([nfdump], [1.6.13], [phaag@users.sourceforge.net])
m4trace:configure.ac:6: -1- m4_pattern_forbid([^_?A[CHUM]_])
m4trace:configure.ac:6: -1- m4_pattern_forbid([_AC_])
m4trace:configure.ac:6: -1- m4_pattern_forbid([^LIBOBJS$], [do not use LIBOBJS directly, use AC_LIBOBJ (see section `AC_LIBOBJ vs LIBOBJS'])
m4trace:configure.ac:6: -1- m4_pattern_allow([^AS_FLAGS$])
m4trace:configure.ac:6: -1- m4_pattern_forbid([^_?m4_])
m4trace:configure.ac:6: -1- m4_pattern_forbid([^dnl$])
m4trace:configure.ac:6: -1- m4_pattern_forbid([^_?AS_])
m4trace:configure.ac:6: -1- AC_SUBST([SHELL])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([SHELL])
m4trace:configure.ac:6: -1- m4_pattern_allow([^SHELL$])
m4trace:configure.ac:6: -1- AC_SUBST([PATH_SEPARATOR])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PATH_SEPARATOR])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PATH_SEPARATOR$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_NAME], [m4_ifdef([AC_PACKAGE_NAME], ['AC_PACKAGE_NAME'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_NAME])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_NAME$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_TARNAME], [m4_ifdef([AC_PACKAGE_TARNAME], ['AC_PACKAGE_TARNAME'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_TARNAME])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_TARNAME$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_VERSION], [m4_ifdef([AC_PACKAGE_VERSION], ['AC_PACKAGE_VERSION'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_VERSION])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_VERSION$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_STRING], [m4_ifdef([AC_PACKAGE_STRING], ['AC_PACKAGE_STRING'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_STRING])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_STRING$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_BUGREPORT], [m4_ifdef([AC_PACKAGE_BUGREPORT], ['AC_PACKAGE_BUGREPORT'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_BUGREPORT])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_BUGREPORT$])
m4trace:configure.ac:6: -1- AC_SUBST([PACKAGE_URL], [m4_ifdef([AC_PACKAGE_URL], ['AC_PACKAGE_URL'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([PACKAGE_URL])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_URL$])
m4trace:configure.ac:6: -1- AC_SUBST([exec_prefix], [NONE])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([exec_prefix])
m4trace:configure.ac:6: -1- m4_pattern_allow([^exec_prefix$])
m4trace:configure.ac:6: -1- AC_SUBST([prefix], [NONE])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([prefix])
m4trace:configure.ac:6: -1- m4_pattern_allow([^prefix$])
m4trace:configure.ac:6: -1- AC_SUBST([program_transform_name], [s,x,x,])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([program_transform_name])
m4trace:configure.ac:6: -1- m4_pattern_allow([^program_transform_name$])
m4trace:configure.ac:6: -1- AC_SUBST([bindir], ['${exec_prefix}/bin'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([bindir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^bindir$])
m4trace:configure.ac:6: -1- AC_SUBST([sbindir], ['${exec_prefix}/sbin'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([sbindir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^sbindir$])
m4trace:configure.ac:6: -1- AC_SUBST([libexecdir], ['${exec_prefix}/libexec'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([libexecdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^libexecdir$])
m4trace:configure.ac:6: -1- AC_SUBST([datarootdir], ['${prefix}/share'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([datarootdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^datarootdir$])
m4trace:configure.ac:6: -1- AC_SUBST([datadir], ['${datarootdir}'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([datadir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^datadir$])
m4trace:configure.ac:6: -1- AC_SUBST([sysconfdir], ['${prefix}/etc'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([sysconfdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^sysconfdir$])
m4trace:configure.ac:6: -1- AC_SUBST([sharedstatedir], ['${prefix}/com'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([sharedstatedir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^sharedstatedir$])
m4trace:configure.ac:6: -1- AC_SUBST([localstatedir], ['${prefix}/var'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([localstatedir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^localstatedir$])
m4trace:configure.ac:6: -1- AC_SUBST([includedir], ['${prefix}/include'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([includedir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^includedir$])
m4trace:configure.ac:6: -1- AC_SUBST([oldincludedir], ['/usr/include'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([oldincludedir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^oldincludedir$])
m4trace:configure.ac:6: -1- AC_SUBST([docdir], [m4_ifset([AC_PACKAGE_TARNAME],
['${datarootdir}/doc/${PACKAGE_TARNAME}'],
['${datarootdir}/doc/${PACKAGE}'])])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([docdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^docdir$])
m4trace:configure.ac:6: -1- AC_SUBST([infodir], ['${datarootdir}/info'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([infodir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^infodir$])
m4trace:configure.ac:6: -1- AC_SUBST([htmldir], ['${docdir}'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([htmldir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^htmldir$])
m4trace:configure.ac:6: -1- AC_SUBST([dvidir], ['${docdir}'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([dvidir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^dvidir$])
m4trace:configure.ac:6: -1- AC_SUBST([pdfdir], ['${docdir}'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([pdfdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^pdfdir$])
m4trace:configure.ac:6: -1- AC_SUBST([psdir], ['${docdir}'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([psdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^psdir$])
m4trace:configure.ac:6: -1- AC_SUBST([libdir], ['${exec_prefix}/lib'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([libdir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^libdir$])
m4trace:configure.ac:6: -1- AC_SUBST([localedir], ['${datarootdir}/locale'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([localedir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^localedir$])
m4trace:configure.ac:6: -1- AC_SUBST([mandir], ['${datarootdir}/man'])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([mandir])
m4trace:configure.ac:6: -1- m4_pattern_allow([^mandir$])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_NAME])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_NAME$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_NAME], [/* Define to the full name of this package. */
@%:@undef PACKAGE_NAME])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_TARNAME])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_TARNAME$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_TARNAME], [/* Define to the one symbol short name of this package. */
@%:@undef PACKAGE_TARNAME])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_VERSION])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_VERSION$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_VERSION], [/* Define to the version of this package. */
@%:@undef PACKAGE_VERSION])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_STRING])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_STRING$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_STRING], [/* Define to the full name and version of this package. */
@%:@undef PACKAGE_STRING])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_BUGREPORT])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_BUGREPORT$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_BUGREPORT], [/* Define to the address where bug reports for this package should be sent. */
@%:@undef PACKAGE_BUGREPORT])
m4trace:configure.ac:6: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE_URL])
m4trace:configure.ac:6: -1- m4_pattern_allow([^PACKAGE_URL$])
m4trace:configure.ac:6: -1- AH_OUTPUT([PACKAGE_URL], [/* Define to the home page for this package. */
@%:@undef PACKAGE_URL])
m4trace:configure.ac:6: -1- AC_SUBST([DEFS])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([DEFS])
m4trace:configure.ac:6: -1- m4_pattern_allow([^DEFS$])
m4trace:configure.ac:6: -1- AC_SUBST([ECHO_C])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([ECHO_C])
m4trace:configure.ac:6: -1- m4_pattern_allow([^ECHO_C$])
m4trace:configure.ac:6: -1- AC_SUBST([ECHO_N])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([ECHO_N])
m4trace:configure.ac:6: -1- m4_pattern_allow([^ECHO_N$])
m4trace:configure.ac:6: -1- AC_SUBST([ECHO_T])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([ECHO_T])
m4trace:configure.ac:6: -1- m4_pattern_allow([^ECHO_T$])
m4trace:configure.ac:6: -1- AC_SUBST([LIBS])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([LIBS])
m4trace:configure.ac:6: -1- m4_pattern_allow([^LIBS$])
m4trace:configure.ac:6: -1- AC_SUBST([build_alias])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([build_alias])
m4trace:configure.ac:6: -1- m4_pattern_allow([^build_alias$])
m4trace:configure.ac:6: -1- AC_SUBST([host_alias])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([host_alias])
m4trace:configure.ac:6: -1- m4_pattern_allow([^host_alias$])
m4trace:configure.ac:6: -1- AC_SUBST([target_alias])
m4trace:configure.ac:6: -1- AC_SUBST_TRACE([target_alias])
m4trace:configure.ac:6: -1- m4_pattern_allow([^target_alias$])
m4trace:configure.ac:9: -1- AC_CONFIG_HEADERS([config.h])
m4trace:configure.ac:10: -1- AM_INIT_AUTOMAKE([subdir-objects])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AM_[A-Z]+FLAGS$])
m4trace:configure.ac:10: -1- AM_AUTOMAKE_VERSION([1.14.1])
m4trace:configure.ac:10: -1- AC_REQUIRE_AUX_FILE([install-sh])
m4trace:configure.ac:10: -1- AC_SUBST([INSTALL_PROGRAM])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([INSTALL_PROGRAM])
m4trace:configure.ac:10: -1- m4_pattern_allow([^INSTALL_PROGRAM$])
m4trace:configure.ac:10: -1- AC_SUBST([INSTALL_SCRIPT])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([INSTALL_SCRIPT])
m4trace:configure.ac:10: -1- m4_pattern_allow([^INSTALL_SCRIPT$])
m4trace:configure.ac:10: -1- AC_SUBST([INSTALL_DATA])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([INSTALL_DATA])
m4trace:configure.ac:10: -1- m4_pattern_allow([^INSTALL_DATA$])
m4trace:configure.ac:10: -1- AC_SUBST([am__isrc], [' -I$(srcdir)'])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([am__isrc])
m4trace:configure.ac:10: -1- m4_pattern_allow([^am__isrc$])
m4trace:configure.ac:10: -1- _AM_SUBST_NOTMAKE([am__isrc])
m4trace:configure.ac:10: -1- AC_SUBST([CYGPATH_W])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([CYGPATH_W])
m4trace:configure.ac:10: -1- m4_pattern_allow([^CYGPATH_W$])
m4trace:configure.ac:10: -1- AC_SUBST([PACKAGE], ['AC_PACKAGE_TARNAME'])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([PACKAGE])
m4trace:configure.ac:10: -1- m4_pattern_allow([^PACKAGE$])
m4trace:configure.ac:10: -1- AC_SUBST([VERSION], ['AC_PACKAGE_VERSION'])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([VERSION])
m4trace:configure.ac:10: -1- m4_pattern_allow([^VERSION$])
m4trace:configure.ac:10: -1- AC_DEFINE_TRACE_LITERAL([PACKAGE])
m4trace:configure.ac:10: -1- m4_pattern_allow([^PACKAGE$])
m4trace:configure.ac:10: -1- AH_OUTPUT([PACKAGE], [/* Name of package */
@%:@undef PACKAGE])
m4trace:configure.ac:10: -1- AC_DEFINE_TRACE_LITERAL([VERSION])
m4trace:configure.ac:10: -1- m4_pattern_allow([^VERSION$])
m4trace:configure.ac:10: -1- AH_OUTPUT([VERSION], [/* Version number of package */
@%:@undef VERSION])
m4trace:configure.ac:10: -1- AC_REQUIRE_AUX_FILE([missing])
m4trace:configure.ac:10: -1- AC_SUBST([ACLOCAL])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([ACLOCAL])
m4trace:configure.ac:10: -1- m4_pattern_allow([^ACLOCAL$])
m4trace:configure.ac:10: -1- AC_SUBST([AUTOCONF])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AUTOCONF])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AUTOCONF$])
m4trace:configure.ac:10: -1- AC_SUBST([AUTOMAKE])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AUTOMAKE])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AUTOMAKE$])
m4trace:configure.ac:10: -1- AC_SUBST([AUTOHEADER])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AUTOHEADER])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AUTOHEADER$])
m4trace:configure.ac:10: -1- AC_SUBST([MAKEINFO])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([MAKEINFO])
m4trace:configure.ac:10: -1- m4_pattern_allow([^MAKEINFO$])
m4trace:configure.ac:10: -1- AC_SUBST([install_sh])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([install_sh])
m4trace:configure.ac:10: -1- m4_pattern_allow([^install_sh$])
m4trace:configure.ac:10: -1- AC_SUBST([STRIP])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([STRIP])
m4trace:configure.ac:10: -1- m4_pattern_allow([^STRIP$])
m4trace:configure.ac:10: -1- AC_SUBST([INSTALL_STRIP_PROGRAM])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([INSTALL_STRIP_PROGRAM])
m4trace:configure.ac:10: -1- m4_pattern_allow([^INSTALL_STRIP_PROGRAM$])
m4trace:configure.ac:10: -1- AC_REQUIRE_AUX_FILE([install-sh])
m4trace:configure.ac:10: -1- AC_SUBST([MKDIR_P])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([MKDIR_P])
m4trace:configure.ac:10: -1- m4_pattern_allow([^MKDIR_P$])
m4trace:configure.ac:10: -1- AC_SUBST([mkdir_p], ['$(MKDIR_P)'])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([mkdir_p])
m4trace:configure.ac:10: -1- m4_pattern_allow([^mkdir_p$])
m4trace:configure.ac:10: -1- AC_SUBST([AWK])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AWK])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AWK$])
m4trace:configure.ac:10: -1- AC_SUBST([SET_MAKE])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([SET_MAKE])
m4trace:configure.ac:10: -1- m4_pattern_allow([^SET_MAKE$])
m4trace:configure.ac:10: -1- AC_SUBST([am__leading_dot])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([am__leading_dot])
m4trace:configure.ac:10: -1- m4_pattern_allow([^am__leading_dot$])
m4trace:configure.ac:10: -1- AC_SUBST([AMTAR], ['$${TAR-tar}'])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AMTAR])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AMTAR$])
m4trace:configure.ac:10: -1- AC_SUBST([am__tar])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([am__tar])
m4trace:configure.ac:10: -1- m4_pattern_allow([^am__tar$])
m4trace:configure.ac:10: -1- AC_SUBST([am__untar])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([am__untar])
m4trace:configure.ac:10: -1- m4_pattern_allow([^am__untar$])
m4trace:configure.ac:10: -1- AM_SILENT_RULES
m4trace:configure.ac:10: -1- AC_SUBST([AM_V])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AM_V])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AM_V$])
m4trace:configure.ac:10: -1- _AM_SUBST_NOTMAKE([AM_V])
m4trace:configure.ac:10: -1- AC_SUBST([AM_DEFAULT_V])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AM_DEFAULT_V])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AM_DEFAULT_V$])
m4trace:configure.ac:10: -1- _AM_SUBST_NOTMAKE([AM_DEFAULT_V])
m4trace:configure.ac:10: -1- AC_SUBST([AM_DEFAULT_VERBOSITY])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AM_DEFAULT_VERBOSITY])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AM_DEFAULT_VERBOSITY$])
m4trace:configure.ac:10: -1- AC_SUBST([AM_BACKSLASH])
m4trace:configure.ac:10: -1- AC_SUBST_TRACE([AM_BACKSLASH])
m4trace:configure.ac:10: -1- m4_pattern_allow([^AM_BACKSLASH$])
m4trace:configure.ac:10: -1- _AM_SUBST_NOTMAKE([AM_BACKSLASH])
m4trace:configure.ac:13: -1- AC_SUBST([CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CC$])
m4trace:configure.ac:13: -1- AC_SUBST([CFLAGS])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CFLAGS])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CFLAGS$])
m4trace:configure.ac:13: -1- AC_SUBST([LDFLAGS])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([LDFLAGS])
m4trace:configure.ac:13: -1- m4_pattern_allow([^LDFLAGS$])
m4trace:configure.ac:13: -1- AC_SUBST([LIBS])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([LIBS])
m4trace:configure.ac:13: -1- m4_pattern_allow([^LIBS$])
m4trace:configure.ac:13: -1- AC_SUBST([CPPFLAGS])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CPPFLAGS])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CPPFLAGS$])
m4trace:configure.ac:13: -1- AC_SUBST([CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CC$])
m4trace:configure.ac:13: -1- AC_SUBST([CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CC$])
m4trace:configure.ac:13: -1- AC_SUBST([CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CC$])
m4trace:configure.ac:13: -1- AC_SUBST([CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CC$])
m4trace:configure.ac:13: -1- AC_SUBST([ac_ct_CC])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([ac_ct_CC])
m4trace:configure.ac:13: -1- m4_pattern_allow([^ac_ct_CC$])
m4trace:configure.ac:13: -1- AC_SUBST([EXEEXT], [$ac_cv_exeext])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([EXEEXT])
m4trace:configure.ac:13: -1- m4_pattern_allow([^EXEEXT$])
m4trace:configure.ac:13: -1- AC_SUBST([OBJEXT], [$ac_cv_objext])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([OBJEXT])
m4trace:configure.ac:13: -1- m4_pattern_allow([^OBJEXT$])
m4trace:configure.ac:13: -1- AC_REQUIRE_AUX_FILE([compile])
m4trace:configure.ac:13: -1- AC_SUBST([DEPDIR], ["${am__leading_dot}deps"])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([DEPDIR])
m4trace:configure.ac:13: -1- m4_pattern_allow([^DEPDIR$])
m4trace:configure.ac:13: -1- AC_SUBST([am__include])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([am__include])
m4trace:configure.ac:13: -1- m4_pattern_allow([^am__include$])
m4trace:configure.ac:13: -1- AC_SUBST([am__quote])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([am__quote])
m4trace:configure.ac:13: -1- m4_pattern_allow([^am__quote$])
m4trace:configure.ac:13: -1- AM_CONDITIONAL([AMDEP], [test "x$enable_dependency_tracking" != xno])
m4trace:configure.ac:13: -1- AC_SUBST([AMDEP_TRUE])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([AMDEP_TRUE])
m4trace:configure.ac:13: -1- m4_pattern_allow([^AMDEP_TRUE$])
m4trace:configure.ac:13: -1- AC_SUBST([AMDEP_FALSE])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([AMDEP_FALSE])
m4trace:configure.ac:13: -1- m4_pattern_allow([^AMDEP_FALSE$])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([AMDEP_TRUE])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([AMDEP_FALSE])
m4trace:configure.ac:13: -1- AC_SUBST([AMDEPBACKSLASH])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([AMDEPBACKSLASH])
m4trace:configure.ac:13: -1- m4_pattern_allow([^AMDEPBACKSLASH$])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([AMDEPBACKSLASH])
m4trace:configure.ac:13: -1- AC_SUBST([am__nodep])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([am__nodep])
m4trace:configure.ac:13: -1- m4_pattern_allow([^am__nodep$])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([am__nodep])
m4trace:configure.ac:13: -1- AC_SUBST([CCDEPMODE], [depmode=$am_cv_CC_dependencies_compiler_type])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([CCDEPMODE])
m4trace:configure.ac:13: -1- m4_pattern_allow([^CCDEPMODE$])
m4trace:configure.ac:13: -1- AM_CONDITIONAL([am__fastdepCC], [
test "x$enable_dependency_tracking" != xno \
&& test "$am_cv_CC_dependencies_compiler_type" = gcc3])
m4trace:configure.ac:13: -1- AC_SUBST([am__fastdepCC_TRUE])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([am__fastdepCC_TRUE])
m4trace:configure.ac:13: -1- m4_pattern_allow([^am__fastdepCC_TRUE$])
m4trace:configure.ac:13: -1- AC_SUBST([am__fastdepCC_FALSE])
m4trace:configure.ac:13: -1- AC_SUBST_TRACE([am__fastdepCC_FALSE])
m4trace:configure.ac:13: -1- m4_pattern_allow([^am__fastdepCC_FALSE$])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([am__fastdepCC_TRUE])
m4trace:configure.ac:13: -1- _AM_SUBST_NOTMAKE([am__fastdepCC_FALSE])
m4trace:configure.ac:14: -1- AM_PROG_CC_C_O
m4trace:configure.ac:18: -1- _m4_warn([obsolete], [The macro `ac_cv_prog_gcc' is obsolete.
You should run autoupdate.], [../../lib/autoconf/c.m4:436: ac_cv_prog_gcc is expanded from...
configure.ac:18: the top level])
m4trace:configure.ac:65: -1- AC_SUBST([YACC])
m4trace:configure.ac:65: -1- AC_SUBST_TRACE([YACC])
m4trace:configure.ac:65: -1- m4_pattern_allow([^YACC$])
m4trace:configure.ac:65: -1- AC_SUBST([YACC])
m4trace:configure.ac:65: -1- AC_SUBST_TRACE([YACC])
m4trace:configure.ac:65: -1- m4_pattern_allow([^YACC$])
m4trace:configure.ac:65: -1- AC_SUBST([YFLAGS])
m4trace:configure.ac:65: -1- AC_SUBST_TRACE([YFLAGS])
m4trace:configure.ac:65: -1- m4_pattern_allow([^YFLAGS$])
m4trace:configure.ac:66: -1- AC_SUBST([LEX])
m4trace:configure.ac:66: -1- AC_SUBST_TRACE([LEX])
m4trace:configure.ac:66: -1- m4_pattern_allow([^LEX$])
m4trace:configure.ac:66: -1- AC_SUBST([LEX_OUTPUT_ROOT], [$ac_cv_prog_lex_root])
m4trace:configure.ac:66: -1- AC_SUBST_TRACE([LEX_OUTPUT_ROOT])
m4trace:configure.ac:66: -1- m4_pattern_allow([^LEX_OUTPUT_ROOT$])
m4trace:configure.ac:66: -1- AC_SUBST([LEXLIB])
m4trace:configure.ac:66: -1- AC_SUBST_TRACE([LEXLIB])
m4trace:configure.ac:66: -1- m4_pattern_allow([^LEXLIB$])
m4trace:configure.ac:66: -1- AC_DEFINE_TRACE_LITERAL([YYTEXT_POINTER])
m4trace:configure.ac:66: -1- m4_pattern_allow([^YYTEXT_POINTER$])
m4trace:configure.ac:66: -1- AH_OUTPUT([YYTEXT_POINTER], [/* Define to 1 if `lex\' declares `yytext\' as a `char *\' by default, not a
`char@<:@@:>@\'. */
@%:@undef YYTEXT_POINTER])
m4trace:configure.ac:105: -2- AH_OUTPUT([HAVE_LIBZ], [/* Define to 1 if you have the `z\' library (-lz). */
@%:@undef HAVE_LIBZ])
m4trace:configure.ac:105: -2- AC_DEFINE_TRACE_LITERAL([HAVE_LIBZ])
m4trace:configure.ac:105: -2- m4_pattern_allow([^HAVE_LIBZ$])
m4trace:configure.ac:120: -2- AM_CONDITIONAL([FT2NFDUMP], [true])
m4trace:configure.ac:120: -2- AC_SUBST([FT2NFDUMP_TRUE])
m4trace:configure.ac:120: -2- AC_SUBST_TRACE([FT2NFDUMP_TRUE])
m4trace:configure.ac:120: -2- m4_pattern_allow([^FT2NFDUMP_TRUE$])
m4trace:configure.ac:120: -2- AC_SUBST([FT2NFDUMP_FALSE])
m4trace:configure.ac:120: -2- AC_SUBST_TRACE([FT2NFDUMP_FALSE])
m4trace:configure.ac:120: -2- m4_pattern_allow([^FT2NFDUMP_FALSE$])
m4trace:configure.ac:120: -2- _AM_SUBST_NOTMAKE([FT2NFDUMP_TRUE])
m4trace:configure.ac:120: -2- _AM_SUBST_NOTMAKE([FT2NFDUMP_FALSE])
m4trace:configure.ac:122: -2- AM_CONDITIONAL([FT2NFDUMP], [false])
m4trace:configure.ac:122: -2- AC_SUBST([FT2NFDUMP_TRUE])
m4trace:configure.ac:122: -2- AC_SUBST_TRACE([FT2NFDUMP_TRUE])
m4trace:configure.ac:122: -2- m4_pattern_allow([^FT2NFDUMP_TRUE$])
m4trace:configure.ac:122: -2- AC_SUBST([FT2NFDUMP_FALSE])
m4trace:configure.ac:122: -2- AC_SUBST_TRACE([FT2NFDUMP_FALSE])
m4trace:configure.ac:122: -2- m4_pattern_allow([^FT2NFDUMP_FALSE$])
m4trace:configure.ac:122: -2- _AM_SUBST_NOTMAKE([FT2NFDUMP_TRUE])
m4trace:configure.ac:122: -2- _AM_SUBST_NOTMAKE([FT2NFDUMP_FALSE])
m4trace:configure.ac:156: -2- AM_CONDITIONAL([NFPROFILE], [false])
m4trace:configure.ac:156: -2- AC_SUBST([NFPROFILE_TRUE])
m4trace:configure.ac:156: -2- AC_SUBST_TRACE([NFPROFILE_TRUE])
m4trace:configure.ac:156: -2- m4_pattern_allow([^NFPROFILE_TRUE$])
m4trace:configure.ac:156: -2- AC_SUBST([NFPROFILE_FALSE])
m4trace:configure.ac:156: -2- AC_SUBST_TRACE([NFPROFILE_FALSE])
m4trace:configure.ac:156: -2- m4_pattern_allow([^NFPROFILE_FALSE$])
m4trace:configure.ac:156: -2- _AM_SUBST_NOTMAKE([NFPROFILE_TRUE])
m4trace:configure.ac:156: -2- _AM_SUBST_NOTMAKE([NFPROFILE_FALSE])
m4trace:configure.ac:126: -1- AC_SUBST([RRD_LIBS])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([RRD_LIBS])
m4trace:configure.ac:126: -1- m4_pattern_allow([^RRD_LIBS$])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_RRD_H], [/* Define to 1 if you have the <rrd.h> header file. */
@%:@undef HAVE_RRD_H])
m4trace:configure.ac:126: -1- AC_SUBST([CPP])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([CPP])
m4trace:configure.ac:126: -1- m4_pattern_allow([^CPP$])
m4trace:configure.ac:126: -1- AC_SUBST([CPPFLAGS])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([CPPFLAGS])
m4trace:configure.ac:126: -1- m4_pattern_allow([^CPPFLAGS$])
m4trace:configure.ac:126: -1- AC_SUBST([CPP])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([CPP])
m4trace:configure.ac:126: -1- m4_pattern_allow([^CPP$])
m4trace:configure.ac:126: -1- AC_SUBST([GREP])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([GREP])
m4trace:configure.ac:126: -1- m4_pattern_allow([^GREP$])
m4trace:configure.ac:126: -1- AC_SUBST([EGREP])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([EGREP])
m4trace:configure.ac:126: -1- m4_pattern_allow([^EGREP$])
m4trace:configure.ac:126: -1- AC_DEFINE_TRACE_LITERAL([STDC_HEADERS])
m4trace:configure.ac:126: -1- m4_pattern_allow([^STDC_HEADERS$])
m4trace:configure.ac:126: -1- AH_OUTPUT([STDC_HEADERS], [/* Define to 1 if you have the ANSI C header files. */
@%:@undef STDC_HEADERS])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_SYS_TYPES_H], [/* Define to 1 if you have the <sys/types.h> header file. */
@%:@undef HAVE_SYS_TYPES_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_SYS_STAT_H], [/* Define to 1 if you have the <sys/stat.h> header file. */
@%:@undef HAVE_SYS_STAT_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_STDLIB_H], [/* Define to 1 if you have the <stdlib.h> header file. */
@%:@undef HAVE_STDLIB_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_STRING_H], [/* Define to 1 if you have the <string.h> header file. */
@%:@undef HAVE_STRING_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_MEMORY_H], [/* Define to 1 if you have the <memory.h> header file. */
@%:@undef HAVE_MEMORY_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_STRINGS_H], [/* Define to 1 if you have the <strings.h> header file. */
@%:@undef HAVE_STRINGS_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_INTTYPES_H], [/* Define to 1 if you have the <inttypes.h> header file. */
@%:@undef HAVE_INTTYPES_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_STDINT_H], [/* Define to 1 if you have the <stdint.h> header file. */
@%:@undef HAVE_STDINT_H])
m4trace:configure.ac:126: -1- AH_OUTPUT([HAVE_UNISTD_H], [/* Define to 1 if you have the <unistd.h> header file. */
@%:@undef HAVE_UNISTD_H])
m4trace:configure.ac:126: -1- AC_DEFINE_TRACE_LITERAL([HAVE_RRD_H])
m4trace:configure.ac:126: -1- m4_pattern_allow([^HAVE_RRD_H$])
m4trace:configure.ac:126: -1- AM_CONDITIONAL([NFPROFILE], [true])
m4trace:configure.ac:126: -1- AC_SUBST([NFPROFILE_TRUE])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([NFPROFILE_TRUE])
m4trace:configure.ac:126: -1- m4_pattern_allow([^NFPROFILE_TRUE$])
m4trace:configure.ac:126: -1- AC_SUBST([NFPROFILE_FALSE])
m4trace:configure.ac:126: -1- AC_SUBST_TRACE([NFPROFILE_FALSE])
m4trace:configure.ac:126: -1- m4_pattern_allow([^NFPROFILE_FALSE$])
m4trace:configure.ac:126: -1- _AM_SUBST_NOTMAKE([NFPROFILE_TRUE])
m4trace:configure.ac:126: -1- _AM_SUBST_NOTMAKE([NFPROFILE_FALSE])
m4trace:configure.ac:126: -1- _m4_warn([cross], [AC_RUN_IFELSE called without default to allow cross compiling], [../../lib/autoconf/general.m4:2748: AC_RUN_IFELSE is expanded from...
../../lib/m4sugar/m4sh.m4:639: AS_IF is expanded from...
../../lib/autoconf/general.m4:1462: AC_ARG_ENABLE is expanded from...
configure.ac:126: the top level])
m4trace:configure.ac:189: -2- AM_CONDITIONAL([NFTRACK], [false])
m4trace:configure.ac:189: -2- AC_SUBST([NFTRACK_TRUE])
m4trace:configure.ac:189: -2- AC_SUBST_TRACE([NFTRACK_TRUE])
m4trace:configure.ac:189: -2- m4_pattern_allow([^NFTRACK_TRUE$])
m4trace:configure.ac:189: -2- AC_SUBST([NFTRACK_FALSE])
m4trace:configure.ac:189: -2- AC_SUBST_TRACE([NFTRACK_FALSE])
m4trace:configure.ac:189: -2- m4_pattern_allow([^NFTRACK_FALSE$])
m4trace:configure.ac:189: -2- _AM_SUBST_NOTMAKE([NFTRACK_TRUE])
m4trace:configure.ac:189: -2- _AM_SUBST_NOTMAKE([NFTRACK_FALSE])
m4trace:configure.ac:159: -1- AC_SUBST([RRD_LIBS])
m4trace:configure.ac:159: -1- AC_SUBST_TRACE([RRD_LIBS])
m4trace:configure.ac:159: -1- m4_pattern_allow([^RRD_LIBS$])
m4trace:configure.ac:159: -1- AH_OUTPUT([HAVE_RRD_H], [/* Define to 1 if you have the <rrd.h> header file. */
@%:@undef HAVE_RRD_H])
m4trace:configure.ac:159: -1- AC_DEFINE_TRACE_LITERAL([HAVE_RRD_H])
m4trace:configure.ac:159: -1- m4_pattern_allow([^HAVE_RRD_H$])
m4trace:configure.ac:159: -1- AM_CONDITIONAL([NFTRACK], [true])
m4trace:configure.ac:159: -1- AC_SUBST([NFTRACK_TRUE])
m4trace:configure.ac:159: -1- AC_SUBST_TRACE([NFTRACK_TRUE])
m4trace:configure.ac:159: -1- m4_pattern_allow([^NFTRACK_TRUE$])
m4trace:configure.ac:159: -1- AC_SUBST([NFTRACK_FALSE])
m4trace:configure.ac:159: -1- AC_SUBST_TRACE([NFTRACK_FALSE])
m4trace:configure.ac:159: -1- m4_pattern_allow([^NFTRACK_FALSE$])
m4trace:configure.ac:159: -1- _AM_SUBST_NOTMAKE([NFTRACK_TRUE])
m4trace:configure.ac:159: -1- _AM_SUBST_NOTMAKE([NFTRACK_FALSE])
m4trace:configure.ac:159: -1- _m4_warn([cross], [AC_RUN_IFELSE called without default to allow cross compiling], [../../lib/autoconf/general.m4:2748: AC_RUN_IFELSE is expanded from...
../../lib/m4sugar/m4sh.m4:639: AS_IF is expanded from...
../../lib/autoconf/general.m4:1462: AC_ARG_ENABLE is expanded from...
configure.ac:159: the top level])
m4trace:configure.ac:194: -1- AM_CONDITIONAL([SFLOW], [test "$enable_sflow" = yes])
m4trace:configure.ac:194: -1- AC_SUBST([SFLOW_TRUE])
m4trace:configure.ac:194: -1- AC_SUBST_TRACE([SFLOW_TRUE])
m4trace:configure.ac:194: -1- m4_pattern_allow([^SFLOW_TRUE$])
m4trace:configure.ac:194: -1- AC_SUBST([SFLOW_FALSE])
m4trace:configure.ac:194: -1- AC_SUBST_TRACE([SFLOW_FALSE])
m4trace:configure.ac:194: -1- m4_pattern_allow([^SFLOW_FALSE$])
m4trace:configure.ac:194: -1- _AM_SUBST_NOTMAKE([SFLOW_TRUE])
m4trace:configure.ac:194: -1- _AM_SUBST_NOTMAKE([SFLOW_FALSE])
m4trace:configure.ac:198: -1- AM_CONDITIONAL([READPCAP], [test "$enable_readpcap" = yes])
m4trace:configure.ac:198: -1- AC_SUBST([READPCAP_TRUE])
m4trace:configure.ac:198: -1- AC_SUBST_TRACE([READPCAP_TRUE])
m4trace:configure.ac:198: -1- m4_pattern_allow([^READPCAP_TRUE$])
m4trace:configure.ac:198: -1- AC_SUBST([READPCAP_FALSE])
m4trace:configure.ac:198: -1- AC_SUBST_TRACE([READPCAP_FALSE])
m4trace:configure.ac:198: -1- m4_pattern_allow([^READPCAP_FALSE$])
m4trace:configure.ac:198: -1- _AM_SUBST_NOTMAKE([READPCAP_TRUE])
m4trace:configure.ac:198: -1- _AM_SUBST_NOTMAKE([READPCAP_FALSE])
m4trace:configure.ac:202: -1- AM_CONDITIONAL([BUILDNFPCAPD], [test "$enable_nfpcapd" = yes])
m4trace:configure.ac:202: -1- AC_SUBST([BUILDNFPCAPD_TRUE])
m4trace:configure.ac:202: -1- AC_SUBST_TRACE([BUILDNFPCAPD_TRUE])
m4trace:configure.ac:202: -1- m4_pattern_allow([^BUILDNFPCAPD_TRUE$])
m4trace:configure.ac:202: -1- AC_SUBST([BUILDNFPCAPD_FALSE])
m4trace:configure.ac:202: -1- AC_SUBST_TRACE([BUILDNFPCAPD_FALSE])
m4trace:configure.ac:202: -1- m4_pattern_allow([^BUILDNFPCAPD_FALSE$])
m4trace:configure.ac:202: -1- _AM_SUBST_NOTMAKE([BUILDNFPCAPD_TRUE])
m4trace:configure.ac:202: -1- _AM_SUBST_NOTMAKE([BUILDNFPCAPD_FALSE])
m4trace:configure.ac:206: -2- AC_DEFINE_TRACE_LITERAL([HAVE_SOCKADDR_SA_LEN])
m4trace:configure.ac:206: -2- m4_pattern_allow([^HAVE_SOCKADDR_SA_LEN$])
m4trace:configure.ac:206: -2- AH_OUTPUT([HAVE_SOCKADDR_SA_LEN], [/* define if socket address structures have length fields */
@%:@undef HAVE_SOCKADDR_SA_LEN])
m4trace:configure.ac:210: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY])
m4trace:configure.ac:210: -1- m4_pattern_allow([^HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY$])
m4trace:configure.ac:210: -1- AH_OUTPUT([HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY], [/* Define to 1 if `ss_family\' is a member of `struct sockaddr_storage\'. */
@%:@undef HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY])
m4trace:configure.ac:210: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRUCT_SOCKADDR_STORAGE___SS_FAMILY])
m4trace:configure.ac:210: -1- m4_pattern_allow([^HAVE_STRUCT_SOCKADDR_STORAGE___SS_FAMILY$])
m4trace:configure.ac:210: -1- AH_OUTPUT([HAVE_STRUCT_SOCKADDR_STORAGE___SS_FAMILY], [/* Define to 1 if `__ss_family\' is a member of `struct sockaddr_storage\'. */
@%:@undef HAVE_STRUCT_SOCKADDR_STORAGE___SS_FAMILY])
m4trace:configure.ac:210: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN])
m4trace:configure.ac:210: -1- m4_pattern_allow([^HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN$])
m4trace:configure.ac:210: -1- AH_OUTPUT([HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN], [/* Define to 1 if `ss_len\' is a member of `struct sockaddr_storage\'. */
@%:@undef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN])
m4trace:configure.ac:210: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRUCT_SOCKADDR_STORAGE___SS_LEN])
m4trace:configure.ac:210: -1- m4_pattern_allow([^HAVE_STRUCT_SOCKADDR_STORAGE___SS_LEN$])
m4trace:configure.ac:210: -1- AH_OUTPUT([HAVE_STRUCT_SOCKADDR_STORAGE___SS_LEN], [/* Define to 1 if `__ss_len\' is a member of `struct sockaddr_storage\'. */
@%:@undef HAVE_STRUCT_SOCKADDR_STORAGE___SS_LEN])
m4trace:configure.ac:210: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRUCT_SOCKADDR_SA_LEN])
m4trace:configure.ac:210: -1- m4_pattern_allow([^HAVE_STRUCT_SOCKADDR_SA_LEN$])
m4trace:configure.ac:210: -1- AH_OUTPUT([HAVE_STRUCT_SOCKADDR_SA_LEN], [/* Define to 1 if `sa_len\' is a member of `struct sockaddr\'. */
@%:@undef HAVE_STRUCT_SOCKADDR_SA_LEN])
m4trace:configure.ac:220: -1- AC_SUBST([FT_INCLUDES])
m4trace:configure.ac:220: -1- AC_SUBST_TRACE([FT_INCLUDES])
m4trace:configure.ac:220: -1- m4_pattern_allow([^FT_INCLUDES$])
m4trace:configure.ac:221: -1- AC_SUBST([FT_LDFLAGS])
m4trace:configure.ac:221: -1- AC_SUBST_TRACE([FT_LDFLAGS])
m4trace:configure.ac:221: -1- m4_pattern_allow([^FT_LDFLAGS$])
m4trace:configure.ac:222: -1- AC_SUBST([LFLAGS])
m4trace:configure.ac:222: -1- AC_SUBST_TRACE([LFLAGS])
m4trace:configure.ac:222: -1- m4_pattern_allow([^LFLAGS$])
m4trace:configure.ac:226: -1- AH_OUTPUT([HAVE_GETHOSTBYNAME], [/* Define to 1 if you have the `gethostbyname\' function. */
@%:@undef HAVE_GETHOSTBYNAME])
m4trace:configure.ac:226: -1- AC_DEFINE_TRACE_LITERAL([HAVE_GETHOSTBYNAME])
m4trace:configure.ac:226: -1- m4_pattern_allow([^HAVE_GETHOSTBYNAME$])
m4trace:configure.ac:226: -1- AH_OUTPUT([HAVE_LIBNSL], [/* Define to 1 if you have the `nsl\' library (-lnsl). */
@%:@undef HAVE_LIBNSL])
m4trace:configure.ac:226: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBNSL])
m4trace:configure.ac:226: -1- m4_pattern_allow([^HAVE_LIBNSL$])
m4trace:configure.ac:226: -1- AH_OUTPUT([HAVE_LIBSOCKET], [/* Define to 1 if you have the `socket\' library (-lsocket). */
@%:@undef HAVE_LIBSOCKET])
m4trace:configure.ac:226: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBSOCKET])
m4trace:configure.ac:226: -1- m4_pattern_allow([^HAVE_LIBSOCKET$])
m4trace:configure.ac:227: -1- AH_OUTPUT([HAVE_SETSOCKOPT], [/* Define to 1 if you have the `setsockopt\' function. */
@%:@undef HAVE_SETSOCKOPT])
m4trace:configure.ac:227: -1- AC_DEFINE_TRACE_LITERAL([HAVE_SETSOCKOPT])
m4trace:configure.ac:227: -1- m4_pattern_allow([^HAVE_SETSOCKOPT$])
m4trace:configure.ac:227: -1- AH_OUTPUT([HAVE_LIBSOCKET], [/* Define to 1 if you have the `socket\' library (-lsocket). */
@%:@undef HAVE_LIBSOCKET])
m4trace:configure.ac:227: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBSOCKET])
m4trace:configure.ac:227: -1- m4_pattern_allow([^HAVE_LIBSOCKET$])
m4trace:configure.ac:230: -1- AH_OUTPUT([HAVE_FPURGE], [/* Define to 1 if you have the `fpurge\' function. */
@%:@undef HAVE_FPURGE])
m4trace:configure.ac:230: -1- AH_OUTPUT([HAVE___FPURGE], [/* Define to 1 if you have the `__fpurge\' function. */
@%:@undef HAVE___FPURGE])
m4trace:configure.ac:235: -1- AC_DEFINE_TRACE_LITERAL([HAVE_HTONLL])
m4trace:configure.ac:235: -1- m4_pattern_allow([^HAVE_HTONLL$])
m4trace:configure.ac:235: -1- AH_OUTPUT([HAVE_HTONLL], [/* Define to 1 if the function (or macro) htonll exists. */
@%:@undef HAVE_HTONLL])
m4trace:configure.ac:254: -1- AH_OUTPUT([HAVE_DIRENT_H], [/* Define to 1 if you have the <dirent.h> header file, and it defines `DIR\'.
*/
@%:@undef HAVE_DIRENT_H])
m4trace:configure.ac:254: -1- AH_OUTPUT([HAVE_SYS_NDIR_H], [/* Define to 1 if you have the <sys/ndir.h> header file, and it defines `DIR\'.
*/
@%:@undef HAVE_SYS_NDIR_H])
m4trace:configure.ac:254: -1- AH_OUTPUT([HAVE_SYS_DIR_H], [/* Define to 1 if you have the <sys/dir.h> header file, and it defines `DIR\'.
*/
@%:@undef HAVE_SYS_DIR_H])
m4trace:configure.ac:254: -1- AH_OUTPUT([HAVE_NDIR_H], [/* Define to 1 if you have the <ndir.h> header file, and it defines `DIR\'. */
@%:@undef HAVE_NDIR_H])
m4trace:configure.ac:255: -1- AC_DEFINE_TRACE_LITERAL([STDC_HEADERS])
m4trace:configure.ac:255: -1- m4_pattern_allow([^STDC_HEADERS$])
m4trace:configure.ac:255: -1- AH_OUTPUT([STDC_HEADERS], [/* Define to 1 if you have the ANSI C header files. */
@%:@undef STDC_HEADERS])
m4trace:configure.ac:256: -1- AH_OUTPUT([HAVE_NAMESER8_COMPAT_H], [/* Define to 1 if you have the <nameser8_compat.h> header file. */
@%:@undef HAVE_NAMESER8_COMPAT_H])
m4trace:configure.ac:256: -1- AC_DEFINE_TRACE_LITERAL([HAVE_NAMESER8_COMPAT_H])
m4trace:configure.ac:256: -1- m4_pattern_allow([^HAVE_NAMESER8_COMPAT_H$])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_FEATURES_H], [/* Define to 1 if you have the <features.h> header file. */
@%:@undef HAVE_FEATURES_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_ARPA_INET_H], [/* Define to 1 if you have the <arpa/inet.h> header file. */
@%:@undef HAVE_ARPA_INET_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_FCNTL_H], [/* Define to 1 if you have the <fcntl.h> header file. */
@%:@undef HAVE_FCNTL_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_NETINET_IN_H], [/* Define to 1 if you have the <netinet/in.h> header file. */
@%:@undef HAVE_NETINET_IN_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_FTS_H], [/* Define to 1 if you have the <fts.h> header file. */
@%:@undef HAVE_FTS_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_STDINT_H], [/* Define to 1 if you have the <stdint.h> header file. */
@%:@undef HAVE_STDINT_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_STDLIB_H], [/* Define to 1 if you have the <stdlib.h> header file. */
@%:@undef HAVE_STDLIB_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_STDDEF_H], [/* Define to 1 if you have the <stddef.h> header file. */
@%:@undef HAVE_STDDEF_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_STRING_H], [/* Define to 1 if you have the <string.h> header file. */
@%:@undef HAVE_STRING_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_SYS_SOCKET_H], [/* Define to 1 if you have the <sys/socket.h> header file. */
@%:@undef HAVE_SYS_SOCKET_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_SYSLOG_H], [/* Define to 1 if you have the <syslog.h> header file. */
@%:@undef HAVE_SYSLOG_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_UNISTD_H], [/* Define to 1 if you have the <unistd.h> header file. */
@%:@undef HAVE_UNISTD_H])
m4trace:configure.ac:257: -1- AH_OUTPUT([HAVE_ISO_LIMITS_ISO_H], [/* Define to 1 if you have the <iso/limits_iso.h> header file. */
@%:@undef HAVE_ISO_LIMITS_ISO_H])
m4trace:configure.ac:258: -1- AH_OUTPUT([HAVE_PCAP_BPF_H], [/* Define to 1 if you have the <pcap-bpf.h> header file. */
@%:@undef HAVE_PCAP_BPF_H])
m4trace:configure.ac:258: -1- AH_OUTPUT([HAVE_NET_BPF_H], [/* Define to 1 if you have the <net/bpf.h> header file. */
@%:@undef HAVE_NET_BPF_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_SYS_TYPES_H], [/* Define to 1 if you have the <sys/types.h> header file. */
@%:@undef HAVE_SYS_TYPES_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_NETINET_IN_H], [/* Define to 1 if you have the <netinet/in.h> header file. */
@%:@undef HAVE_NETINET_IN_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_ARPA_NAMESER_H], [/* Define to 1 if you have the <arpa/nameser.h> header file. */
@%:@undef HAVE_ARPA_NAMESER_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_ARPA_NAMESER_COMPAT_H], [/* Define to 1 if you have the <arpa/nameser_compat.h> header file. */
@%:@undef HAVE_ARPA_NAMESER_COMPAT_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_NETDB_H], [/* Define to 1 if you have the <netdb.h> header file. */
@%:@undef HAVE_NETDB_H])
m4trace:configure.ac:260: -1- AH_OUTPUT([HAVE_RESOLV_H], [/* Define to 1 if you have the <resolv.h> header file. */
@%:@undef HAVE_RESOLV_H])
m4trace:configure.ac:279: -1- AC_SUBST([FTS_OBJ])
m4trace:configure.ac:279: -1- AC_SUBST_TRACE([FTS_OBJ])
m4trace:configure.ac:279: -1- m4_pattern_allow([^FTS_OBJ$])
m4trace:configure.ac:283: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_VOID_P])
m4trace:configure.ac:283: -1- m4_pattern_allow([^SIZEOF_VOID_P$])
m4trace:configure.ac:283: -1- AH_OUTPUT([SIZEOF_VOID_P], [/* The size of `void *\', as computed by sizeof. */
@%:@undef SIZEOF_VOID_P])
m4trace:configure.ac:285: -1- AC_DEFINE_TRACE_LITERAL([const])
m4trace:configure.ac:285: -1- m4_pattern_allow([^const$])
m4trace:configure.ac:285: -1- AH_OUTPUT([const], [/* Define to empty if `const\' does not conform to ANSI C. */
@%:@undef const])
m4trace:configure.ac:286: -1- AH_OUTPUT([inline], [/* Define to `__inline__\' or `__inline\' if that\'s what the C compiler
calls it, or to nothing if \'inline\' is not supported under any name. */
#ifndef __cplusplus
#undef inline
#endif])
m4trace:configure.ac:287: -1- AH_OUTPUT([WORDS_BIGENDIAN], [/* Define WORDS_BIGENDIAN to 1 if your processor stores words with the most
significant byte first (like Motorola and SPARC, unlike Intel). */
#if defined AC_APPLE_UNIVERSAL_BUILD
# if defined __BIG_ENDIAN__
# define WORDS_BIGENDIAN 1
# endif
#else
# ifndef WORDS_BIGENDIAN
# undef WORDS_BIGENDIAN
# endif
#endif])
m4trace:configure.ac:287: -1- AC_DEFINE_TRACE_LITERAL([WORDS_BIGENDIAN])
m4trace:configure.ac:287: -1- m4_pattern_allow([^WORDS_BIGENDIAN$])
m4trace:configure.ac:287: -1- AC_DEFINE_TRACE_LITERAL([AC_APPLE_UNIVERSAL_BUILD])
m4trace:configure.ac:287: -1- m4_pattern_allow([^AC_APPLE_UNIVERSAL_BUILD$])
m4trace:configure.ac:287: -1- AH_OUTPUT([AC_APPLE_UNIVERSAL_BUILD], [/* Define if building universal (internal helper macro) */
@%:@undef AC_APPLE_UNIVERSAL_BUILD])
m4trace:configure.ac:288: -1- AC_DEFINE_TRACE_LITERAL([pid_t])
m4trace:configure.ac:288: -1- m4_pattern_allow([^pid_t$])
m4trace:configure.ac:288: -1- AH_OUTPUT([pid_t], [/* Define to `int\' if <sys/types.h> does not define. */
@%:@undef pid_t])
m4trace:configure.ac:289: -1- AC_DEFINE_TRACE_LITERAL([size_t])
m4trace:configure.ac:289: -1- m4_pattern_allow([^size_t$])
m4trace:configure.ac:289: -1- AH_OUTPUT([size_t], [/* Define to `unsigned int\' if <sys/types.h> does not define. */
@%:@undef size_t])
m4trace:configure.ac:290: -1- AC_DEFINE_TRACE_LITERAL([TM_IN_SYS_TIME])
m4trace:configure.ac:290: -1- m4_pattern_allow([^TM_IN_SYS_TIME$])
m4trace:configure.ac:290: -1- AH_OUTPUT([TM_IN_SYS_TIME], [/* Define to 1 if your <sys/time.h> declares `struct tm\'. */
@%:@undef TM_IN_SYS_TIME])
m4trace:configure.ac:291: -1- AC_DEFINE_TRACE_LITERAL([HAVE__BOOL])
m4trace:configure.ac:291: -1- m4_pattern_allow([^HAVE__BOOL$])
m4trace:configure.ac:291: -1- AH_OUTPUT([HAVE__BOOL], [/* Define to 1 if the system has the type `_Bool\'. */
@%:@undef HAVE__BOOL])
m4trace:configure.ac:291: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STDBOOL_H])
m4trace:configure.ac:291: -1- m4_pattern_allow([^HAVE_STDBOOL_H$])
m4trace:configure.ac:291: -1- AH_OUTPUT([HAVE_STDBOOL_H], [/* Define to 1 if stdbool.h conforms to C99. */
@%:@undef HAVE_STDBOOL_H])
m4trace:configure.ac:294: -1- AC_DEFINE_TRACE_LITERAL([CLOSEDIR_VOID])
m4trace:configure.ac:294: -1- m4_pattern_allow([^CLOSEDIR_VOID$])
m4trace:configure.ac:294: -1- AH_OUTPUT([CLOSEDIR_VOID], [/* Define to 1 if the `closedir\' function returns void instead of `int\'. */
@%:@undef CLOSEDIR_VOID])
m4trace:configure.ac:295: -1- AH_OUTPUT([HAVE_VFORK_H], [/* Define to 1 if you have the <vfork.h> header file. */
@%:@undef HAVE_VFORK_H])
m4trace:configure.ac:295: -1- AC_DEFINE_TRACE_LITERAL([HAVE_VFORK_H])
m4trace:configure.ac:295: -1- m4_pattern_allow([^HAVE_VFORK_H$])
m4trace:configure.ac:295: -1- AH_OUTPUT([HAVE_FORK], [/* Define to 1 if you have the `fork\' function. */
@%:@undef HAVE_FORK])
m4trace:configure.ac:295: -1- AH_OUTPUT([HAVE_VFORK], [/* Define to 1 if you have the `vfork\' function. */
@%:@undef HAVE_VFORK])
m4trace:configure.ac:295: -1- AC_DEFINE_TRACE_LITERAL([HAVE_WORKING_VFORK])
m4trace:configure.ac:295: -1- m4_pattern_allow([^HAVE_WORKING_VFORK$])
m4trace:configure.ac:295: -1- AH_OUTPUT([HAVE_WORKING_VFORK], [/* Define to 1 if `vfork\' works. */
@%:@undef HAVE_WORKING_VFORK])
m4trace:configure.ac:295: -1- AC_DEFINE_TRACE_LITERAL([vfork])
m4trace:configure.ac:295: -1- m4_pattern_allow([^vfork$])
m4trace:configure.ac:295: -1- AH_OUTPUT([vfork], [/* Define as `fork\' if `vfork\' does not work. */
@%:@undef vfork])
m4trace:configure.ac:295: -1- AC_DEFINE_TRACE_LITERAL([HAVE_WORKING_FORK])
m4trace:configure.ac:295: -1- m4_pattern_allow([^HAVE_WORKING_FORK$])
m4trace:configure.ac:295: -1- AH_OUTPUT([HAVE_WORKING_FORK], [/* Define to 1 if `fork\' works. */
@%:@undef HAVE_WORKING_FORK])
m4trace:configure.ac:296: -1- AH_OUTPUT([HAVE_STDLIB_H], [/* Define to 1 if you have the <stdlib.h> header file. */
@%:@undef HAVE_STDLIB_H])
m4trace:configure.ac:296: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STDLIB_H])
m4trace:configure.ac:296: -1- m4_pattern_allow([^HAVE_STDLIB_H$])
m4trace:configure.ac:296: -1- AC_DEFINE_TRACE_LITERAL([HAVE_MALLOC])
m4trace:configure.ac:296: -1- m4_pattern_allow([^HAVE_MALLOC$])
m4trace:configure.ac:296: -1- AH_OUTPUT([HAVE_MALLOC], [/* Define to 1 if your system has a GNU libc compatible `malloc\' function, and
to 0 otherwise. */
@%:@undef HAVE_MALLOC])
m4trace:configure.ac:296: -1- AC_DEFINE_TRACE_LITERAL([HAVE_MALLOC])
m4trace:configure.ac:296: -1- m4_pattern_allow([^HAVE_MALLOC$])
m4trace:configure.ac:296: -1- AC_SUBST([LIB@&t@OBJS], ["$LIB@&t@OBJS malloc.$ac_objext"])
m4trace:configure.ac:296: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:296: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:296: -1- AC_LIBSOURCE([malloc.c])
m4trace:configure.ac:296: -1- AC_DEFINE_TRACE_LITERAL([malloc])
m4trace:configure.ac:296: -1- m4_pattern_allow([^malloc$])
m4trace:configure.ac:296: -1- AH_OUTPUT([malloc], [/* Define to rpl_malloc if the replacement function should be used. */
@%:@undef malloc])
m4trace:configure.ac:297: -1- AC_DEFINE_TRACE_LITERAL([TIME_WITH_SYS_TIME])
m4trace:configure.ac:297: -1- m4_pattern_allow([^TIME_WITH_SYS_TIME$])
m4trace:configure.ac:297: -1- AH_OUTPUT([TIME_WITH_SYS_TIME], [/* Define to 1 if you can safely include both <sys/time.h> and <time.h>. */
@%:@undef TIME_WITH_SYS_TIME])
m4trace:configure.ac:297: -1- AH_OUTPUT([HAVE_SYS_TIME_H], [/* Define to 1 if you have the <sys/time.h> header file. */
@%:@undef HAVE_SYS_TIME_H])
m4trace:configure.ac:297: -1- AH_OUTPUT([HAVE_UNISTD_H], [/* Define to 1 if you have the <unistd.h> header file. */
@%:@undef HAVE_UNISTD_H])
m4trace:configure.ac:297: -1- AH_OUTPUT([HAVE_ALARM], [/* Define to 1 if you have the `alarm\' function. */
@%:@undef HAVE_ALARM])
m4trace:configure.ac:297: -1- AC_SUBST([LIB@&t@OBJS], ["$LIB@&t@OBJS mktime.$ac_objext"])
m4trace:configure.ac:297: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:297: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:297: -1- AC_LIBSOURCE([mktime.c])
m4trace:configure.ac:298: -1- AH_OUTPUT([HAVE_STDLIB_H], [/* Define to 1 if you have the <stdlib.h> header file. */
@%:@undef HAVE_STDLIB_H])
m4trace:configure.ac:298: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STDLIB_H])
m4trace:configure.ac:298: -1- m4_pattern_allow([^HAVE_STDLIB_H$])
m4trace:configure.ac:298: -1- AC_DEFINE_TRACE_LITERAL([HAVE_REALLOC])
m4trace:configure.ac:298: -1- m4_pattern_allow([^HAVE_REALLOC$])
m4trace:configure.ac:298: -1- AH_OUTPUT([HAVE_REALLOC], [/* Define to 1 if your system has a GNU libc compatible `realloc\' function,
and to 0 otherwise. */
@%:@undef HAVE_REALLOC])
m4trace:configure.ac:298: -1- AC_DEFINE_TRACE_LITERAL([HAVE_REALLOC])
m4trace:configure.ac:298: -1- m4_pattern_allow([^HAVE_REALLOC$])
m4trace:configure.ac:298: -1- AC_SUBST([LIB@&t@OBJS], ["$LIB@&t@OBJS realloc.$ac_objext"])
m4trace:configure.ac:298: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:298: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:298: -1- AC_LIBSOURCE([realloc.c])
m4trace:configure.ac:298: -1- AC_DEFINE_TRACE_LITERAL([realloc])
m4trace:configure.ac:298: -1- m4_pattern_allow([^realloc$])
m4trace:configure.ac:298: -1- AH_OUTPUT([realloc], [/* Define to rpl_realloc if the replacement function should be used. */
@%:@undef realloc])
m4trace:configure.ac:299: -1- AC_DEFINE_TRACE_LITERAL([LSTAT_FOLLOWS_SLASHED_SYMLINK])
m4trace:configure.ac:299: -1- m4_pattern_allow([^LSTAT_FOLLOWS_SLASHED_SYMLINK$])
m4trace:configure.ac:299: -1- AH_OUTPUT([LSTAT_FOLLOWS_SLASHED_SYMLINK], [/* Define to 1 if `lstat\' dereferences a symlink specified with a trailing
slash. */
@%:@undef LSTAT_FOLLOWS_SLASHED_SYMLINK])
m4trace:configure.ac:299: -1- AC_SUBST([LIB@&t@OBJS], ["$LIB@&t@OBJS lstat.$ac_objext"])
m4trace:configure.ac:299: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:299: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:299: -1- AC_LIBSOURCE([lstat.c])
m4trace:configure.ac:299: -1- AC_SUBST([LIB@&t@OBJS], ["$LIB@&t@OBJS stat.$ac_objext"])
m4trace:configure.ac:299: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:299: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:299: -1- AC_LIBSOURCE([stat.c])
m4trace:configure.ac:299: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STAT_EMPTY_STRING_BUG])
m4trace:configure.ac:299: -1- m4_pattern_allow([^HAVE_STAT_EMPTY_STRING_BUG$])
m4trace:configure.ac:299: -1- AH_OUTPUT([HAVE_STAT_EMPTY_STRING_BUG], [/* Define to 1 if `stat\' has the bug that it succeeds when given the
zero-length file name argument. */
@%:@undef HAVE_STAT_EMPTY_STRING_BUG])
m4trace:configure.ac:300: -1- AH_OUTPUT([HAVE_STRFTIME], [/* Define to 1 if you have the `strftime\' function. */
@%:@undef HAVE_STRFTIME])
m4trace:configure.ac:300: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRFTIME])
m4trace:configure.ac:300: -1- m4_pattern_allow([^HAVE_STRFTIME$])
m4trace:configure.ac:300: -1- AC_DEFINE_TRACE_LITERAL([HAVE_STRFTIME])
m4trace:configure.ac:300: -1- m4_pattern_allow([^HAVE_STRFTIME$])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_INET_NTOA], [/* Define to 1 if you have the `inet_ntoa\' function. */
@%:@undef HAVE_INET_NTOA])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_SOCKET], [/* Define to 1 if you have the `socket\' function. */
@%:@undef HAVE_SOCKET])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_STRCHR], [/* Define to 1 if you have the `strchr\' function. */
@%:@undef HAVE_STRCHR])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_STRDUP], [/* Define to 1 if you have the `strdup\' function. */
@%:@undef HAVE_STRDUP])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_STRERROR], [/* Define to 1 if you have the `strerror\' function. */
@%:@undef HAVE_STRERROR])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_STRRCHR], [/* Define to 1 if you have the `strrchr\' function. */
@%:@undef HAVE_STRRCHR])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_STRSTR], [/* Define to 1 if you have the `strstr\' function. */
@%:@undef HAVE_STRSTR])
m4trace:configure.ac:301: -1- AH_OUTPUT([HAVE_SCANDIR], [/* Define to 1 if you have the `scandir\' function. */
@%:@undef HAVE_SCANDIR])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBRESOLV], [/* Define to 1 if you have the `resolv\' library (-lresolv). */
@%:@undef HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBRESOLV$])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBRESOLV], [/* Define to 1 if you have the `resolv\' library (-lresolv). */
@%:@undef HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBRESOLV$])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBSOCKET])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBSOCKET$])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBSOCKET], [/* */
@%:@undef HAVE_LIBSOCKET])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBRESOLV$])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBRESOLV], [/* */
@%:@undef HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBRESOLV], [/* Define to 1 if you have the `resolv\' library (-lresolv). */
@%:@undef HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBRESOLV$])
m4trace:configure.ac:306: -1- AH_OUTPUT([HAVE_LIBRESOLV], [/* Define to 1 if you have the `resolv\' library (-lresolv). */
@%:@undef HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- AC_DEFINE_TRACE_LITERAL([HAVE_LIBRESOLV])
m4trace:configure.ac:306: -1- m4_pattern_allow([^HAVE_LIBRESOLV$])
m4trace:configure.ac:321: -1- AC_DEFINE_TRACE_LITERAL([ptrdiff_t])
m4trace:configure.ac:321: -1- m4_pattern_allow([^ptrdiff_t$])
m4trace:configure.ac:321: -1- AH_OUTPUT([ptrdiff_t], [/* Define to `long\' if <sys/types.h> does not define. */
@%:@undef ptrdiff_t])
m4trace:configure.ac:322: -1- AC_DEFINE_TRACE_LITERAL([size_t])
m4trace:configure.ac:322: -1- m4_pattern_allow([^size_t$])
m4trace:configure.ac:322: -1- AH_OUTPUT([size_t], [/* Define to `unsigned int\' if <sys/types.h> does not define. */
@%:@undef size_t])
m4trace:configure.ac:323: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_SHORT])
m4trace:configure.ac:323: -1- m4_pattern_allow([^SIZEOF_SHORT$])
m4trace:configure.ac:323: -1- AH_OUTPUT([SIZEOF_SHORT], [/* The size of `short\', as computed by sizeof. */
@%:@undef SIZEOF_SHORT])
m4trace:configure.ac:324: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_INT])
m4trace:configure.ac:324: -1- m4_pattern_allow([^SIZEOF_INT$])
m4trace:configure.ac:324: -1- AH_OUTPUT([SIZEOF_INT], [/* The size of `int\', as computed by sizeof. */
@%:@undef SIZEOF_INT])
m4trace:configure.ac:325: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_LONG])
m4trace:configure.ac:325: -1- m4_pattern_allow([^SIZEOF_LONG$])
m4trace:configure.ac:325: -1- AH_OUTPUT([SIZEOF_LONG], [/* The size of `long\', as computed by sizeof. */
@%:@undef SIZEOF_LONG])
m4trace:configure.ac:326: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_LONG_LONG])
m4trace:configure.ac:326: -1- m4_pattern_allow([^SIZEOF_LONG_LONG$])
m4trace:configure.ac:326: -1- AH_OUTPUT([SIZEOF_LONG_LONG], [/* The size of `long long\', as computed by sizeof. */
@%:@undef SIZEOF_LONG_LONG])
m4trace:configure.ac:327: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF___INT64])
m4trace:configure.ac:327: -1- m4_pattern_allow([^SIZEOF___INT64$])
m4trace:configure.ac:327: -1- AH_OUTPUT([SIZEOF___INT64], [/* The size of `__int64\', as computed by sizeof. */
@%:@undef SIZEOF___INT64])
m4trace:configure.ac:328: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_VOID_P])
m4trace:configure.ac:328: -1- m4_pattern_allow([^SIZEOF_VOID_P$])
m4trace:configure.ac:328: -1- AH_OUTPUT([SIZEOF_VOID_P], [/* The size of `void *\', as computed by sizeof. */
@%:@undef SIZEOF_VOID_P])
m4trace:configure.ac:329: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_SIZE_T])
m4trace:configure.ac:329: -1- m4_pattern_allow([^SIZEOF_SIZE_T$])
m4trace:configure.ac:329: -1- AH_OUTPUT([SIZEOF_SIZE_T], [/* The size of `size_t\', as computed by sizeof. */
@%:@undef SIZEOF_SIZE_T])
m4trace:configure.ac:330: -1- AC_DEFINE_TRACE_LITERAL([SIZEOF_PTRDIFF_T])
m4trace:configure.ac:330: -1- m4_pattern_allow([^SIZEOF_PTRDIFF_T$])
m4trace:configure.ac:330: -1- AH_OUTPUT([SIZEOF_PTRDIFF_T], [/* The size of `ptrdiff_t\', as computed by sizeof. */
@%:@undef SIZEOF_PTRDIFF_T])
m4trace:configure.ac:331: -1- AC_DEFINE_TRACE_LITERAL([const])
m4trace:configure.ac:331: -1- m4_pattern_allow([^const$])
m4trace:configure.ac:331: -1- AH_OUTPUT([const], [/* Define to empty if `const\' does not conform to ANSI C. */
@%:@undef const])
m4trace:configure.ac:332: -1- AH_OUTPUT([HAVE_MEMCMP], [/* Define to 1 if you have the `memcmp\' function. */
@%:@undef HAVE_MEMCMP])
m4trace:configure.ac:332: -1- AH_OUTPUT([HAVE_MEMCPY], [/* Define to 1 if you have the `memcpy\' function. */
@%:@undef HAVE_MEMCPY])
m4trace:configure.ac:332: -1- AH_OUTPUT([HAVE_MEMMOVE], [/* Define to 1 if you have the `memmove\' function. */
@%:@undef HAVE_MEMMOVE])
m4trace:configure.ac:332: -1- AH_OUTPUT([HAVE_MEMSET], [/* Define to 1 if you have the `memset\' function. */
@%:@undef HAVE_MEMSET])
m4trace:configure.ac:335: -1- _m4_warn([obsolete], [The macro `AC_TRY_COMPILE' is obsolete.
You should run autoupdate.], [../../lib/autoconf/general.m4:2614: AC_TRY_COMPILE is expanded from...
../../lib/m4sugar/m4sh.m4:639: AS_IF is expanded from...
../../lib/autoconf/general.m4:2031: AC_CACHE_VAL is expanded from...
configure.ac:335: the top level])
m4trace:configure.ac:353: -1- AC_DEFINE_TRACE_LITERAL([HAVE_SEMUN])
m4trace:configure.ac:353: -1- m4_pattern_allow([^HAVE_SEMUN$])
m4trace:configure.ac:353: -1- AH_OUTPUT([HAVE_SEMUN], [/* Define if sys/sem.h defines struct semun */
@%:@undef HAVE_SEMUN])
m4trace:configure.ac:357: -1- _m4_warn([obsolete], [The macro `AC_TRY_RUN' is obsolete.
You should run autoupdate.], [../../lib/autoconf/general.m4:2764: AC_TRY_RUN is expanded from...
configure.ac:357: the top level])
m4trace:configure.ac:357: -1- AC_DEFINE_TRACE_LITERAL([HAVE_SIZE_T_Z_FORMAT])
m4trace:configure.ac:357: -1- m4_pattern_allow([^HAVE_SIZE_T_Z_FORMAT$])
m4trace:configure.ac:357: -1- AH_OUTPUT([HAVE_SIZE_T_Z_FORMAT], [/* Define to 1 if you have a printf() that supports the %z format string. */
@%:@undef HAVE_SIZE_T_Z_FORMAT])
m4trace:configure.ac:386: -1- AC_CONFIG_FILES([Makefile bin/Makefile man/Makefile])
m4trace:configure.ac:386: -1- _m4_warn([obsolete], [AC_OUTPUT should be used without arguments.
You should run autoupdate.], [])
m4trace:configure.ac:386: -1- AC_SUBST([LIB@&t@OBJS], [$ac_libobjs])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([LIB@&t@OBJS])
m4trace:configure.ac:386: -1- m4_pattern_allow([^LIB@&t@OBJS$])
m4trace:configure.ac:386: -1- AC_SUBST([LTLIBOBJS], [$ac_ltlibobjs])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([LTLIBOBJS])
m4trace:configure.ac:386: -1- m4_pattern_allow([^LTLIBOBJS$])
m4trace:configure.ac:386: -1- AM_CONDITIONAL([am__EXEEXT], [test -n "$EXEEXT"])
m4trace:configure.ac:386: -1- AC_SUBST([am__EXEEXT_TRUE])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([am__EXEEXT_TRUE])
m4trace:configure.ac:386: -1- m4_pattern_allow([^am__EXEEXT_TRUE$])
m4trace:configure.ac:386: -1- AC_SUBST([am__EXEEXT_FALSE])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([am__EXEEXT_FALSE])
m4trace:configure.ac:386: -1- m4_pattern_allow([^am__EXEEXT_FALSE$])
m4trace:configure.ac:386: -1- _AM_SUBST_NOTMAKE([am__EXEEXT_TRUE])
m4trace:configure.ac:386: -1- _AM_SUBST_NOTMAKE([am__EXEEXT_FALSE])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([top_builddir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([top_build_prefix])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([srcdir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([abs_srcdir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([top_srcdir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([abs_top_srcdir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([builddir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([abs_builddir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([abs_top_builddir])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([INSTALL])
m4trace:configure.ac:386: -1- AC_SUBST_TRACE([MKDIR_P])

11
bin/AddExtension.txt Executable file
View File

@ -0,0 +1,11 @@
Add extensions:
===============
1. Add appropriate definitions for new extension in nffile.h
2. Update extension_descriptor array in nfx.c to reflect the new extension.
Increase user index ( 3rd value )
3. Add filter in grammar.y
4. test filter with nftest.
5. nffile_inline.c
6. nf_common.c define printing functions
7. nfsta.c add stat definitions

130
bin/Makefile.am Executable file
View File

@ -0,0 +1,130 @@
bin_PROGRAMS = nfcapd nfdump nfreplay nfexpire nfanon
EXTRA_PROGRAMS = nftest nfgen nfreader
check_PROGRAMMS = test.sh
TESTS = nftest test.sh
if SFLOW
bin_PROGRAMS += sfcapd
endif
if NFPROFILE
bin_PROGRAMS += nfprofile
endif
if NFTRACK
bin_PROGRAMS += nftrack
endif
if FT2NFDUMP
bin_PROGRAMS += ft2nfdump
endif
BUILT_SOURCES = grammar.h
AM_YFLAGS = -d
AM_CPPFLAGS = $(DEPS_CFLAGS)
LDADD = $(DEPS_LIBS)
#Add extra debug info for gdb
AM_CFLAGS = -ggdb
common = nf_common.c nf_common.h
util = util.c util.h
filelzo = minilzo.c minilzo.h lzoconf.h lzodefs.h nffile.c nffile.h nfx.c nfx.h nfxstat.h nfxstat.c
nflist = flist.c flist.h fts_compat.c fts_compat.h
filter = grammar.y scanner.l nftree.c nftree.h ipconv.c ipconv.h rbtree.h
nfprof = nfprof.c nfprof.h
nfnet = nfnet.c nfnet.h
anon = panonymizer.c panonymizer.h rijndael.c rijndael.h
collector = collector.c collector.h
nfv1 = netflow_v1.c netflow_v1.h
nfv9 = netflow_v9.c netflow_v9.h
pcaproc = pcaproc.c pcaproc.h flowtree.c flowtree.h ipfrag.c ipfrag.h
content = content_dns.c content_dns.h
netflow_pcap = netflow_pcap.c netflow_pcap.h
ipfix = ipfix.c ipfix.h
nfv5v7 = netflow_v5_v7.c netflow_v5_v7.h
nfstatfile = nfstatfile.c nfstatfile.h
nflowcache = nflowcache.c nflowcache.h
bookkeeper = bookkeeper.c bookkeeper.h
exporter = exporter.c exporter.h
expire= expire.c expire.h
launch = launch.c launch.h
nfdump_SOURCES = nfdump.c nfdump.h nfstat.c nfstat.h nfexport.c nfexport.h \
$(common) $(nflowcache) $(util) $(filelzo) $(nflist) $(filter) $(nfprof) $(exporter)
nfreplay_SOURCES = nfreplay.c \
$(common) $(util) $(filelzo) $(nflist) $(filter) $(nfprof) \
$(nfnet) $(collector) $(nfv1) $(nfv9) $(nfv5v7) $(ipfix) $(exporter)
nfprofile_SOURCES = nfprofile.c profile.c profile.h \
$(common) $(util) $(filelzo) $(nflist) $(filter) $(nfstatfile) $(exporter)
nfprofile_LDADD = -lrrd
nftrack_SOURCES = ../extra/nftrack/nftrack.c \
../extra/nftrack/nftrack_rrd.c ../extra/nftrack/nftrack_rrd.h \
../extra/nftrack/nftrack_stat.c ../extra/nftrack/nftrack_stat.h \
$(common) $(util) $(filelzo) $(nflist) $(filter)
nftrack_CFLAGS = -I ../extra/nftrack
nftrack_LDADD = -lrrd
nfcapd_SOURCES = nfcapd.c \
$(common) $(util) $(filelzo) $(nflist) $(nfstatfile) $(launch) \
$(nfnet) $(collector) $(nfv1) $(nfv5v7) $(nfv9) $(ipfix) $(bookkeeper) $(expire)
nfpcapd_SOURCES = nfpcapd.c \
$(pcaproc) $(netflow_pcap) \
$(common) $(util) $(filelzo) $(nflist) $(nfstatfile) $(launch) \
$(nfnet) $(collector) $(bookkeeper) $(expire) $(content)
if READPCAP
nfcapd_CFLAGS = -DPCAP
nfcapd_SOURCES += pcap_reader.c pcap_reader.h
nfcapd_LDADD = -lpcap
endif
if BUILDNFPCAPD
bin_PROGRAMS += nfpcapd
nfpcapd_CFLAGS = -D_BSD_SOURCE
nfpcapd_LDADD = -lpcap
nfpcapd_LDFLAGS = -pthread
endif
sfcapd_SOURCES = sfcapd.c sflow.c sflow.h sflow_proto.h \
$(common) $(util) $(filelzo) $(nflist) $(nfstatfile) $(launch) \
$(nfnet) $(collector) $(bookkeeper) $(expire)
if READPCAP
sfcapd_CFLAGS = -DPCAP
sfcapd_SOURCES += pcap_reader.c pcap_reader.h
sfcapd_LDADD = -lpcap
endif
nfreader_SOURCES = nfreader.c \
$(util) $(filelzo) $(nflist) $(exporter)
nfanon_SOURCES = nfanon.c \
$(util) $(filelzo) $(nflist) $(anon)
nfgen_SOURCES = nfgen.c $(util) $(filelzo) $(nflist)
nfexpire_SOURCES = nfexpire.c \
$(bookkeeper) $(expire) $(util) $(nfstatfile)
nfexpire_LDADD = @FTS_OBJ@
nftest_SOURCES = nftest.c $(common) $(util) $(filter) $(filelzo)
nftest_DEPENDENCIES = nfgen
if FT2NFDUMP
ft2nfdump_SOURCES = ft2nfdump.c $(common) $(filelzo) $(util)
ft2nfdump_CFLAGS = @FT_INCLUDES@
ft2nfdump_LDADD = -lft -lz
ft2nfdump_LDADD += @FT_LDFLAGS@
endif
EXTRA_DIST = inline.c collector_inline.c nffile_inline.c nfdump_inline.c heapsort_inline.c applybits_inline.c test.sh nfdump.test.out parse_csv.pl
CLEANFILES = lex.yy.c grammar.c grammar.h scanner.c scanner.h

2549
bin/Makefile.in Normal file
View File

File diff suppressed because it is too large Load Diff

92
bin/applybits_inline.c Executable file
View File

@ -0,0 +1,92 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: applybits_inline.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
static inline void ApplyNetMaskBits(master_record_t *flow_record, int apply_netbits);
static inline void ApplyNetMaskBits(master_record_t *flow_record, int apply_netbits) {
if ( (flow_record->flags & FLAG_IPV6_ADDR ) != 0 ) { // IPv6
if ( apply_netbits & 1 ) {
uint64_t mask;
uint32_t mask_bits = flow_record->src_mask;
if ( mask_bits > 64 ) {
mask = 0xffffffffffffffffLL << ( 128 - mask_bits );
flow_record->v6.srcaddr[1] &= mask;
} else {
mask = 0xffffffffffffffffLL << ( 64 - mask_bits );
flow_record->v6.srcaddr[0] &= mask;
flow_record->v6.srcaddr[1] = 0;
}
}
if ( apply_netbits & 2 ) {
uint64_t mask;
uint32_t mask_bits = flow_record->dst_mask;
if ( mask_bits > 64 ) {
mask = 0xffffffffffffffffLL << ( 128 - mask_bits );
flow_record->v6.dstaddr[1] &= mask;
} else {
mask = 0xffffffffffffffffLL << ( 64 - mask_bits );
flow_record->v6.dstaddr[0] &= mask;
flow_record->v6.dstaddr[1] = 0;
}
}
} else { // IPv4
if ( apply_netbits & 1 ) {
uint32_t srcmask = 0xffffffff << ( 32 - flow_record->src_mask );
flow_record->v4.srcaddr &= srcmask;
}
if ( apply_netbits & 2 ) {
uint32_t dstmask = 0xffffffff << ( 32 - flow_record->dst_mask );
flow_record->v4.dstaddr &= dstmask;
}
}
} // End of ApplyNetMaskBits
static inline void ApplyAggrMask(master_record_t *record, master_record_t *mask) {
uint64_t *r = (uint64_t *)record;
uint64_t *m = (uint64_t *)mask;
int i, max_offset;
max_offset = offsetof(master_record_t, map_ref) >> 3;
for (i=2; i<max_offset; i++) {
r[i] &= m[i];
}
} // End of ApplyAggrMask

619
bin/bookkeeper.c Normal file
View File

@ -0,0 +1,619 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: bookkeeper.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/sem.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <time.h>
#include <signal.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifndef HAVE_SEMUN
union semun {
int val; // value for SETVAL
struct semid_ds *buf; // buffer for IPC_STAT & IPC_SET
u_short *array; // array for GETALL & SETALL
};
#endif
#include "config.h"
#include "bookkeeper.h"
static bookkeeper_list_t *bookkeeper_list = NULL;
/* function prototypes */
/*
* bookkeeper.c is needed for daemon code as well as normal stdio code
* therefore a generic LogError is defined, which maps to the
* approriate logging channel - either stderr or syslog
*/
void LogError(char *format, ...);
static key_t _ftok(const char *path, int id);
static void sem_lock(int sem_set_id);
static void sem_unlock(int sem_set_id);
static inline bookkeeper_list_t *Get_bookkeeper_list_entry(bookkeeper_t *bookkeeper);
/* Create shared memory object and set its size */
/* our own ftok implementation - the standard C library ftok is not reliable enough */
static key_t _ftok(const char *path, int id) {
struct stat st;
if (stat(path, &st) < 0)
return (key_t)-1;
return (key_t) ( ((st.st_dev & 0xffff) << 16) ^ st.st_ino ) + id;
}
// locks the semaphore, for exclusive access to the bookkeeping record
static void sem_lock(int sem_set_id) {
struct sembuf sem_op;
/* wait on the semaphore, unless it's value is non-negative. */
sem_op.sem_num = 0;
sem_op.sem_op = -1;
sem_op.sem_flg = 0;
if ( semop(sem_set_id, &sem_op, 1) == 0 )
return;
LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
} // End of sem_lock
// sem_unlock. un-locks the semaphore.
static void sem_unlock(int sem_set_id) {
struct sembuf sem_op;
/* signal the semaphore - increase its value by one. */
sem_op.sem_num = 0;
sem_op.sem_op = 1;
sem_op.sem_flg = 0;
if ( semop(sem_set_id, &sem_op, 1) == 0 )
return;
LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
} // End of sem_unlock
static inline bookkeeper_list_t *Get_bookkeeper_list_entry(bookkeeper_t *bookkeeper) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( bookkeeper == NULL )
return NULL;
bookkeeper_list_entry = bookkeeper_list;
while ( bookkeeper_list_entry != NULL && bookkeeper_list_entry->bookkeeper != bookkeeper )
bookkeeper_list_entry = bookkeeper_list_entry->next;
return bookkeeper_list_entry;
} // End of Get_bookkeeper_list_entry
int InitBookkeeper(bookkeeper_t **bookkeeper, char *path, pid_t nfcapd_pid, pid_t launcher_pid) {
int sem_key, shm_key, shm_id, sem_id;
union semun sem_val;
bookkeeper_list_t **bookkeeper_list_entry;
*bookkeeper = NULL;
shm_key = _ftok(path, 1);
if ( shm_key == - 1 )
return ERR_PATHACCESS;
// check if the shared memory is already allocated
shm_id = shmget(shm_key, sizeof(bookkeeper_t), 0600);
if ( shm_id >= 0 ) {
// the segment already exists. Either a running process is active
// or an unclean shutdown happened
// map the segement and check the record
*bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0);
if ( *bookkeeper == (bookkeeper_t *)-1 ) {
LogError("shmat() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
if ( (*bookkeeper)->nfcapd_pid <= 0 ) {
// rubbish or invalid pid of nfcapd process.
// Assume unclean shutdown or something else. We clean up and take this record.
memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t));
} else {
// check if the process created this record is still alive
int ret = kill((*bookkeeper)->nfcapd_pid, 0);
if ( ret == - 1 ) {
switch (errno) {
case ESRCH:
// process does not exist, we can clean up this record and use it
memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t));
break;
case EPERM:
// A process exists, but we are not allowed to signal this process
LogError("Another collector with pid %i but different user ID is already running, and configured for '%s'",
(*bookkeeper)->nfcapd_pid, path);
return ERR_EXISTS;
break;
default:
// This should never happen, but catch it anyway
LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
} else {
// process exists;
LogError("Another collector with pid %i is already running, and configured for '%s'",
(*bookkeeper)->nfcapd_pid, path);
return ERR_EXISTS;
}
// if we pass this point, we have recycled an existing record
}
} else {
// no valid shared segment was found
switch (errno) {
case ENOENT:
// this is ok - no shared segemtn exists, we can create a new one below
break;
case EACCES:
// there is such a segment, but we are not allowed to get it
// Assume it's another nfcapd
LogError("Access denied to collector bookkeeping record.");
return ERR_EXISTS;
break;
default:
// This should never happen, but catch it anyway
LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
// we now create a new segement, this hould not fail now
shm_id = shmget(shm_key, sizeof(bookkeeper_t), IPC_CREAT | 0600);
if ( shm_id == - 1 ) {
// but did anyway - give up
LogError("shmget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
*bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0);
if ( (*bookkeeper) == (bookkeeper_t *)-1 ) {
LogError("shmget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
memset((void *)(*bookkeeper), 0, sizeof(bookkeeper_t));
}
// at this point we now have a valid record and can proceed
(*bookkeeper)->nfcapd_pid = nfcapd_pid;
(*bookkeeper)->launcher_pid = launcher_pid;
(*bookkeeper)->sequence++;
// create semaphore
sem_key = _ftok(path, 2);
// this should never fail, as we aleady got a key for the shared memory
if ( sem_key == - 1 ) {
// .. but catch it anyway .. and release shared memory. something is fishy
struct shmid_ds buf;
shmdt((void *)(*bookkeeper));
shmctl(shm_id, IPC_RMID, &buf);
return ERR_FAILED;
}
// get the semaphore
sem_id = semget(sem_key, 1, IPC_CREAT | 0600);
if ( sem_id == - 1 ) {
struct shmid_ds buf;
// this should not have failed
LogError("semget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
// release shared memory
shmdt((void *)(*bookkeeper));
shmctl(shm_id, IPC_RMID, &buf);
return ERR_FAILED;
}
// initialize the semaphore
sem_val.val = 1;
if ( semctl(sem_id, 0, SETVAL, sem_val) == -1) {
struct shmid_ds buf;
// this should not have failed
LogError("semctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
// release shared memory
shmdt((void *)(*bookkeeper));
shmctl(shm_id, IPC_RMID, &buf);
return ERR_FAILED;
}
bookkeeper_list_entry = &bookkeeper_list;
while ( *bookkeeper_list_entry != NULL )
bookkeeper_list_entry = &((*bookkeeper_list_entry)->next);
(*bookkeeper_list_entry) = (bookkeeper_list_t *)malloc(sizeof(bookkeeper_list_t));
if ( !*bookkeeper_list_entry ) {
struct shmid_ds buf;
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
shmdt((void *)(*bookkeeper));
shmctl(shm_id, IPC_RMID, &buf);
semctl( sem_id, 0, IPC_RMID );
return ERR_FAILED;
}
memset((void *)*bookkeeper_list_entry, 0, sizeof(bookkeeper_list_t));
(*bookkeeper_list_entry)->shm_id = shm_id;
(*bookkeeper_list_entry)->sem_id = sem_id;
(*bookkeeper_list_entry)->bookkeeper = *bookkeeper;
(*bookkeeper_list_entry)->next = NULL;
// we are done
return BOOKKEEPER_OK;
} // End of InitBookkeeper
int AccessBookkeeper(bookkeeper_t **bookkeeper, char *path) {
bookkeeper_list_t **bookkeeper_list_entry;
int sem_key, shm_key, shm_id, sem_id;
*bookkeeper = NULL;
shm_key = _ftok(path, 1);
if ( shm_key == - 1 )
return ERR_PATHACCESS;
// check if the shared memory is already allocated
shm_id = shmget(shm_key, sizeof(bookkeeper_t), 0600);
if ( shm_id < 0 ) {
// the segment does not exists. Check why
switch (errno) {
case ENOENT:
// no shared segemtn exists.
return ERR_NOTEXISTS;
break;
case EACCES:
// there is such a segment, but we are not allowed to get it
// Assume it's another nfcapd
LogError("Access denied to collector bookkeeping record.");
return ERR_FAILED;
break;
default:
// This should never happen, but catch it anyway
LogError("semop() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
// not reached
}
// at this point we now have a valid record and can proceed
// create semaphore
sem_key = _ftok(path, 2);
// this should never fail, as we aleady got a key for the shared memory
if ( sem_key == - 1 ) {
// .. but catch it anyway .. and release shared memory. something is fishy
return ERR_FAILED;
}
// get the semaphore
sem_id = semget(sem_key, 1, 0600);
if ( sem_id == - 1 ) {
// this should not have failed
LogError("semget() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
// map the shared segment
*bookkeeper = (bookkeeper_t *)shmat(shm_id, NULL, 0);
if ( *bookkeeper == (bookkeeper_t *)-1 ) {
LogError("shmat() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
bookkeeper_list_entry = &bookkeeper_list;
while ( *bookkeeper_list_entry != NULL && (*bookkeeper_list_entry)->bookkeeper != NULL )
bookkeeper_list_entry = &((*bookkeeper_list_entry)->next);
// allocate new slot, else use unused slot
if ( *bookkeeper_list_entry == NULL ) {
(*bookkeeper_list_entry) = (bookkeeper_list_t *)malloc(sizeof(bookkeeper_list_t));
if ( !*bookkeeper_list_entry ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return ERR_FAILED;
}
memset((void *)*bookkeeper_list_entry, 0, sizeof(bookkeeper_list_t));
}
(*bookkeeper_list_entry)->shm_id = shm_id;
(*bookkeeper_list_entry)->sem_id = sem_id;
(*bookkeeper_list_entry)->bookkeeper = *bookkeeper;
(*bookkeeper_list_entry)->next = NULL;
return BOOKKEEPER_OK;
} // End of AccessBookkeeper
void ReleaseBookkeeper(bookkeeper_t *bookkeeper, int destroy) {
bookkeeper_list_t *bookkeeper_list_entry;
struct shmid_ds buf;
if ( !bookkeeper )
return;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return;
}
// detach from my process addr space memory
if ( shmdt((void *)bookkeeper) == -1 ) {
// ups ..
LogError("shmdt() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
}
bookkeeper = NULL;
if ( destroy == 0 ) {
// Entry no longer valid
bookkeeper_list_entry->bookkeeper = NULL;
bookkeeper_list_entry->shm_id = 0;
bookkeeper_list_entry->sem_id = 0;
return;
}
// prevent other proceeses to access the share memory, while we are removing it
// try to clean up.
sem_lock(bookkeeper_list_entry->sem_id);
if ( shmctl(bookkeeper_list_entry->shm_id, IPC_RMID, &buf) ) {
// ups ..
LogError("shmctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
}
sem_unlock(bookkeeper_list_entry->sem_id);
if ( semctl( bookkeeper_list_entry->sem_id, 0, IPC_RMID ) == -1 ) {
// ups ..
LogError("semctl() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
}
// Entry no longer valid
bookkeeper_list_entry->bookkeeper = NULL;
bookkeeper_list_entry->shm_id = 0;
bookkeeper_list_entry->sem_id = 0;
} // End of ReleaseBookkeeper
int LookBooks(bookkeeper_t *bookkeeper) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( !bookkeeper )
return 0;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return 1;
}
sem_lock(bookkeeper_list_entry->sem_id);
return 0;
} // End of LookBooks
int UnlookBooks(bookkeeper_t *bookkeeper) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( !bookkeeper )
return 0;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return 1;
}
sem_unlock(bookkeeper_list_entry->sem_id);
return 0;
} // End of UnlookBooks
void ClearBooks(bookkeeper_t *bookkeeper, bookkeeper_t *tmp_books) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( !bookkeeper )
return;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return;
}
sem_lock(bookkeeper_list_entry->sem_id);
// backup copy
if ( tmp_books != NULL ) {
memcpy((void *)tmp_books, (void *)bookkeeper, sizeof(bookkeeper_t));
}
bookkeeper->first = 0;
bookkeeper->last = 0;
bookkeeper->numfiles = 0;
bookkeeper->filesize = 0;
bookkeeper->sequence++;
sem_unlock(bookkeeper_list_entry->sem_id);
} // End of ClearBooks
uint64_t BookSequence(bookkeeper_t *bookkeeper) {
bookkeeper_list_t *bookkeeper_list_entry;
uint64_t seq;
if ( !bookkeeper )
return 0;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return 0;
}
sem_lock(bookkeeper_list_entry->sem_id);
seq = bookkeeper->sequence;
sem_unlock(bookkeeper_list_entry->sem_id);
return seq;
} // End of BookSequence
void UpdateBooks(bookkeeper_t *bookkeeper, time_t when, uint64_t size) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( !bookkeeper )
return;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return;
}
sem_lock(bookkeeper_list_entry->sem_id);
if ( bookkeeper->first == 0 )
bookkeeper->first = when;
bookkeeper->last = when;
bookkeeper->numfiles++;
bookkeeper->filesize += size;
bookkeeper->sequence++;
sem_unlock(bookkeeper_list_entry->sem_id);
} // End of UpdateBooks
void UpdateBooksParam(bookkeeper_t *bookkeeper, time_t lifetime, uint64_t maxsize) {
bookkeeper_list_t *bookkeeper_list_entry;
if ( !bookkeeper )
return;
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return;
}
sem_lock(bookkeeper_list_entry->sem_id);
bookkeeper->max_lifetime = lifetime;
bookkeeper->max_filesize = maxsize;
bookkeeper->sequence++;
sem_unlock(bookkeeper_list_entry->sem_id);
} // End of UpdateBooksParam
void PrintBooks(bookkeeper_t *bookkeeper) {
bookkeeper_list_t *bookkeeper_list_entry;
struct tm *ts;
time_t t;
char string[32];
if ( !bookkeeper ) {
printf("No bookkeeper record available!\n");
return;
}
bookkeeper_list_entry = Get_bookkeeper_list_entry(bookkeeper);
if ( !bookkeeper_list_entry ) {
// this should never happen
LogError("Software error in %s line %d: %s", __FILE__, __LINE__, "Entry not found in list");
return;
}
sem_lock(bookkeeper_list_entry->sem_id);
printf("Collector process: %lu\n", (unsigned long)bookkeeper->nfcapd_pid);
if ( bookkeeper->launcher_pid )
printf("Launcher process: %lu\n", (unsigned long)bookkeeper->launcher_pid);
else
printf("Launcher process: <none>\n");
printf("Record sequence : %llu\n", (unsigned long long)bookkeeper->sequence);
t = bookkeeper->first;
ts = localtime(&t);
strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts);
string[31] = '\0';
printf("First : %s\n", bookkeeper->first ? string : "<not set>");
t = bookkeeper->last;
ts = localtime(&t);
strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts);
string[31] = '\0';
printf("Last : %s\n", bookkeeper->last ? string : "<not set>");
printf("Number of files : %llu\n", (unsigned long long)bookkeeper->numfiles);
printf("Total file size : %llu\n", (unsigned long long)bookkeeper->filesize);
printf("Max file size : %llu\n", (unsigned long long)bookkeeper->max_filesize);
printf("Max life time : %llu\n", (unsigned long long)bookkeeper->max_lifetime);
sem_unlock(bookkeeper_list_entry->sem_id);
} // End of PrintBooks

100
bin/bookkeeper.h Normal file
View File

@ -0,0 +1,100 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: bookkeeper.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
#ifndef _BOOKKEEPER_H
#define _BOOKKEEPER_H 1
enum { BOOKKEEPER_OK = 0, ERR_FAILED, ERR_NOTEXISTS, ERR_PATHACCESS, ERR_EXISTS };
#define DETACH_ONLY 0
#define DESTROY_BOOKKEEPER 1
typedef struct bookkeeper_s {
// collector infos
pid_t nfcapd_pid;
pid_t launcher_pid;
// track info
uint64_t sequence;
// file infos
time_t first;
time_t last;
uint64_t numfiles;
uint64_t filesize;
uint64_t max_filesize;
uint64_t max_lifetime;
} bookkeeper_t;
// All bookkeepers are put into a linked list, to have all the shm_id,sem_id
typedef struct bookkeeper_list_s {
struct bookkeeper_list_s *next;
bookkeeper_t *bookkeeper;
// shared parameters
int sem_id;
int shm_id;
} bookkeeper_list_t;
/* function prototypes */
int InitBookkeeper(bookkeeper_t **bookkeeper, char *path, pid_t nfcapd_pid, pid_t launcher_pid);
int AccessBookkeeper(bookkeeper_t **bookkeeper, char *path);
void ReleaseBookkeeper(bookkeeper_t *bookkeeper, int destroy);
void ClearBooks(bookkeeper_t *bookkeeper, bookkeeper_t *tmp_books);
int LookBooks(bookkeeper_t *bookkeeper);
int UnlookBooks(bookkeeper_t *bookkeeper);
uint64_t BookSequence(bookkeeper_t *bookkeeper);
void UpdateBooks(bookkeeper_t *bookkeeper, time_t when, uint64_t size);
void UpdateBooksParam(bookkeeper_t *bookkeeper, time_t lifetime, uint64_t maxsize);
void PrintBooks(bookkeeper_t *bookkeeper);
#endif //_BOOKKEEPER_H

676
bin/collector.c Executable file
View File

@ -0,0 +1,676 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: collector.c 69 2010-09-09 07:17:43Z haag $
*
* $LastChangedRevision: 69 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/socket.h>
#include <sys/param.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <syslog.h>
#include <stdarg.h>
#include <time.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#include "util.h"
#include "nf_common.h"
#include "nffile.h"
#include "nfxstat.h"
#include "bookkeeper.h"
#include "collector.h"
#include "nfx.h"
#include "nffile_inline.c"
/* globals */
uint32_t default_sampling = 1;
uint32_t overwrite_sampling = 0;
/* local variables */
static uint32_t exporter_sysid = 0;
static char *DynamicSourcesDir = NULL;
/* local prototypes */
static uint32_t AssignExporterID(void);
/* local functions */
static uint32_t AssignExporterID(void) {
if ( exporter_sysid >= 0xFFFF ) {
LogError("Too many exporters (id > 65535). Flow records collected but without reference to exporter");
return 0;
}
return ++exporter_sysid;
} // End of AssignExporterID
/* global functions */
int SetDynamicSourcesDir(FlowSource_t **FlowSource, char *dir) {
if ( *FlowSource )
return 0;
DynamicSourcesDir = dir;
return 1;
} // End of SetDynamicSourcesDir
int AddFlowSource(FlowSource_t **FlowSource, char *ident) {
FlowSource_t **source;
struct stat fstat;
char *p, *q, s[MAXPATHLEN];
int has_any_source = 0;
int ok;
if ( DynamicSourcesDir )
return 0;
source = FlowSource;
while ( *source ) {
has_any_source |= (*source)->any_source;
source = &((*source)->next);
}
if ( has_any_source ) {
fprintf(stderr, "Ambiguous idents not allowed\n");
return 0;
}
*source = (FlowSource_t *)calloc(1, sizeof(FlowSource_t));
if ( !*source ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
(*source)->next = NULL;
(*source)->bookkeeper = NULL;
(*source)->any_source = 0;
(*source)->exporter_data = NULL;
(*source)->xstat = NULL;
(*FlowSource)->exporter_count = 0;
// separate IP address from ident
if ( ( p = strchr(ident, ',')) == NULL ) {
fprintf(stderr, "Syntax error for netflow source definition. Expect -n ident,IP,path\n");
return 0;
}
*p++ = '\0';
// separate path from IP
if ( ( q = strchr(p, ',')) == NULL ) {
fprintf(stderr, "Syntax error for netflow source definition. Expect -n ident,IP,path\n");
return 0;
}
*q++ = '\0';
if ( strchr(p, ':') != NULL ) {
uint64_t _ip[2];
ok = inet_pton(PF_INET6, p, _ip);
(*source)->sa_family = PF_INET6;
(*source)->ip.v6[0] = ntohll(_ip[0]);
(*source)->ip.v6[1] = ntohll(_ip[1]);
} else {
uint32_t _ip;
ok = inet_pton(PF_INET, p, &_ip);
(*source)->sa_family = PF_INET;
(*source)->ip.v6[0] = 0;
(*source)->ip.v6[1] = 0;
(*source)->ip.v4 = ntohl(_ip);
}
switch (ok) {
case 0:
fprintf(stderr, "Unparsable IP address: %s\n", p);
return 0;
case 1:
// success
break;
case -1:
fprintf(stderr, "Error while parsing IP address: %s\n", strerror(errno));
return 0;
break;
}
// fill in ident
if ( strlen(ident) >= IDENTLEN ) {
fprintf(stderr, "Source identifier too long: %s\n", ident);
return 0;
}
if ( strchr(ident, ' ') ) {
fprintf(stderr,"Illegal characters in ident %s\n", ident);
exit(255);
}
strncpy((*source)->Ident, ident, IDENTLEN-1 );
(*source)->Ident[IDENTLEN-1] = '\0';
if ( strlen(q) >= MAXPATHLEN ) {
fprintf(stderr,"Path too long: %s\n", q);
exit(255);
}
// check for existing path
if ( stat(q, &fstat) ) {
fprintf(stderr, "stat() error %s: %s\n", q, strerror(errno));
return 0;
}
if ( !(fstat.st_mode & S_IFDIR) ) {
fprintf(stderr, "No such directory: %s\n", q);
return 0;
}
// remember path
(*source)->datadir = strdup(q);
if ( !(*source)->datadir ) {
fprintf(stderr, "strdup() error: %s\n", strerror(errno));
return 0;
}
// cache current collector file
if ( snprintf(s, MAXPATHLEN-1, "%s/%s.%lu", (*source)->datadir , NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) {
fprintf(stderr, "Path too long: %s\n", q);
return 0;
}
(*source)->current = strdup(s);
if ( !(*source)->current ) {
fprintf(stderr, "strdup() error: %s\n", strerror(errno));
return 0;
}
return 1;
} // End of AddFlowSource
int AddDefaultFlowSource(FlowSource_t **FlowSource, char *ident, char *path) {
struct stat fstat;
char s[MAXPATHLEN];
if ( DynamicSourcesDir )
return 0;
*FlowSource = (FlowSource_t *)calloc(1,sizeof(FlowSource_t));
if ( !*FlowSource ) {
fprintf(stderr, "calloc() allocation error: %s\n", strerror(errno));
return 0;
}
(*FlowSource)->next = NULL;
(*FlowSource)->bookkeeper = NULL;
(*FlowSource)->any_source = 1;
(*FlowSource)->exporter_data = NULL;
(*FlowSource)->xstat = NULL;
(*FlowSource)->exporter_count = 0;
// fill in ident
if ( strlen(ident) >= IDENTLEN ) {
fprintf(stderr, "Source identifier too long: %s\n", ident);
return 0;
}
if ( strchr(ident, ' ') ) {
fprintf(stderr,"Illegal characters in ident %s\n", ident);
return 0;
}
strncpy((*FlowSource)->Ident, ident, IDENTLEN-1 );
(*FlowSource)->Ident[IDENTLEN-1] = '\0';
if ( strlen(path) >= MAXPATHLEN ) {
fprintf(stderr,"Path too long: %s\n",path);
return 0;
}
// check for existing path
if ( stat(path, &fstat) ) {
fprintf(stderr, "stat() error %s: %s\n", path, strerror(errno));
return 0;
}
if ( !(fstat.st_mode & S_IFDIR) ) {
fprintf(stderr, "No such directory: %s\n", path);
return 0;
}
// remember path
(*FlowSource)->datadir = strdup(path);
if ( !(*FlowSource)->datadir ) {
fprintf(stderr, "strdup() error: %s\n", strerror(errno));
return 0;
}
// cache current collector file
if ( snprintf(s, MAXPATHLEN-1, "%s/%s.%lu", (*FlowSource)->datadir , NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) {
fprintf(stderr, "Path too long: %s\n", path);
return 0;
}
(*FlowSource)->current = strdup(s);
if ( !(*FlowSource)->current ) {
fprintf(stderr, "strdup() error: %s\n", strerror(errno));
return 0;
}
return 1;
} // End of AddDefaultFlowSource
FlowSource_t *AddDynamicSource(FlowSource_t **FlowSource, struct sockaddr_storage *ss) {
FlowSource_t **source;
void *ptr;
char *s, ident[100], path[MAXPATHLEN];
int err;
union {
struct sockaddr_storage *ss;
struct sockaddr *sa;
struct sockaddr_in *sa_in;
struct sockaddr_in6 *sa_in6;
} u;
u.ss = ss;
if ( !DynamicSourcesDir )
return NULL;
source = FlowSource;
while ( *source ) {
source = &((*source)->next);
}
*source = (FlowSource_t *)calloc(1, sizeof(FlowSource_t));
if ( !*source ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
(*source)->next = NULL;
(*source)->bookkeeper = NULL;
(*source)->any_source = 0;
(*source)->exporter_data = NULL;
(*source)->xstat = NULL;
(*FlowSource)->exporter_count = 0;
switch (ss->ss_family) {
case PF_INET: {
#ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN
if (ss->ss_len != sizeof(struct sockaddr_in) ) {
// malformed struct
LogError("Malformed IPv4 socket struct in '%s', line '%d'", __FILE__, __LINE__ );
free(*source);
*source = NULL;
return NULL;
}
#endif
(*source)->sa_family = PF_INET;
(*source)->ip.v6[0] = 0;
(*source)->ip.v6[1] = 0;
(*source)->ip.v4 = ntohl(u.sa_in->sin_addr.s_addr);
ptr = &u.sa_in->sin_addr;
} break;
case PF_INET6: {
uint64_t *ip_ptr = (uint64_t *)u.sa_in6->sin6_addr.s6_addr;
#ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN
if (ss->ss_len != sizeof(struct sockaddr_in6) ) {
// malformed struct
LogError("Malformed IPv6 socket struct in '%s', line '%d'", __FILE__, __LINE__ );
free(*source);
*source = NULL;
return NULL;
}
#endif
// ptr = &((struct sockaddr_in6 *)sa)->sin6_addr;
(*source)->sa_family = PF_INET6;
(*source)->ip.v6[0] = ntohll(ip_ptr[0]);
(*source)->ip.v6[1] = ntohll(ip_ptr[1]);
ptr = &u.sa_in6->sin6_addr;
} break;
default:
// keep compiler happy
(*source)->ip.v6[0] = 0;
(*source)->ip.v6[1] = 0;
ptr = NULL;
LogError("Unknown sa fanily: %d in '%s', line '%d'", ss->ss_family, __FILE__, __LINE__ );
free(*source);
*source = NULL;
return NULL;
}
if ( !ptr ) {
free(*source);
*source = NULL;
return NULL;
}
inet_ntop(ss->ss_family, ptr, ident, sizeof(ident));
ident[99] = '\0';
dbg_printf("Dynamic Flow Source IP: %s\n", ident);
if ( strchr(ident, ':') ) { // condense IPv6 addresses
condense_v6(ident);
}
s = ident;
while ( *s != '\0' ) {
if ( *s == '.' || *s == ':' )
*s = '-';
s++;
}
dbg_printf("Dynamic Flow Source ident: %s\n", ident);
strncpy((*source)->Ident, ident, IDENTLEN-1 );
(*source)->Ident[IDENTLEN-1] = '\0';
snprintf(path, MAXPATHLEN-1, "%s/%s", DynamicSourcesDir, ident);
path[MAXPATHLEN-1] = '\0';
err = mkdir(path, 0755);
if ( err != 0 && errno != EEXIST ) {
LogError("mkdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
free(*source);
*source = NULL;
return NULL;
}
(*source)->datadir = strdup(path);
if ( snprintf(path, MAXPATHLEN-1, "%s/%s.%lu", (*source)->datadir, NF_DUMPFILE, (unsigned long)getpid() ) >= (MAXPATHLEN-1)) {
fprintf(stderr, "Path too long: %s\n", path);
free(*source);
*source = NULL;
return NULL;
}
(*source)->current = strdup(path);
LogInfo("Dynamically add source ident: %s in directory: %s", ident, path);
return *source;
} // End of AddDynamicSource
int InitExtensionMapList(FlowSource_t *fs) {
fs->extension_map_list.maps = (extension_map_t **)calloc(BLOCK_SIZE, sizeof(extension_map_t *));
if ( !fs->extension_map_list.maps ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
fs->extension_map_list.max_maps = BLOCK_SIZE;
fs->extension_map_list.next_free = 0;
fs->extension_map_list.num_maps = 0;
return 1;
} // End of InitExtensionMapList
int ReInitExtensionMapList(FlowSource_t *fs) {
dbg_printf("Flush all extension maps!\n");
free(fs->extension_map_list.maps);
fs->extension_map_list.maps = NULL;
return InitExtensionMapList(fs);
} // End of ReInitExtensionMapList
int RemoveExtensionMap(FlowSource_t *fs, extension_map_t *map) {
int slot = map->map_id;
dbg_printf("Remove extension map ID: %d\n", slot);
if ( slot >= fs->extension_map_list.max_maps ) {
// XXX hmm .. is simply return correct ///
LogError("*** software error *** Try to remove extension map %d, while only %d slots are available\n", slot, fs->extension_map_list.max_maps);
return 0;
}
fs->extension_map_list.maps[slot] = NULL;
return 1;
} // End of RemoveExtensionMap
int AddExtensionMap(FlowSource_t *fs, extension_map_t *map) {
int next_slot = fs->extension_map_list.next_free;
dbg_printf("Add extension map\n");
// is it a new map, we have not yet in the list
if ( map->map_id == INIT_ID ) {
if ( next_slot >= fs->extension_map_list.max_maps ) {
// extend map list
dbg_printf("List full - extend extension list to %d slots\n", fs->extension_map_list.max_maps + BLOCK_SIZE);
extension_map_t **p = realloc((void *)fs->extension_map_list.maps,
(fs->extension_map_list.max_maps + BLOCK_SIZE ) * sizeof(extension_map_t *));
if ( !p ) {
LogError("realloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
fs->extension_map_list.maps = p;
fs->extension_map_list.max_maps += BLOCK_SIZE;
}
dbg_printf("Add map to slot %d\n", next_slot);
fs->extension_map_list.maps[next_slot] = map;
map->map_id = next_slot;
fs->extension_map_list.num_maps++;
if ( (next_slot + 1) == fs->extension_map_list.num_maps ) {
// sequencially filled slots
// next free is next slot
fs->extension_map_list.next_free++;
dbg_printf("Next slot is sequential: %d\n", fs->extension_map_list.next_free);
} else {
// fill gap in list - search for next free
int i;
dbg_printf("Search next slot ... \n");
for ( i = (next_slot + 1); i < fs->extension_map_list.max_maps; i++ ) {
if ( fs->extension_map_list.maps[i] == NULL ) {
// empty slot found
dbg_printf("Empty slot found at %d\n", i);
break;
}
}
// assign next_free - if none found up to max, the list will get extended
// in the next round
dbg_printf("Set next free to %d\n", i);
fs->extension_map_list.next_free = i;
}
}
AppendToBuffer(fs->nffile, (void *)map, map->size);
return 1;
} // End of AddExtensionMap
int FlushInfoExporter(FlowSource_t *fs, exporter_info_record_t *exporter) {
exporter->sysid = AssignExporterID();
fs->exporter_count++;
AppendToBuffer(fs->nffile, (void *)exporter, exporter->header.size);
#ifdef DEVEL
{
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
printf("Flush Exporter: ");
if ( exporter->sa_family == AF_INET ) {
uint32_t _ip = htonl(exporter->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %16s, version: %u, ID: %2u\n", exporter->sysid,
ipstr, exporter->version, exporter->id);
} else if ( exporter->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(exporter->ip.v6[0]);
_ip[1] = htonll(exporter->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %40s, version: %u, ID: %2u\n", exporter->sysid,
ipstr, exporter->version, exporter->id);
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
printf("**** Exporter IP version unknown ****\n");
}
}
#endif
return 1;
} // End of FlushInfoExporter
int FlushInfoSampler(FlowSource_t *fs, sampler_info_record_t *sampler) {
AppendToBuffer(fs->nffile, (void *)sampler, sampler->header.size);
#ifdef DEVEL
{
printf("Flush Sampler: ");
if ( sampler->id < 0 ) {
printf("Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n",
sampler->exporter_sysid, sampler->mode, sampler->interval);
} else {
printf("Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n",
sampler->exporter_sysid, sampler->id, sampler->mode, sampler->interval);
}
}
#endif
return 1;
} // End of FlushInfoSampler
void FlushStdRecords(FlowSource_t *fs) {
generic_exporter_t *e = fs->exporter_data;
int i;
while ( e ) {
generic_sampler_t *sampler = e->sampler;
AppendToBuffer(fs->nffile, (void *)&(e->info), e->info.header.size);
while ( sampler ) {
AppendToBuffer(fs->nffile, (void *)&(sampler->info), sampler->info.header.size);
sampler = sampler->next;
}
e = e->next;
}
for ( i=0; i<fs->extension_map_list.next_free; i++ ) {
extension_map_t *map = fs->extension_map_list.maps[i];
if ( map )
AppendToBuffer(fs->nffile, (void *)map, map->size);
}
} // End of FlushStdRecords
void FlushExporterStats(FlowSource_t *fs) {
generic_exporter_t *e = fs->exporter_data;
exporter_stats_record_t *exporter_stats;
uint32_t i, size;
// idle collector ..
if ( !fs->exporter_count )
return;
size = sizeof(exporter_stats_record_t) + (fs->exporter_count -1) * sizeof(struct exporter_stat_s);
exporter_stats = ( exporter_stats_record_t *)malloc(size);
if ( !exporter_stats ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
exporter_stats->header.type = ExporterStatRecordType;
exporter_stats->header.size = size;
exporter_stats->stat_count = fs->exporter_count;
#ifdef DEVEL
printf("Flush Exporter Stats: %u exporters, size: %u\n", fs->exporter_count, size);
#endif
i = 0;
while ( e ) {
// prevent memory corruption - just in case .. should not happen anyway
// continue loop for error reporting after while
if ( i >= fs->exporter_count ) {
i++;
e = e->next;
continue;
}
exporter_stats->stat[i].sysid = e->info.sysid;
exporter_stats->stat[i].sequence_failure = e->sequence_failure;
exporter_stats->stat[i].packets = e->packets;
exporter_stats->stat[i].flows = e->flows;
#ifdef DEVEL
printf("Stat: SysID: %u, version: %u, ID: %2u, Packets: %llu, Flows: %llu, Sequence Failures: %u\n", e->info.sysid,
e->info.version, e->info.id, e->packets, e->flows, e->sequence_failure);
#endif
// reset counters
e->sequence_failure = 0;
e->packets = 0;
e->flows = 0;
i++;
e = e->next;
}
AppendToBuffer(fs->nffile, (void *)exporter_stats, size);
free(exporter_stats);
if ( i != fs->exporter_count ) {
LogError("ERROR: exporter stats: Expected %u records, but found %u in %s line %d: %s\n",
fs->exporter_count, i, __FILE__, __LINE__, strerror(errno) );
}
} // End of FlushExporterStats
int HasOptionTable(FlowSource_t *fs, uint16_t id ) {
option_offset_t *t;
t = fs->option_offset_table;
while ( t && t->id != id )
t = t->next;
dbg_printf("Has option table: %s\n", t == NULL ? "not found" : "found");
return t != NULL;
} // End of HasOptionTable

184
bin/collector.h Executable file
View File

@ -0,0 +1,184 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: collector.h 51 2010-01-29 09:01:54Z haag $
*
* $LastChangedRevision: 51 $
*
*/
#ifndef _COLLECTOR_H
#define _COLLECTOR_H 1
#define FNAME_SIZE 256
#define IDENT_SIZE 32
typedef struct srecord_s {
char fname[FNAME_SIZE]; // file name
char subdir[FNAME_SIZE]; // subdir name
char tstring[16]; // actually 12 needed e.g. 200411011230
time_t tstamp; // UNIX time stamp
int failed; // in case of an error
} srecord_t;
// common_record_t defines ext_map as uint_8, so max 256 extension maps allowed.
// should be enough anyway
typedef struct option_offset_s {
struct option_offset_s *next;
uint32_t id; // table id
uint32_t flags; // info about this map
// sampling offsets
#define HAS_SAMPLER_DATA 1
uint16_t offset_id;
uint16_t sampler_id_length;
uint16_t offset_mode;
uint16_t offset_interval;
#define HAS_STD_SAMPLER_DATA 2
uint16_t offset_std_sampler_interval;
uint16_t offset_std_sampler_algorithm;
} option_offset_t;
typedef struct generic_sampler_s {
struct generic_sampler_s *next;
sampler_info_record_t info;
} generic_sampler_t;
typedef struct generic_exporter_s {
// link chain
struct generic_exporter_s *next;
// generic exporter information
exporter_info_record_t info;
uint64_t packets; // number of packets sent by this exporter
uint64_t flows; // number of flow records sent by this exporter
uint32_t sequence_failure; // number of sequence failues
generic_sampler_t *sampler;
} generic_exporter_t;
typedef struct FlowSource_s {
// link
struct FlowSource_s *next;
// exporter identifiers
char Ident[IDENTLEN];
ip_addr_t ip;
uint32_t sa_family;
int any_source;
bookkeeper_t *bookkeeper;
// all about data storage
char *datadir; // where to store data for this source
char *current; // current file name - typically nfcad.current.pid
nffile_t *nffile; // the writing file handle
// statistical data per source
uint32_t bad_packets;
uint64_t first_seen; // in msec
uint64_t last_seen; // in msec
// port histogram data
xstat_t *xstat;
// Any exporter specific data
generic_exporter_t *exporter_data;
uint32_t exporter_count;
struct timeval received;
// extension map list
struct {
#define BLOCK_SIZE 16
int next_free;
int max_maps;
int num_maps;
extension_map_t **maps;
} extension_map_list;
option_offset_t *option_offset_table;
} FlowSource_t;
/* input buffer size, to read data from the network */
#define NETWORK_INPUT_BUFF_SIZE 65535 // Maximum UDP message size
// prototypes
int AddFlowSource(FlowSource_t **FlowSource, char *ident);
int AddDefaultFlowSource(FlowSource_t **FlowSource, char *ident, char *path);
int SetDynamicSourcesDir(FlowSource_t **FlowSource, char *dir);
FlowSource_t *AddDynamicSource(FlowSource_t **FlowSource, struct sockaddr_storage *ss);
int InitExtensionMapList(FlowSource_t *fs);
int ReInitExtensionMapList(FlowSource_t *fs);
int RemoveExtensionMap(FlowSource_t *fs, extension_map_t *map);
int AddExtensionMap(FlowSource_t *fs, extension_map_t *map);
void FlushStdRecords(FlowSource_t *fs);
void FlushExporterStats(FlowSource_t *fs);
int FlushInfoExporter(FlowSource_t *fs, exporter_info_record_t *exporter);
int FlushInfoSampler(FlowSource_t *fs, sampler_info_record_t *sampler);
int HasOptionTable(FlowSource_t *fs, uint16_t id );
void launcher (char *commbuff, FlowSource_t *FlowSource, char *process, int expire);
/* Default time window in seconds to rotate files */
#define TIME_WINDOW 300
/* overdue time:
* if nfcapd does not get any data, wake up the receive system call
* at least after OVERDUE_TIME seconds after the time window
*/
#define OVERDUE_TIME 10
// time nfcapd will wait for launcher to terminate
#define LAUNCHER_TIMEOUT 60
#define SYSLOG_FACILITY "daemon"
#endif //_COLLECTOR_H

126
bin/collector_inline.c Executable file
View File

@ -0,0 +1,126 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: collector_inline.c 37 2009-11-10 08:40:30Z haag $
*
* $LastChangedRevision: 37 $
*
*/
static inline FlowSource_t *GetFlowSource(struct sockaddr_storage *ss) {
FlowSource_t *fs;
void *ptr;
ip_addr_t ip;
char as[100];
union {
struct sockaddr_storage *ss;
struct sockaddr *sa;
struct sockaddr_in *sa_in;
struct sockaddr_in6 *sa_in6;
} u;
u.ss = ss;
switch (ss->ss_family) {
case PF_INET: {
#ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN
if (ss->ss_len != sizeof(struct sockaddr_in) ) {
// malformed struct
LogError("Malformed IPv4 socket struct in '%s', line '%d'", __FILE__, __LINE__ );
return NULL;
}
#endif
ip.v6[0] = 0;
ip.v6[1] = 0;
ip.v4 = ntohl(u.sa_in->sin_addr.s_addr);
ptr = &u.sa_in->sin_addr;
} break;
case PF_INET6: {
uint64_t *ip_ptr = (uint64_t *)u.sa_in6->sin6_addr.s6_addr;
#ifdef HAVE_STRUCT_SOCKADDR_STORAGE_SS_LEN
if (ss->ss_len != sizeof(struct sockaddr_in6) ) {
// malformed struct
LogError("Malformed IPv6 socket struct in '%s', line '%d'", __FILE__, __LINE__ );
return NULL;
}
#endif
// ptr = &((struct sockaddr_in6 *)sa)->sin6_addr;
ip.v6[0] = ntohll(ip_ptr[0]);
ip.v6[1] = ntohll(ip_ptr[1]);
ptr = &u.sa_in6->sin6_addr;
} break;
default:
// keep compiler happy
ip.v6[0] = 0;
ip.v6[1] = 0;
ptr = NULL;
LogError("Unknown sa fanily: %d in '%s', line '%d'", ss->ss_family, __FILE__, __LINE__ );
return NULL;
}
#ifdef DEVEL
inet_ntop(ss->ss_family, ptr, as, sizeof(as));
as[99] = '\0';
printf("Flow Source IP: %s\n", as);
#endif
fs = FlowSource;
while ( fs ) {
if ( ip.v6[0] == fs->ip.v6[0] && ip.v6[1] == fs->ip.v6[1] )
return fs;
// if we match any source, store the current IP address - works as faster cache next time
// and identifies the current source by IP
if ( fs->any_source ) {
fs->ip = ip;
fs->sa_family = ss->ss_family;
return fs;
}
fs = fs->next;
}
if ( ptr ) {
inet_ntop (ss->ss_family, ptr, as, 100);
} else
strncpy(as, "<unknown>", 99);
as[99] = '\0';
LogError("Unknown flow source: '%s'" , as);
return NULL;
} // End of GetFlowSource

273
bin/content_dns.c Normal file
View File

@ -0,0 +1,273 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2013, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author$
*
* $Id$
*
* $LastChangedRevision$
*
*/
#include "config.h"
#ifdef HAVE_FEATURES_H
#include <features.h>
#endif
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <unistd.h>
#include <netinet/in.h>
#ifdef HAVE_ARPA_NAMESER_H
#include <arpa/nameser.h>
#endif
#ifdef HAVE_ARPA_NAMESER_COMPAT_H
#include <arpa/nameser_compat.h>
#endif
#include <resolv.h>
#include <pthread.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <netinet/ip6.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>
#include <netinet/ip_icmp.h>
#include <netinet/icmp6.h>
#include <arpa/inet.h>
#include <pcap.h>
#include "util.h"
#include "nffile.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "flowtree.h"
#include "content_dns.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#include "inline.c"
/*
* Structure for query header. The order of the fields is machine- and
* compiler-dependent, depending on the byte/bit order and the layout
* of bit fields. We use bit fields only in int variables, as this
* is all ANSI requires. This requires a somewhat confusing rearrangement.
*/
typedef struct dns_header_s {
unsigned id :16; /* query identification number */
#ifdef WORDS_BIGENDIAN
/* fields in third byte */
unsigned qr: 1; /* response flag */
unsigned opcode: 4; /* purpose of message */
unsigned aa: 1; /* authoritive answer */
unsigned tc: 1; /* truncated message */
unsigned rd: 1; /* recursion desired */
/* fields in fourth byte */
unsigned ra: 1; /* recursion available */
unsigned unused :3; /* unused bits (MBZ as of 4.9.3a3) */
unsigned rcode :4; /* response code */
#else
/* fields in third byte */
unsigned rd :1; /* recursion desired */
unsigned tc :1; /* truncated message */
unsigned aa :1; /* authoritive answer */
unsigned opcode :4; /* purpose of message */
unsigned qr :1; /* response flag */
/* fields in fourth byte */
unsigned rcode :4; /* response code */
unsigned unused :3; /* unused bits (MBZ as of 4.9.3a3) */
unsigned ra :1; /* recursion available */
#endif
/* remaining bytes */
unsigned qdcount :16; /* number of question entries */
unsigned ancount :16; /* number of answer entries */
unsigned nscount :16; /* number of authority entries */
unsigned arcount :16; /* number of resource entries */
} dns_header_t;
#define DNS_QUERY_TYPE_A (1)
#define DNS_QUERY_TYPE_AAAA (2)
#define DNS_QUERY_TYPE_SRV (3)
typedef struct dns_host_st {
struct dns_host_st *next;
unsigned int type;
unsigned int class;
unsigned int ttl;
void *rr;
} dns_host_t;
typedef struct dns_srv_st {
unsigned int priority;
unsigned int weight;
unsigned int port;
unsigned int rweight;
char name[256];
} dns_srv_t;
static void *_a_rr(void **p) {
struct in_addr in;
in.s_addr = ntohl(Get_val32(*p)); *p += 4;
return strdup(inet_ntoa(in));
}
static void *_aaaa_rr(void **p) {
char addr[INET6_ADDRSTRLEN];
uint64_t sa6[2];
memcpy((void *)sa6, *p, 16);
inet_ntop(AF_INET6, (struct sockaddr_storage *)&sa6, addr, sizeof(addr));
return strdup(addr);
}
void content_decode_dns(struct FlowNode *node, uint8_t *payload, uint32_t payload_size) {
dns_header_t *dns_header = (dns_header_t *)payload;
uint32_t qdcount, ancount;
void *p, *eod;
#define DN_LENGTH 256
char dn[DN_LENGTH];
int i;
if ( payload_size < sizeof(dns_header_t) ) {
LogError("Unable to decode short DNS packet");
return;
}
// no of query packets
qdcount = ntohs(dns_header->qdcount);
dbg_printf("DNS Queries: %u\n", qdcount);
// no of answer packets
ancount = ntohs(dns_header->ancount);
dbg_printf("DNS Answers: %u\n", ancount);
// end of dns packet
eod = (void *)(payload + payload_size);
// reord pointer
p = (void *)(payload + sizeof(dns_header_t));
for (i=0; i<qdcount && p < eod; i++ ) {
int32_t len = dn_expand(payload, eod, p, dn, DN_LENGTH);
if (len < 0) {
LogError("dn_expand() failed: %s", "");
}
dbg_printf("DNS Query dn_expand: %s\n", dn);
p = (void *) (p + len + 4); // + 4 bytes of fixed data in query
}
for (i=0; i<ancount && p < eod; i++ ) {
uint32_t type, class, ttl;
int32_t len = dn_expand(payload, eod, p, dn, DN_LENGTH);
if(len < 0) {
LogError("dn_expand() failed: %s", "");
}
dbg_printf("DNS Answer %i dn_expand: %s ", i, dn);
p += len;
/* extract the various parts of the record */
type = Get_val16(p); p += 2;
class = Get_val16(p); p += 2;
ttl = Get_val32(p); p += 4;
len = Get_val16(p); p += 2;
dbg_printf(" Type: %u, class: %u, ttl: %u, len: %u ", type, class, ttl, len);
/* type-specific processing */
switch(type) {
char *s;
#ifdef T_A
case T_A:
#else
case ns_t_a:
#endif
s = _a_rr(&p);
dbg_printf("A: %s\n", s);
free(s);
break;
#ifdef T_A6
case T_A6:
#endif
#ifdef T_AAAA
case T_AAAA:
#else
case ns_t_a6:
case ns_t_aaaa:
#endif
s = _aaaa_rr(&p);
dbg_printf("AAAA: %s\n", s);
free(s);
break;
#ifdef T_CNAME
case T_CNAME: {
#else
case ns_t_cname: {
#endif
int32_t len = dn_expand(payload, eod, p, dn, DN_LENGTH);
dbg_printf("CNAME: %s\n", dn);
p = (void *)(p + len);
} break;
default:
dbg_printf("<unkn>\n");
p = (void *)(p + len);
continue;
}
}
} // End of content_decode_dns

39
bin/content_dns.h Normal file
View File

@ -0,0 +1,39 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2013, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author$
*
* $Id$
*
* $LastChangedRevision$
*
*/
void content_decode_dns(struct FlowNode *node, uint8_t *payload, uint32_t payload_size);

822
bin/expire.c Normal file
View File

@ -0,0 +1,822 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: expire.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/uio.h>
#include <unistd.h>
#include <signal.h>
#include <fcntl.h>
#include <errno.h>
#include <ctype.h>
#include <time.h>
#ifdef HAVE_FTS_H
# include <fts.h>
#else
# include "fts_compat.h"
#define fts_children fts_children_compat
#define fts_close fts_close_compat
#define fts_open fts_open_compat
#define fts_read fts_read_compat
#define fts_set fts_set_compat
#endif
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "bookkeeper.h"
#include "nfstatfile.h"
#include "expire.h"
static uint32_t timeout = 0;
/*
* expire.c is needed for daemon code as well as normal stdio code
* therefore a generic LogError is defined, which maps to the
* approriate logging channel - either stderr or syslog
*/
void LogError(char *format, ...);
static void PrepareDirLists(channel_t *channel);
static int compare(const FTSENT **f1, const FTSENT **f2);
#if 0
#define unlink unlink_debug
static int unlink_debug (const char *path) {
printf("Unlink %s\n", path);
return 0;
} // End of unlink_debug
#endif
static void IntHandler(int signal) {
switch (signal) {
case SIGALRM:
timeout = 1;
break;
case SIGHUP:
case SIGINT:
case SIGTERM:
timeout = 1;
break;
break;
case SIGCHLD:
default:
// ignore everything we don't know
break;
}
} /* End of IntHandler */
static void SetupSignalHandler(void) {
struct sigaction act;
memset((void *)&act,0,sizeof(struct sigaction));
act.sa_handler = IntHandler;
sigemptyset(&act.sa_mask);
act.sa_flags = 0;
sigaction(SIGTERM, &act, NULL);
sigaction(SIGINT, &act, NULL);
sigaction(SIGHUP, &act, NULL);
sigaction(SIGALRM, &act, NULL);
sigaction(SIGCHLD, &act, NULL);
} // End of SetupSignalHandler
uint64_t ParseSizeDef(char *s, uint64_t *value) {
char *p;
uint64_t fac;
int dot;
dot = 0;
*value = 0;
p = s;
while ( *p ) {
if ( *p < '0' || *p > '9' ) {
if ( *p == '.' ) {
if ( dot ) {
break;
} else {
dot = 1;
}
} else
break;
}
p++;
}
if ( p == s ) {
fprintf(stderr, "Missing number in '%s'\n", s);
return 0;
}
fac = 0;
switch (*p) {
case '\0':
case 'b':
case 'B':
fac = 1;
break;
case 'k':
case 'K':
fac = 1024;
break;
case 'm':
case 'M':
fac = 1024 * 1024;
break;
case 'g':
case 'G':
fac = 1024 * 1024 * 1024;
break;
case 't':
case 'T':
fac = 1024LL * 1024LL * 1024LL * 1024LL;
break;
default:
fprintf(stderr, "Garbage character(s) '%s' in '%s'\n", p, s);
return 0;
}
if ( *p ) {
char *r = p++;
// skip optional 'B' for Bytes in KB etc.
if ( fac != 1 && (*p == 'B' || *p == 'b') ) p++;
if ( *p ) {
// extra garbage after factor
fprintf(stderr, "Garbage character(s) '%s''in '%s'\n", p, s);
return 0;
}
*r = '\0';
}
// *value = strtoll(s, NULL, 10) * fac;
*value = (uint64_t)(atof(s) * (double)fac);
return 1;
} // End of ParseSizeDef
/*
* Accepted time scales
*
* w week
* d day
* H hour
* M minute
*/
uint64_t ParseTimeDef(char *s, uint64_t *value) {
char *p;
uint64_t weeks, days, hours, minutes;
*value = 0;
weeks=days=hours=minutes=0;
p = s;
while ( *p ) {
char *q = p;
while ( *p ) {
if ( *p < '0' || *p > '9' ) {
break;
}
p++;
}
if ( p == q ) {
fprintf(stderr, "Missing number before '%s'\n", q);
return 0;
}
switch (*p) {
case 'w':
*p++ = '\0';
if ( weeks ) {
fprintf(stderr, "Ambiguous weeks %sw\n", q);
return 0;
}
weeks = strtoll(q, NULL, 10);
break;
case 'd':
*p++ = '\0';
if ( days ) {
fprintf(stderr, "Ambiguous days %sD\n", q);
return 0;
}
days = strtoll(q, NULL, 10);
break;
case '\0': // without any units, assume hours
case 'H':
if ( *p ) *p++ = '\0';
if ( hours ) {
fprintf(stderr, "Ambiguous hours %sH\n", q);
return 0;
}
hours = strtoll(q, NULL, 10);
break;
case 'M':
*p++ = '\0';
if ( minutes ) {
fprintf(stderr, "Ambiguous minutes %sM\n", q);
return 0;
}
minutes = strtoll(q, NULL, 10);
break;
default:
fprintf(stderr, "Unknown time unit '%s'\n", q);
return 0;
}
}
*value = minutes*60 + hours*3600 + days*24*3600 + weeks*7*24*3600;
return 1;
} // End of ParseTimeDef
static int compare(const FTSENT **f1, const FTSENT **f2) {
return strcmp( (*f1)->fts_name, (*f2)->fts_name);
} // End of compare
void RescanDir(char *dir, dirstat_t *dirstat) {
FTS *fts;
FTSENT *ftsent;
char *const path[] = { dir, NULL };
char first_timestring[16], last_timestring[16];
dirstat->filesize = dirstat->numfiles = 0;
dirstat->first = 0;
dirstat->last = 0;
strncpy(first_timestring, "999999999999", 15);
strncpy(last_timestring, "000000000000", 15);
fts = fts_open(path, FTS_LOGICAL, compare);
if ( !fts ) {
LogError( "fts_open() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
while ( (ftsent = fts_read(fts)) != NULL) {
if ( ftsent->fts_info == FTS_F && ftsent->fts_namelen == 19 ) {
// nfcapd.200604301200 strlen = 19
if ( strncmp(ftsent->fts_name, "nfcapd.", 7) == 0 ) {
char *s, *p = &(ftsent->fts_name[7]);
// make sure, we have only digits
s = p;
while ( *s ) {
if ( *s < '0' || *s > '9' )
break;
s++;
}
// otherwise skip
if ( *s )
continue;
if ( strcmp(p, first_timestring) < 0 ) {
first_timestring[0] = '\0';
strncat(first_timestring, p, 15);
}
if ( strcmp(p, last_timestring) > 0 ) {
last_timestring[0] = '\0';
strncat(last_timestring, p, 15);
}
dirstat->filesize += 512 * ftsent->fts_statp->st_blocks;
dirstat->numfiles++;
}
} else {
switch (ftsent->fts_info) {
case FTS_D:
// skip all '.' entries as well as hidden directories
if ( ftsent->fts_level > 0 && ftsent->fts_name[0] == '.' )
fts_set(fts, ftsent, FTS_SKIP);
// any valid dirctory need to start with a digit ( %Y -> year )
if ( ftsent->fts_level > 0 && !isdigit(ftsent->fts_name[0]) )
fts_set(fts, ftsent, FTS_SKIP);
break;
case FTS_DP:
break;
}
}
}
fts_close(fts);
// no files means do rebuild next time, otherwise the stat record may not be accurate
if ( dirstat->numfiles == 0 ) {
dirstat->first = dirstat->last = time(NULL);
dirstat->status = FORCE_REBUILD;
} else {
dirstat->first = ISO2UNIX(first_timestring);
dirstat->last = ISO2UNIX(last_timestring);
dirstat->status = STATFILE_OK;
}
} // End of RescanDir
void ExpireDir(char *dir, dirstat_t *dirstat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ) {
FTS *fts;
FTSENT *ftsent;
uint64_t sizelimit, num_expired;
int done, size_done, lifetime_done, dir_files;
char *const path[] = { dir, NULL };
char *expire_timelimit = NULL;
time_t now = time(NULL);
dir_files = 0;
if ( dirstat->low_water == 0 )
dirstat->low_water = 95;
if ( runtime ) {
SetupSignalHandler();
alarm(runtime);
}
if ( maxlife ) {
// build an appropriate string for comparing
time_t t_expire = now - maxlife;
time_t t_watermark = now - (time_t)((maxlife * dirstat->low_water)/100);
// printf("Expire files before %s", ctime(&t_expire));
expire_timelimit = strdup(UNIX2ISO(t_watermark));
// printf("down to %s", ctime(&t_watermark));
// printf("Diff: %i\n", t_watermark - t_expire );
if ( dirstat->last < t_expire && (isatty(STDIN_FILENO) ) ) {
// this means all files will get expired - are you sure ?
char *s, s1[32], s2[32];
time_t t;
struct tm *when;
t = t_expire;
when = localtime(&t);
strftime(s1, 31, "%Y-%m-%d %H:%M:%S", when);
s1[31] = '\0';
t = dirstat->last;
when = localtime(&t);
strftime(s2, 31, "%Y-%m-%d %H:%M:%S", when);
s2[31] = '\0';
printf("Your max lifetime of %s will expire all file before %s\n", ScaleTime(maxlife), s1);
printf("Your latest files are dated %s. This means all files will be deleted!\n", s2);
printf("Are you sure? yes/[no] ");
s = fgets(s1, 31, stdin);
s1[31] = '\0';
if ( s && strncasecmp(s1, "yes\n", 31) == 0 ) {
printf("Ok - you've beeen warned!\n");
} else {
printf("Expire canceled!\n");
return;
}
}
}
done = 0;
size_done = maxsize == 0 || dirstat->filesize < maxsize;
lifetime_done = maxlife == 0 || ( now - dirstat->first ) < maxlife;
sizelimit = (dirstat->low_water * maxsize)/100;
num_expired = 0;
fts = fts_open(path, FTS_LOGICAL, compare);
while ( !done && ((ftsent = fts_read(fts)) != NULL) ) {
if ( ftsent->fts_info == FTS_F ) {
dir_files++; // count files in directories
if ( ftsent->fts_namelen == 19 && strncmp(ftsent->fts_name, "nfcapd.", 7) == 0 ) {
// nfcapd.200604301200 strlen = 19
char *s, *p = &(ftsent->fts_name[7]);
// process only nfcapd. files
// make sure it's really an nfcapd. file and we have
// only digits in the rest of the file name
s = p;
while ( *s ) {
if ( *s < '0' || *s > '9' )
break;
s++;
}
// otherwise skip
if ( *s )
continue;
// expire size-wise if needed
if ( !size_done ) {
if ( dirstat->filesize > sizelimit ) {
if ( unlink(ftsent->fts_path) == 0 ) {
dirstat->filesize -= 512 * ftsent->fts_statp->st_blocks;
num_expired++;
dir_files--;
} else {
LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
continue; // next file if file was unlinked
} else {
dirstat->first = ISO2UNIX(p); // time of first file not expired
size_done = 1;
}
}
// expire time-wise if needed
// this part of the code is executed only when size-wise is fullfilled
if ( !lifetime_done ) {
if ( expire_timelimit && strcmp(p, expire_timelimit) < 0 ) {
if ( unlink(ftsent->fts_path) == 0 ) {
dirstat->filesize -= 512 * ftsent->fts_statp->st_blocks;
num_expired++;
dir_files--;
} else {
LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
lifetime_done = 0;
} else {
dirstat->first = ISO2UNIX(p); // time of first file not expired
lifetime_done = 1;
}
}
done = (size_done && lifetime_done) || timeout;
}
} else {
switch (ftsent->fts_info) {
case FTS_D:
// set pre-order flag
dir_files = 0;
// skip all '.' entries as well as hidden directories
if ( ftsent->fts_level > 0 && ftsent->fts_name[0] == '.' )
fts_set(fts, ftsent, FTS_SKIP);
// any valid directory needs to start with a digit ( %Y -> year )
if ( ftsent->fts_level > 0 && !isdigit(ftsent->fts_name[0]) )
fts_set(fts, ftsent, FTS_SKIP);
break;
case FTS_DP:
// do not delete base data directory ( level == 0 )
if ( dir_files == 0 && ftsent->fts_level > 0 ) {
// directory is empty and can be deleted
// printf("Will remove directory %s\n", ftsent->fts_path);
if ( rmdir(ftsent->fts_path) != 0 ) {
LogError( "rmdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
}
break;
}
}
}
fts_close(fts);
if ( !done ) {
// all files expired and limits not reached
// this may be possible, when files get time-wise expired and
// the time limit is shorter than the latest file
dirstat->first = dirstat->last;
}
if ( runtime )
alarm(0);
if ( timeout ) {
LogError( "Maximum execution time reached! Interrupt expire.\n");
}
if ( num_expired > dirstat->numfiles ) {
LogError( "Error updating stat record: Number of files inconsistent!\n");
LogError( "Will automatically rebuild this directory next time\n");
dirstat->numfiles = 0;
dirstat->status = FORCE_REBUILD;
} else {
dirstat->numfiles -= num_expired;
}
if ( dirstat->numfiles == 0 ) {
dirstat->first = dirstat->last = time(NULL);
dirstat->status = FORCE_REBUILD;
}
free(expire_timelimit);
} // End of ExpireDir
static void PrepareDirLists(channel_t *channel) {
channel_t *current_channel = channel;
while ( current_channel ) {
char *const path[] = { current_channel->datadir, NULL };
current_channel->fts = fts_open(path, FTS_LOGICAL|FTS_NOCHDIR, compare);
if ( !current_channel->fts ) {
LogError( "fts_open() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
continue;
}
// get first entry
current_channel->ftsent = fts_read(current_channel->fts);
if ( current_channel->ftsent )
// use fts_number as the number of files already seen in this directory.
current_channel->ftsent->fts_number = 0;
while ( current_channel->ftsent ) {
/*
FTSENT *ftsent = current_channel->ftsent;
char *finfo;
switch (ftsent->fts_info) {
case FTS_ERR:
case FTS_NS:
LogError( "fts_read() %s error in %s line %d: %s\n",
current_channel->ftsent->fts_path, __FILE__, __LINE__, strerror(current_channel->ftsent->fts_errno) );
break;
case FTS_D:
finfo = "DIR pre ";
break;
case FTS_DP:
finfo = "DIR post";
break;
case FTS_F:
finfo = "FILE ";
break;
default:
finfo = "<undef> ";
}
printf("%u %i %s %s %s\n", ftsent->fts_info, ftsent->fts_level, finfo, ftsent->fts_path, ftsent->fts_name);
*/
if ( current_channel->ftsent->fts_info == FTS_ERR ||
current_channel->ftsent->fts_info == FTS_NS) {
LogError( "fts_read() %s error in %s line %d: %s\n",
current_channel->ftsent->fts_path, __FILE__, __LINE__, strerror(current_channel->ftsent->fts_errno) );
continue;
}
if ( current_channel->ftsent->fts_info != FTS_F ) {
current_channel->ftsent = fts_read(current_channel->fts);
continue;
}
// it's now FTS_F
current_channel->ftsent->fts_number++;
// if ftsent points to first valid file, break
if ( current_channel->ftsent->fts_namelen == 19 && strncmp(current_channel->ftsent->fts_name, "nfcapd.", 7) == 0 )
break;
// otherwise loop
current_channel->ftsent = fts_read(current_channel->fts);
}
current_channel = current_channel->next;
}
} // End of PrepareDirLists
void ExpireProfile(channel_t *channel, dirstat_t *current_stat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime ) {
int size_done, lifetime_done, done;
char *expire_timelimit = "";
time_t now = time(NULL);
uint64_t sizelimit, num_expired;
if ( !channel )
return;
done = 0;
SetupSignalHandler();
if ( maxlife ) {
// time_t t_expire = now - maxlife;
// build an appropriate string for comparing
time_t t_watermark = now - (time_t)((maxlife * current_stat->low_water)/100);
// printf("Expire files before %s", ctime(&t_expire));
expire_timelimit = strdup(UNIX2ISO(t_watermark));
// printf("down to %s", ctime(&t_watermark));
// printf("Diff: %i\n", t_watermark - t_expire );
}
size_done = maxsize == 0 || current_stat->filesize < maxsize;
sizelimit = (current_stat->low_water * maxsize)/100;
lifetime_done = maxlife == 0 || ( now - current_stat->first ) < maxlife;
num_expired = 0;
PrepareDirLists(channel);
if ( runtime )
alarm(runtime);
while ( !done ) {
char *p;
int file_removed;
// search for the channel with oldest file. If all channel have same age,
// get the last in the list
channel_t *expire_channel = channel;
channel_t *compare_channel = expire_channel->next;
while ( compare_channel ) {
if ( expire_channel->ftsent == NULL ) {
expire_channel = compare_channel;
}
if ( compare_channel->ftsent == NULL ) {
compare_channel = compare_channel->next;
continue;
}
// at this point expire_channel and current_channel fts entries are valid
if ( strcmp(expire_channel->ftsent->fts_name, compare_channel->ftsent->fts_name) >= 0 ) {
expire_channel = compare_channel;
}
compare_channel = compare_channel->next;
}
if ( !expire_channel->ftsent ) {
// no more entries in any channel - we are done
done = 1;
continue;
}
// flag is file got removed
file_removed = 0;
// expire_channel now points to the channel with oldest file
// do expire
p = &(expire_channel->ftsent->fts_name[7]);
// printf("File: %s\n", expire_channel->ftsent->fts_path);
if ( !size_done ) {
// expire size-wise if needed
// printf(" Size expire %llu %llu\n", current_stat->filesize, sizelimit);
if ( current_stat->filesize > sizelimit ) {
// need to delete this file
if ( unlink(expire_channel->ftsent->fts_path) == 0 ) {
// Update profile stat
current_stat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks;
current_stat->numfiles--;
// Update channel stat
expire_channel->dirstat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks;
expire_channel->dirstat->numfiles--;
// decrement number of files seen in this directory
expire_channel->ftsent->fts_number--;
file_removed = 1;
num_expired++;
} else {
LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
} else {
// we are done size-wise
// time of first file not expired = start time of channel/profile
expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p);
size_done = 1;
}
} else if ( !lifetime_done ) {
// printf(" Time expire \n");
// expire time-wise if needed
// this part of the code is executed only when size-wise is already fullfilled
if ( strcmp(p, expire_timelimit) < 0 ) {
// need to delete this file
if ( unlink(expire_channel->ftsent->fts_path) == 0 ) {
// Update profile stat
current_stat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks;
current_stat->numfiles--;
// Update channel stat
expire_channel->dirstat->filesize -= 512 * expire_channel->ftsent->fts_statp->st_blocks;
expire_channel->dirstat->numfiles--;
// decrement number of files seen in this directory
expire_channel->ftsent->fts_number--;
file_removed = 1;
num_expired++;
} else {
LogError( "unlink() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
} else {
// we are done time-wise
// time of first file not expired = start time of channel/profile
expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p);
lifetime_done = 1;
}
} else
// all done
done = 1;
if ( timeout )
done = 1;
// advance fts entry in expire channel to next file, if file was removed
if ( file_removed ) {
expire_channel->ftsent = fts_read(expire_channel->fts);
while ( expire_channel->ftsent ) {
if ( expire_channel->ftsent->fts_info == FTS_F ) { // entry is a file
expire_channel->ftsent->fts_number++;
if ( expire_channel->ftsent->fts_namelen == 19 &&
strncmp(expire_channel->ftsent->fts_name, "nfcapd.", 7) == 0 ) {
// if ftsent points to next valid file
char *p = &(expire_channel->ftsent->fts_name[7]);
// next file is first (oldest) for channel and for profile - update first mark
expire_channel->dirstat->first = current_stat->first = ISO2UNIX(p);
break;
}
} else {
switch (expire_channel->ftsent->fts_info) {
case FTS_D: // entry is a directory
// set number of files seen in this directory = 0
expire_channel->ftsent->fts_number = 0;
// skip all '.' entries as well as hidden directories
if ( expire_channel->ftsent->fts_level > 0 && expire_channel->ftsent->fts_name[0] == '.' )
fts_set(expire_channel->fts, expire_channel->ftsent, FTS_SKIP);
// any valid directory needs to start with a digit ( %Y -> year )
if ( expire_channel->ftsent->fts_level > 0 && !isdigit(expire_channel->ftsent->fts_name[0]) )
fts_set(expire_channel->fts, expire_channel->ftsent, FTS_SKIP);
break;
case FTS_DP:
// do not delete base data directory ( level == 0 )
if ( expire_channel->ftsent->fts_number == 0 && expire_channel->ftsent->fts_level > 0 ) {
// directory is empty and can be deleted
// printf("Will remove directory %s\n", expire_channel->ftsent->fts_path);
if ( rmdir(expire_channel->ftsent->fts_path) != 0 ) {
LogError( "rmdir() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
}
break;
}
}
// otherwise loop
expire_channel->ftsent = fts_read(expire_channel->fts);
} // end advance fts entry
file_removed = 0;
}
if ( expire_channel->ftsent == NULL ) {
// this channel has no more files now
expire_channel->dirstat->first = expire_channel->dirstat->last;
if ( expire_channel->dirstat->numfiles ) {
// if channel is empty, no files must be reported, but rebuild is done anyway
LogError( "Inconsitency detected in channel %s. Will rebuild automatically.\n", expire_channel->datadir);
LogError( "No more files found, but %llu expected.\n", expire_channel->dirstat->numfiles);
}
expire_channel->dirstat->numfiles = 0;
expire_channel->dirstat->status = FORCE_REBUILD;
}
} // while ( !done )
if ( runtime )
alarm(0);
if ( timeout ) {
LogError( "Maximum execution time reached! Interrupt expire.\n");
}
} // End of ExpireProfile
void UpdateBookStat(dirstat_t *dirstat, bookkeeper_t *books) {
if ( books->numfiles ) {
/* prevent some faults and dublicates:
* book records can never be timewise smaller than directory records => fishy!
* in case book records == directory records, the user stopped and restarted nfcapd
* this is not necessarily wrong, but results in overwriting an existing file
* which results in wrong stats => rescan needed
*/
if ( books->last <= dirstat->last || books->first <= dirstat->first) {
dirstat->status = FORCE_REBUILD;
return;
}
dirstat->last = books->last;
dirstat->numfiles += books->numfiles;
dirstat->filesize += books->filesize;
}
} // End of UpdateBookStat

68
bin/expire.h Normal file
View File

@ -0,0 +1,68 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: expire.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _EXPIRE_H
#define _EXPIRE_H 1
typedef struct channel_s {
struct channel_s *next;
char *datadir;
dirstat_t *dirstat;
bookkeeper_t *books;
int books_stat;
int do_rescan;
int status;
FTS *fts;
FTSENT *ftsent;
} channel_t;
enum { OK = 0, NOFILES };
uint64_t ParseSizeDef(char *s, uint64_t *value);
uint64_t ParseTimeDef(char *s, uint64_t *value);
void RescanDir(char *dir, dirstat_t *dirstat);
void ExpireDir(char *dir, dirstat_t *dirstat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime );
void ExpireProfile(channel_t *channel, dirstat_t *current_stat, uint64_t maxsize, uint64_t maxlife, uint32_t runtime );
void UpdateBookStat(dirstat_t *dirstat, bookkeeper_t *books);
#endif //_EXPIRE_H

465
bin/exporter.c Executable file
View File

@ -0,0 +1,465 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2012, Peter Haag
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: exporter.c 224 2014-02-16 12:59:29Z peter $
*
* $LastChangedRevision: 224 $
*
*/
#include "config.h"
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#include <ctype.h>
#include <string.h>
#include <syslog.h>
#include <stdarg.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#include "util.h"
#include "nffile.h"
#include "nfx.h"
#include "nfnet.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "nf_common.h"
#include "netflow_v1.h"
#include "netflow_v5_v7.h"
#include "netflow_v9.h"
#include "ipfix.h"
/* global */
generic_exporter_t **exporter_list;
/* local variables */
#define MAX_EXPORTERS 65535
static generic_exporter_t *exporter_root;
#include "nffile_inline.c"
/* local prototypes */
/* functions */
int InitExporterList(void) {
exporter_list = calloc(MAX_EXPORTERS, sizeof(generic_exporter_t *));
if ( !exporter_list ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
exporter_root = NULL;
return 1;
} // End of InitExporterList
int AddExporterInfo(exporter_info_record_t *exporter_record) {
uint32_t id = exporter_record->sysid;
int i;
char *p1, *p2;
// sanity check
if ( id >= MAX_EXPORTERS ) {
LogError("Exporter id: %u out of range. Skip exporter", id);
return 0;
}
if ( exporter_list[id] != NULL ) {
// slot already taken - check if exporters are identical
exporter_record->sysid = exporter_list[id]->info.sysid;
if ( memcmp((void *)exporter_record, (void *)&(exporter_list[id]->info), sizeof(exporter_info_record_t)) == 0 ) {
dbg_printf("Found identical exporter record at SysID: %i, Slot: %u\n", exporter_record->sysid, id);
// we are done
return 2;
} else {
// exporters not identical - move current slot
int i;
// search first emty slot at the top of the list
for ( i = id+1; i < MAX_EXPORTERS && exporter_list[i] != NULL; i++ ) {;}
if ( i >= MAX_EXPORTERS ) {
// all slots taken
LogError("Too many exporters (>256)\n");
return 0;
}
dbg_printf("Move existing exporter from slot %u, to %i\n", id, i);
// else - move slot
exporter_list[i] = exporter_list[id];
exporter_list[id] = NULL;
exporter_record->sysid = i;
}
}
// slot[id] is now available
exporter_list[id] = (generic_exporter_t *)calloc(1, sizeof(generic_exporter_t));
if ( !exporter_list[id] ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
// SPARC gcc fails here, if we use directly a pointer to the struct.
// SPARC barfs and core dumps otherwise
// memcpy((void *)&(exporter_list[id]->info), (void *)exporter_record, sizeof(exporter_info_record_t));
p1 = (char *)&(exporter_list[id]->info);
p2 = (char *)exporter_record;
for ( i=0; i<sizeof(exporter_info_record_t); i++ )
*p1++ = *p2++;
dbg_printf("Insert exporter record in Slot: %i, Sysid: %u\n", id, exporter_record->sysid);
#ifdef DEVEL
{
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
if ( exporter_record->sa_family == AF_INET ) {
uint32_t _ip = htonl(exporter_record->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %16s, version: %u, ID: %2u, Slot: %u\n", exporter_record->sysid,
ipstr, exporter_record->version, exporter_record->id, id);
} else if ( exporter_record->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(exporter_record->ip.v6[0]);
_ip[1] = htonll(exporter_record->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %40s, version: %u, ID: %2u, Slot: %u\n", exporter_record->sysid,
ipstr, exporter_record->version, exporter_record->id, id);
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
printf("**** Exporter IP version unknown ****\n");
}
}
printf("\n");
#endif
if ( !exporter_root ) {
exporter_root = exporter_list[id];
}
return 1;
} // End of AddExporterInfo
int AddSamplerInfo(sampler_info_record_t *sampler_record) {
uint32_t id = sampler_record->exporter_sysid;
generic_sampler_t **sampler;
if ( !exporter_list[id] ) {
LogError("Exporter SysID: %u not found! - Skip sampler record", id);
return 0;
}
sampler = &exporter_list[id]->sampler;
while ( *sampler ) {
if ( memcmp((void *)&(*sampler)->info, (void *)sampler_record, sizeof(sampler_info_record_t)) == 0 ) {
// Found identical sampler already registered
dbg_printf("Identical sampler already registered: Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n",
sampler_record->exporter_sysid, sampler_record->id, sampler_record->mode, sampler_record->interval);
return 2;
}
sampler = &((*sampler)->next);
}
*sampler = (generic_sampler_t *)malloc(sizeof(generic_sampler_t));
if ( !*sampler ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return 0;
}
(*sampler)->next = NULL;
sampler_record->exporter_sysid = exporter_list[id]->info.sysid;
memcpy((void *)&(*sampler)->info, (void *)sampler_record, sizeof(sampler_info_record_t));
dbg_printf("Insert sampler record for exporter at slot %i:\n", id);
#ifdef DEVEL
{
if ( sampler_record->id < 0 ) {
printf("Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n",
sampler_record->exporter_sysid, sampler_record->mode, sampler_record->interval);
} else {
printf("Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n",
sampler_record->exporter_sysid, sampler_record->id, sampler_record->mode, sampler_record->interval);
}
}
#endif
return 1;
} // End of AddSamplerInfo
int AddExporterStat(exporter_stats_record_t *stat_record) {
int i, use_copy;
exporter_stats_record_t *rec;
// 64bit counters can be potentially unaligned
if ( ((ptrdiff_t)stat_record & 0x7) != 0 ) {
rec = malloc(stat_record->header.size);
if ( !rec ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
memcpy(rec, stat_record, stat_record->header.size);
use_copy = 1;
} else {
rec = stat_record;
use_copy = 0;
}
for (i=0; i<rec->stat_count; i++ ) {
uint32_t id = rec->stat[i].sysid;
if ( !exporter_list[id] ) {
LogError("Exporter SysID: %u not found! - Skip stat record record.\n");
continue;
}
exporter_list[id]->sequence_failure += rec->stat[i].sequence_failure;
exporter_list[id]->packets += rec->stat[i].packets;
exporter_list[id]->flows += rec->stat[i].flows;
dbg_printf("Update exporter stat for SysID: %i: Sequence failures: %u, packets: %llu, flows: %llu\n",
id, exporter_list[id]->sequence_failure, exporter_list[id]->packets, exporter_list[id]->flows);
}
if ( use_copy )
free(rec);
return 1;
} // End of AddExporterStat
void ExportExporterList( nffile_t *nffile ) {
int i;
// sysid 0 unused -> no exporter available
i = 1;
while ( i < MAX_EXPORTERS && exporter_list[i] != NULL ) {
exporter_info_record_t *exporter;
generic_sampler_t *sampler;
exporter = &exporter_list[i]->info;
AppendToBuffer(nffile, (void *)exporter, exporter->header.size);
sampler = exporter_list[i]->sampler;
while ( sampler ) {
AppendToBuffer(nffile, (void *)&(sampler->info), sampler->info.header.size);
sampler = sampler->next;
}
i++;
}
} // End of ExportExporterList
void PrintExporters(char *filename) {
int i, done, found = 0;
nffile_t *nffile;
record_header_t *record;
uint32_t skipped_blocks;
uint64_t total_bytes;
printf("Exporters:\n");
nffile = OpenFile(filename, NULL);
if ( !nffile ) {
return;
}
total_bytes = 0;
skipped_blocks = 0;
done = 0;
while ( !done ) {
int i, ret;
// get next data block from file
ret = ReadBlock(nffile);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
LogError("Corrupt data file '%s': '%s'\n",filename);
else
LogError("Read error in file '%s': %s\n",filename, strerror(errno) );
done = 1;
continue;
break;
// fall through - get next file in chain
case NF_EOF:
done = 1;
continue;
break;
default:
// successfully read block
total_bytes += ret;
}
if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) {
skipped_blocks++;
continue;
}
// block type = 2
record = (record_header_t *)nffile->buff_ptr;
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
switch ( record->type ) {
// ExporterRecordType and SamplerRecordype tc versions only
case ExporterRecordType: {
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
exporter_record_t *exporter_record = (exporter_record_t *)record ;
found = 1;
printf("\n");
if ( exporter_record->sa_family == AF_INET ) {
uint32_t _ip = htonl(exporter_record->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %16s, version: %u, ID: %2u, Sequence Failures: %u\n", exporter_record->sysid,
ipstr, exporter_record->version, exporter_record->exporter_id, exporter_record->sequence_failure);
} else if ( exporter_record->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(exporter_record->ip.v6[0]);
_ip[1] = htonll(exporter_record->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
printf("SysID: %u, IP: %40s, version: %u, ID: %2u, Sequence Failures: %u\n", exporter_record->sysid,
ipstr, exporter_record->version, exporter_record->exporter_id, exporter_record->sequence_failure);
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
printf("**** Exporter IP version unknown ****\n");
}
} break;
case SamplerRecordype: {
sampler_record_t *sampler_record = (sampler_record_t *)record;;
if ( sampler_record->id < 0 ) {
printf(" Generic Sampler: mode: %u, interval: %u\n",
sampler_record->mode, sampler_record->interval);
} else {
printf(" Sampler: id: %i, mode: %u, interval: %u\n",
sampler_record->id, sampler_record->mode, sampler_record->interval);
}
} break;
case ExporterInfoRecordType:
found = 1;
if ( !AddExporterInfo((exporter_info_record_t *)record) ) {
LogError("Failed to add Exporter Record\n");
}
break;
case ExporterStatRecordType:
AddExporterStat((exporter_stats_record_t *)record);
break;
case SamplerInfoRecordype:
if ( !AddSamplerInfo((sampler_info_record_t *)record) ) {
LogError("Failed to add Sampler Record\n");
}
break;
}
// Advance pointer by number of bytes for netflow record
record = (record_header_t *)((pointer_addr_t)record + record->size);
}
}
CloseFile(nffile);
DisposeFile(nffile);
if ( !found ) {
printf("No Exporter records found\n");
}
i = 1;
while ( i < MAX_EXPORTERS && exporter_list[i] != NULL ) {
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
exporter_info_record_t *exporter;
generic_sampler_t *sampler;
printf("\n");
exporter = &exporter_list[i]->info;
if ( exporter->sa_family == AF_INET ) {
uint32_t _ip = htonl(exporter->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
if ( exporter_list[i]->flows )
printf("SysID: %u, IP: %16s, version: %u, ID: %2u, Sequence failures: %u, packets: %llu, flows: %llu\n",
exporter->sysid, ipstr, exporter->version, exporter->id,
exporter_list[i]->sequence_failure,
(long long unsigned)exporter_list[i]->packets,
(long long unsigned)exporter_list[i]->flows);
else
printf("SysID: %u, IP: %16s, version: %u, ID: %2u\n",
exporter->sysid, ipstr, exporter->version, exporter->id);
} else if ( exporter->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(exporter->ip.v6[0]);
_ip[1] = htonll(exporter->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
if ( exporter_list[i]->flows )
printf("SysID: %u, IP: %40s, version: %u, ID: %2u, Sequence failures: %u, packets: %llu, flows: %llu\n ",
exporter->sysid, ipstr, exporter->version, exporter->id,
exporter_list[i]->sequence_failure,
(long long unsigned)exporter_list[i]->packets,
(long long unsigned)exporter_list[i]->flows);
else
printf("SysID: %u, IP: %40s, version: %u, ID: %2u\n ",
exporter->sysid, ipstr, exporter->version, exporter->id);
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
printf("**** Exporter IP version unknown ****\n");
}
sampler = exporter_list[i]->sampler;
while ( sampler ) {
if ( sampler->info.id < 0 ) {
printf(" Sampler for Exporter SysID: %u, Generic Sampler: mode: %u, interval: %u\n",
sampler->info.exporter_sysid, sampler->info.mode, sampler->info.interval);
} else {
printf(" Sampler for Exporter SysID: %u, Sampler: id: %i, mode: %u, interval: %u\n",
sampler->info.exporter_sysid, sampler->info.id, sampler->info.mode, sampler->info.interval);
}
sampler = sampler->next;
}
i++;
}
} // End of PrintExporters

53
bin/exporter.h Executable file
View File

@ -0,0 +1,53 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2012, Peter Haag
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: exporter.h 224 2014-02-16 12:59:29Z peter $
*
* $LastChangedRevision: 224 $
*
*/
#ifndef _EXPORTER_H
#define _EXPORTER_H 1
int InitExporterList(void);
int AddExporterInfo(exporter_info_record_t *exporter_record);
int AddSamplerInfo(sampler_info_record_t *sampler_record);
int AddExporterStat(exporter_stats_record_t *stat_record);
void ExportExporterList( nffile_t *nffile );
void PrintExporters(char *filename);
#endif //_EXPORTER_H

1245
bin/flist.c Normal file
View File

File diff suppressed because it is too large Load Diff

52
bin/flist.h Normal file
View File

@ -0,0 +1,52 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: flist.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _FLIST_H
#define _FLIST_H 1
#define EMPTY_LIST ((nffile_t *)-1)
int InitHierPath(int num);
char *GetSubDir(struct tm *now);
int SetupSubDir(char *dir, char *subdir, char *error, size_t errlen );
nffile_t *GetNextFile(nffile_t *nffile, time_t twin_start, time_t twin_end);
#endif //_FLIST_H

626
bin/flowtree.c Normal file
View File

@ -0,0 +1,626 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2011, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author$
*
* $Id$
*
* $LastChangedRevision$
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <stddef.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdint.h>
#include <pthread.h>
#include <assert.h>
#include "rbtree.h"
#include "util.h"
#include "nffile.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "flowtree.h"
#include "netflow_pcap.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
static int FlowNodeCMP(struct FlowNode *e1, struct FlowNode *e2);
// Insert the IP RB tree code here
RB_GENERATE(FlowTree, FlowNode, entry, FlowNodeCMP);
// Flow Cache to store all nodes
#define FLOWELEMENTNUM 1024 * 1024
//#define FLOWELEMENTNUM 128
static struct FlowNode *FlowElementCache;
// free list
static struct FlowNode *FlowNode_FreeList;
static pthread_mutex_t m_FreeList = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t c_FreeList = PTHREAD_COND_INITIALIZER;
static uint32_t CacheOverflow;
static uint32_t Allocated;
// Flow tree
static FlowTree_t *FlowTree;
static int NumFlows;
// Simple unprotected list
typedef struct FlowNode_list_s {
struct FlowNode *list;
struct FlowNode *tail;
uint32_t size;
} Linked_list_t;
static Linked_list_t UDP_list;
/* static prototypes */
static void AppendFlowNode(Linked_list_t *LinkedList, struct FlowNode *node);
static void DisconnectFlowNode(Linked_list_t *LinkedList, struct FlowNode *node);
static void TouchFlowNode(Linked_list_t *LinkedList, struct FlowNode *node);
/* Free list handling functions */
// Get next free node from free list
struct FlowNode *New_Node(void) {
struct FlowNode *node;
#ifdef USE_MALLOC
node = malloc(sizeof(struct FlowNode));
memset((void *)node, 'A', sizeof(struct FlowNode));
node->left = NULL;
node->right = NULL;
node->data = NULL;
node->memflag = NODE_IN_USE;
dbg_printf("New node: %llx\n", (unsigned long long)node);
return node;
#endif
pthread_mutex_lock(&m_FreeList);
while ( FlowNode_FreeList == NULL ) {
CacheOverflow++;
LogError("Free list exhausted: %u, Flows: %u - sleep", Allocated, NumFlows);
pthread_cond_wait(&c_FreeList, &m_FreeList);
}
node = FlowNode_FreeList;
if ( node == NULL ) {
// should never happen , as we were waiting for a free node
LogError("*** Software ERROR *** New_Node() unexpected error in %s line %d: %s: %u\n",
__FILE__, __LINE__, "Node list exhausted", NumFlows);
pthread_mutex_unlock(&m_FreeList);
return NULL;
}
if ( node->memflag != NODE_FREE ) {
LogError("*** Software ERROR *** New_Node() unexpected error in %s line %d: %s\n",
__FILE__, __LINE__, "Tried to allocate a non free Node");
abort();
}
FlowNode_FreeList = node->right;
Allocated++;
pthread_mutex_unlock(&m_FreeList);
node->left = NULL;
node->right = NULL;
node->memflag = NODE_IN_USE;
return node;
} // End of New_node
// return node into free list
void Free_Node(struct FlowNode *node) {
if ( node->memflag == NODE_FREE ) {
LogError("Free_Node() Fatal: Tried to free an already freed Node");
abort();
}
if ( node->memflag != NODE_IN_USE ) {
LogError("Free_Node() Fatal: Tried to free a Node not in use");
abort();
}
if ( node->data ) {
free(node->data);
node->data = NULL;
}
#ifdef DEVEL
assert(node->left == NULL);
assert(node->right == NULL);
#endif
#ifdef USE_MALLOC
dbg_printf("Free node: %llx\n", (unsigned long long)node);
node->memflag = NODE_FREE;
memset((void *)node, 'B', sizeof(struct FlowNode));
free(node);
return;
#endif
memset((void *)node, 0, sizeof(struct FlowNode));
pthread_mutex_lock(&m_FreeList);
node->right = FlowNode_FreeList;
node->left = NULL;
node->memflag = NODE_FREE;
FlowNode_FreeList = node;
Allocated--;
pthread_mutex_unlock(&m_FreeList);
if ( CacheOverflow ) {
CacheOverflow = 0;
pthread_cond_signal(&c_FreeList);
}
} // End of Free_Node
/* safety check - this must never become 0 - otherwise the cache is too small */
uint32_t CacheCheck(void) {
return FLOWELEMENTNUM - NumFlows;
} // End of CacheCheck
/* flow tree functions */
int Init_FlowTree(uint32_t CacheSize) {
int i;
FlowTree = malloc(sizeof(FlowTree_t));
if ( !FlowTree ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return 0;
}
RB_INIT(FlowTree);
if ( CacheSize == 0 )
CacheSize = FLOWELEMENTNUM;
FlowElementCache = calloc(CacheSize, sizeof(struct FlowNode));
if ( !FlowElementCache ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
free(FlowTree);
FlowTree = NULL;
return 0;
}
// init free list
FlowNode_FreeList = FlowElementCache;
FlowNode_FreeList->left = NULL;
FlowNode_FreeList->right = &FlowElementCache[1];
FlowNode_FreeList->memflag = NODE_FREE;
for (i=1; i < (CacheSize-1); i++ ) {
FlowElementCache[i].memflag = NODE_FREE;
FlowElementCache[i].left = &FlowElementCache[i-1];
FlowElementCache[i].right = &FlowElementCache[i+1];
}
FlowElementCache[i].left = &FlowElementCache[i-1];
FlowElementCache[i].right = NULL;
FlowElementCache[i].memflag = NODE_FREE;
CacheOverflow = 0;
Allocated = 0;
NumFlows = 0;
UDP_list.list = NULL;
UDP_list.tail = NULL;
UDP_list.size = 0;
return 1;
} // End of Init_FlowTree
void Dispose_FlowTree(void) {
struct FlowNode *node, *nxt;
// Dump all incomplete flows to the file
for (node = RB_MIN(FlowTree, FlowTree); node != NULL; node = nxt) {
nxt = RB_NEXT(FlowTree, FlowTree, node);
RB_REMOVE(FlowTree, FlowTree, node);
if ( node->data )
free(node->data);
}
free(FlowElementCache);
FlowElementCache = NULL;
FlowNode_FreeList = NULL;
CacheOverflow = 0;
} // End of Dispose_FlowTree
static int FlowNodeCMP(struct FlowNode *e1, struct FlowNode *e2) {
uint64_t *a = e1->src_addr.v6;
uint64_t *b = e2->src_addr.v6;
int i;
#define CMPLEN (offsetof(struct FlowNode, _ENDKEY_) - offsetof(struct FlowNode, src_addr))
i = memcmp((void *)a, (void *)b, CMPLEN );
return i;
} // End of FlowNodeCMP
struct FlowNode *Lookup_Node(struct FlowNode *node) {
return RB_FIND(FlowTree, FlowTree, node);
} // End of Lookup_FlowTree
struct FlowNode *Insert_Node(struct FlowNode *node) {
struct FlowNode *n;
// return RB_INSERT(FlowTree, FlowTree, node);
n = RB_INSERT(FlowTree, FlowTree, node);
if ( n ) { // existing node
return n;
} else {
NumFlows++;
return NULL;
}
} // End of Lookup_FlowTree
void Remove_Node(struct FlowNode *node) {
#ifdef DEVEL
assert(node->memflag == NODE_IN_USE);
if ( NumFlows == 0 ) {
LogError("Remove_Node() Fatal Tried to remove a Node from empty tree");
return;
}
#endif
RB_REMOVE(FlowTree, FlowTree, node);
Free_Node(node);
NumFlows--;
} // End of Lookup_FlowTree
uint32_t Flush_FlowTree(FlowSource_t *fs) {
struct FlowNode *node, *nxt;
uint32_t n = NumFlows;
// Dump all incomplete flows to the file
for (node = RB_MIN(FlowTree, FlowTree); node != NULL; node = nxt) {
StorePcapFlow(fs, node);
nxt = RB_NEXT(FlowTree, FlowTree, node);
#ifdef DEVEL
if ( node->left || node->right ) {
assert(node->proto == 17);
node->left = node->right = NULL;
}
#endif
Remove_Node(node);
}
#ifdef DEVEL
if ( NumFlows != 0 )
LogError("### Flush_FlowTree() remaining flows: %u\n", NumFlows);
#endif
UDP_list.list = NULL;
UDP_list.tail = NULL;
UDP_list.size = 0;
return n;
} // End of Flush_FlowTree
void UDPexpire(FlowSource_t *fs, time_t t_expire) {
struct FlowNode *node;
uint32_t num = 0;
node = UDP_list.list;
while ( node && (node->t_last.tv_sec < t_expire) ) {
struct FlowNode *n = node;
node = node->right;
DisconnectFlowNode(&UDP_list, n);
StorePcapFlow(fs, n);
Remove_Node(n);
num++;
}
dbg_printf("UDP expired %u flows - left %u\n", num, UDP_list.size);
} // End of UDPexpire
void AppendUDPNode(struct FlowNode *node) {
AppendFlowNode(&UDP_list, node);
} // End of AppendUDPNode
static void AppendFlowNode(Linked_list_t *LinkedList, struct FlowNode *node) {
#ifdef DEVEL
if ( LinkedList->tail )
assert(LinkedList->tail->right == NULL);
assert(node->memflag == NODE_IN_USE);
assert(node->left == NULL);
assert(node->right == NULL);
#endif
if ( LinkedList->list == NULL ) {
dbg_printf("AppendFlowNode(): First node\n");
node->left = NULL;
node->right = NULL;
LinkedList->list = node;
LinkedList->tail = node;
LinkedList->size++;
} else {
// new node
dbg_printf("AppendFlowNode(): next node: %u\n", LinkedList->size);
LinkedList->tail->right = node;
node->left = LinkedList->tail;
node->right = NULL;
LinkedList->tail = node;
LinkedList->size++;
}
#ifdef DEVEL
assert(LinkedList->tail->right == NULL);
#endif
} // End of AppendFlowNode
static void DisconnectFlowNode(Linked_list_t *LinkedList, struct FlowNode *node) {
if ( node == NULL )
return;
else {
// disconnect node
struct FlowNode *prev = node->left;
struct FlowNode *next = node->right;
if ( prev )
prev->right = next;
else
LinkedList->list = next;
if ( next )
next->left = prev;
if ( LinkedList->tail == node )
LinkedList->tail = node->left;
node->left = NULL;
node->right = NULL;
LinkedList->size--;
}
} // End of DisconnectFlowNode
void TouchUDPNode(struct FlowNode *node) {
TouchFlowNode(&UDP_list, node);
} // End of TouchUDPNode
static void TouchFlowNode(Linked_list_t *LinkedList, struct FlowNode *node) {
dbg_printf("In TochFlowNode()\n");
if ( LinkedList->list == NULL ) {
// should never happen
LogError("TouchFlowNode() error in %s line %d: %s\n", __FILE__, __LINE__, "Tried to touch node in empty list" );
return;
}
if ( LinkedList->tail == node ) {
// nothing to do
dbg_printf("TochFlowNode() - last node - nothing to do\n");
return;
}
if ( node->left == NULL ) {
// first node - disconnect node
dbg_printf("TochFlowNode() - touch first node\n");
LinkedList->list = node->right;
LinkedList->list->left = NULL;
} else {
dbg_printf("TochFlowNode() - touch middle node\n");
(node->right)->left = node->left;
(node->left)->right = node->right;
}
// append node
LinkedList->tail->right = node;
node->left = LinkedList->tail;
node->right = NULL;
LinkedList->tail = node;
} // End of TouchFlowNode
int AddNodeData(struct FlowNode *node, uint32_t seq, void *payload, uint32_t size) {
return 0;
} // End of AddNodeData
/* Node list functions */
NodeList_t *NewNodeList(void) {
NodeList_t *NodeList;
NodeList = (NodeList_t *)malloc(sizeof(NodeList_t));
if ( !NodeList ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
NodeList->list = NULL;
NodeList->last = NULL;
NodeList->length = 0;
pthread_mutex_init(&NodeList->m_list, NULL);
pthread_cond_init(&NodeList->c_list, NULL);
return NodeList;
} // End of NewNodeList
void DisposeNodeList(NodeList_t *NodeList) {
if ( !NodeList )
return;
if ( NodeList->length ) {
LogError("Try to free non empty NodeList");
return;
}
free(NodeList);
} // End of DisposeNodeList
#ifdef DEVEL
void ListCheck(NodeList_t *NodeList) {
uint32_t len = 0, mem = 0, proto;
static uint32_t loops = 0;
struct FlowNode *node, *n;
// DumpList(NodeList);
loops++;
node = NodeList->list;
while (node) {
len++;
if ( node == NodeList->last ) {
mem = len;
}
if ( node->memflag != NODE_IN_USE ) {
printf("mem flag error : len: %u, last: %u, Nodelist: %u, loops: %u\n", len, mem, NodeList->length, loops);
}
if ( node->right == NULL ) {
proto = node->proto;
n = node;
}
node=node->right;
}
if ( len != NodeList->length) {
printf("Len miss match: len: %u, last: %u, proto: %u, Nodelist: %u, loops: %u, allocated: %u, overlow: %u node: %llx\n", len, mem, proto, NodeList->length, loops, Allocated, CacheOverflow, (long long unsigned)n);
assert(len==NodeList->length);
} else {
printf("Len: %u ok last: %u, proto: %u in loop %u, allocated: %u, overlow: %u\n",
len, mem, proto, loops, Allocated, CacheOverflow);
}
}
#endif
void Push_Node(NodeList_t *NodeList, struct FlowNode *node) {
pthread_mutex_lock(&NodeList->m_list);
if ( NodeList->length == 0 ) {
// empty list
NodeList->list = node;
node->left = NULL;
node->right = NULL;
} else {
NodeList->last->right = node;
node->left = NodeList->last;
node->right = NULL;
}
NodeList->last = node;
NodeList->length++;
#ifdef DEVEL
int proto = node->proto;
printf("pushed node 0x%llx proto: %u, length: %u first: %llx, last: %llx\n",
(unsigned long long)node, proto, NodeList->length, (unsigned long long)NodeList->list, (unsigned long long)NodeList->last);
ListCheck(NodeList);
#endif
pthread_mutex_unlock(&NodeList->m_list);
pthread_cond_signal(&NodeList->c_list);
} // End of Push_Node
struct FlowNode *Pop_Node(NodeList_t *NodeList, int *done) {
struct FlowNode *node;
int proto;
pthread_mutex_lock(&NodeList->m_list);
while ( NodeList->length == 0 && !*done )
pthread_cond_wait(&NodeList->c_list, &NodeList->m_list);
if ( NodeList->length == 0 && *done ) {
pthread_mutex_unlock(&NodeList->m_list);
dbg_printf("Pop_Node done\n");
return NULL;
}
if ( NodeList->list == NULL ) {
// should never happen - list is supposed to have at least one item
pthread_mutex_unlock(&NodeList->m_list);
LogError("Unexpected empty FlowNode_ProcessList");
return NULL;
}
node = NodeList->list;
NodeList->list = node->right;
if ( NodeList->list )
NodeList->list->left = NULL;
else
NodeList->last = NULL;
node->left = NULL;
node->right = NULL;
proto = node->proto;
NodeList->length--;
#ifdef DEVEL
printf("popped node 0x%llx proto: %u, length: %u first: %llx, last: %llx\n",
(unsigned long long)node, proto, NodeList->length, (unsigned long long)NodeList->list, (unsigned long long)NodeList->last);
ListCheck(NodeList);
#endif
pthread_mutex_unlock(&NodeList->m_list);
return node;
} // End of Pop_Node
void DumpList(NodeList_t *NodeList) {
struct FlowNode *node;
printf("FlowNode_ProcessList: 0x%llx, length: %u\n",
(unsigned long long)NodeList->list, NodeList->length);
node = NodeList->list;
while ( node ) {
printf("node: 0x%llx\n", (unsigned long long)node);
printf(" ->left: 0x%llx\n", (unsigned long long)node->left);
printf(" ->right: 0x%llx\n", (unsigned long long)node->right);
node = node->right;
}
printf("tail: 0x%llx\n\n", (unsigned long long)NodeList->last);
} // End of DumpList
void DumpNodeStat(void) {
LogInfo("Nodes in use: %u, Flows: %u CacheOverflow: %u", Allocated, NumFlows, CacheOverflow);
} // End of NodesAllocated
/*
int main(int argc, char **argv) {
return 0;
} // End of main
*/

149
bin/flowtree.h Normal file
View File

@ -0,0 +1,149 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2011, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author$
*
* $Id$
*
* $LastChangedRevision$
*
*/
#include "rbtree.h"
#define v4 ip_union._v4
#define v6 ip_union._v6
struct FlowNode {
// tree
RB_ENTRY(FlowNode) entry;
// linked list
struct FlowNode *left;
struct FlowNode *right;
struct FlowNode *biflow;
// flow key
// IP addr
ip_addr_t src_addr;
ip_addr_t dst_addr;
uint16_t src_port;
uint16_t dst_port;
uint8_t proto;
uint8_t version;
uint16_t _ENDKEY_;
// End of flow key
ip_addr_t tun_src_addr;
ip_addr_t tun_dst_addr;
uint8_t tun_proto;
#define NODE_FREE 0xA5
#define NODE_IN_USE 0x5A
uint16_t memflag; // internal houskeeping flag
uint8_t flags;
#define FIN_NODE 1
#define SIGNAL_NODE 255
uint8_t fin; // double use: 1: fin received - flow can be exported, if complete
// 255: empty node - used to wake up flow thread priodically on quite lines
// flow stat data
struct timeval t_first;
struct timeval t_last;
uint32_t packets; // summed up number of packets
uint32_t bytes; // summed up number of bytes
// flow payload
#define DATABLOCKSIZE 256
uint32_t DataSize; // max size of data buffer
void *data; // start of data buffer
// uint32_t eodata; // offset last byte in buffer
};
typedef struct NodeList_s {
struct FlowNode *list;
struct FlowNode *last;
pthread_mutex_t m_list;
pthread_cond_t c_list;
uint32_t length;
} NodeList_t;
/* flow tree type */
typedef RB_HEAD(FlowTree, FlowNode) FlowTree_t;
// Insert the RB prototypes here
RB_PROTOTYPE(FlowTree, FlowNode, entry, FlowNodeCMP);
int Init_FlowTree(uint32_t CacheSize);
void Dispose_FlowTree(void);
uint32_t Flush_FlowTree(FlowSource_t *fs);
struct FlowNode *Lookup_Node(struct FlowNode *node);
struct FlowNode *New_Node(void);
void Free_Node(struct FlowNode *node);
uint32_t CacheCheck(void);
int AddNodeData(struct FlowNode *node, uint32_t seq, void *payload, uint32_t size);
struct FlowNode *Insert_Node(struct FlowNode *node);
void Remove_Node(struct FlowNode *node);
// Node list functions
NodeList_t *NewNodeList(void);
void DisposeNodeList(NodeList_t *NodeList);
void Push_Node(NodeList_t *NodeList, struct FlowNode *node);
struct FlowNode *Pop_Node(NodeList_t *NodeList, int *done);
void DumpList(NodeList_t *NodeList);
// Liked lists
void AppendUDPNode(struct FlowNode *node);
void TouchUDPNode(struct FlowNode *node);
void UDPexpire(FlowSource_t *fs, time_t t_expire);
// Stat functions
void DumpNodeStat(void);

406
bin/ft2nfdump.c Normal file
View File

@ -0,0 +1,406 @@
/*
* All rights reserved.
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* Copyright (c) 2001 Mark Fullmer and The Ohio State University
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* Flow-Tools related code taken from flow-tools-0.67 cretated by Mark Fullmer
*
* $Author: haag $
*
* $Id: ft2nfdump.c 69 2010-09-09 07:17:43Z haag $
*
* $LastChangedRevision: 69 $
*
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/uio.h>
#include <netinet/in.h>
#include <unistd.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "ftlib.h"
#include "nf_common.h"
#include "nffile.h"
#include "nfx.h"
#include "launch.h"
/* Global defines */
#define MAXRECORDS 30
/* Global consts */
extern extension_descriptor_t extension_descriptor[];
const char *nfdump_version = VERSION;
typedef struct v5_block_s {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t dPkts;
uint32_t dOctets;
uint8_t data[4]; // link to next record
} v5_block_t;
/* externals */
extern uint32_t Max_num_extensions;
/* prototypes */
void usage(char *name);
extension_info_t *GenExtensionMap(struct ftio *ftio);
int flows2nfdump(struct ftio *ftio, extension_info_t *extension_info, int extended, uint32_t limitflows);
#define NEED_PACKRECORD
#include "nffile_inline.c"
#undef NEED_PACKRECORD
void usage(char *name) {
printf("usage %s [options] \n"
"-h\t\tthis text you see right here.\n"
"-E\t\tDump records in ASCII extended format to stdout.\n"
"-c\t\tLimit number of records to convert.\n"
"-m\t\tPrint the extension map and exit.\n"
"-V\t\tPrint version and exit.\n"
"-r\t\tread input from file\n"
"Convert flow-tools format to nfdump format:\n"
"ft2nfdump -r <flow-tools-data-file> | nfdump -z -w <nfdump-file>\n"
, name);
} // End of usage
extension_info_t *GenExtensionMap(struct ftio *ftio) {
extension_info_t *extension_info;
int i;
if (ftio_check_xfield(ftio, FT_XFIELD_DPKTS |
FT_XFIELD_DOCTETS | FT_XFIELD_FIRST | FT_XFIELD_LAST |
FT_XFIELD_SRCADDR | FT_XFIELD_DSTADDR |
FT_XFIELD_SRCPORT | FT_XFIELD_DSTPORT |
FT_XFIELD_UNIX_SECS | FT_XFIELD_UNIX_NSECS | FT_XFIELD_SYSUPTIME |
FT_XFIELD_TOS | FT_XFIELD_TCP_FLAGS | FT_XFIELD_PROT)) {
fprintf(stderr,"Flow-tools record missing required fields.");
return NULL;
}
InitExtensionMaps(NO_EXTENSION_LIST);
extension_info = (extension_info_t *)malloc(sizeof(extension_info_t));
if ( !extension_info ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return NULL;
}
memset((void *)extension_info, 0, sizeof(extension_info_t));
extension_info->map = (extension_map_t *)malloc(sizeof(extension_map_t) + Max_num_extensions * sizeof(uint16_t));
if ( !extension_info->map ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return NULL;
}
i = 0;
extension_info->map->type = ExtensionMapType;
extension_info->map->map_id = 0;
if ( !ftio_check_xfield(ftio, FT_XFIELD_INPUT | FT_XFIELD_OUTPUT )) {
extension_info->map->ex_id[i++] = EX_IO_SNMP_2;
}
if (!ftio_check_xfield(ftio, FT_XFIELD_SRC_AS | FT_XFIELD_DST_AS)) {
extension_info->map->ex_id[i++] = EX_AS_2;
}
if (!ftio_check_xfield(ftio, FT_XFIELD_SRC_MASK | FT_XFIELD_DST_MASK)) {
extension_info->map->ex_id[i++] = EX_MULIPLE;
}
if (!ftio_check_xfield(ftio, FT_XFIELD_NEXTHOP )) {
extension_info->map->ex_id[i++] = EX_NEXT_HOP_v4;
}
if (!ftio_check_xfield(ftio, FT_XFIELD_EXADDR )) {
extension_info->map->ex_id[i++] = EX_ROUTER_IP_v4;
}
if (!ftio_check_xfield(ftio, FT_XFIELD_ENGINE_TYPE )) {
extension_info->map->ex_id[i++] = EX_ROUTER_ID;
}
extension_info->map->ex_id[i++] = 0;
extension_info->map->size = sizeof(extension_map_t) + i * sizeof(uint16_t);
// align 32bits
if (( extension_info->map->size & 0x3 ) != 0 ) {
extension_info->map->size += 4 - ( extension_info->map->size & 0x3 );
}
extension_info->map->extension_size = 0;
i=0;
while (extension_info->map->ex_id[i]) {
int id = extension_info->map->ex_id[i];
extension_info->map->extension_size += extension_descriptor[id].size;
i++;
}
return extension_info;
} // End of GenExtensionMap
int flows2nfdump(struct ftio *ftio, extension_info_t *extension_info, int extended, uint32_t limitflows) {
// required flow tools variables
struct fttime ftt;
struct fts3rec_offsets fo;
struct ftver ftv;
char *rec;
// nfdump variables
nffile_t *nffile;
master_record_t record;
char *s;
uint32_t cnt;
s = "flow-tools";
nffile = OpenNewFile( "-", NULL, 0, 0, s);
if ( !nffile ) {
fprintf(stderr, "%s\n", s);
return 1;
}
AppendToBuffer(nffile, (void *)extension_info->map, extension_info->map->size);
ftio_get_ver(ftio, &ftv);
fts3rec_compute_offsets(&fo, &ftv);
memset((void *)&record, 0, sizeof(record));
record.map_ref = extension_info->map;
record.type = CommonRecordType;
record.exporter_sysid = 0;
// only v4 addresses
ClearFlag(record.flags, FLAG_IPV6_ADDR);
cnt = 0;
while ((rec = ftio_read(ftio))) {
uint32_t when, unix_secs, unix_nsecs, sysUpTime;
int i, id;
unix_secs = *((uint32_t*)(rec+fo.unix_secs));
unix_nsecs = *((uint32_t*)(rec+fo.unix_nsecs));
sysUpTime = *((uint32_t*)(rec+fo.sysUpTime));
when = *((uint32_t*)(rec+fo.First));
ftt = ftltime(sysUpTime, unix_secs, unix_nsecs, when);
record.first = ftt.secs;
record.msec_first = ftt.msecs;
when = *((uint32_t*)(rec+fo.Last));
ftt = ftltime(sysUpTime, unix_secs, unix_nsecs, when);
record.last = ftt.secs;
record.msec_last = ftt.msecs;
record.v4.srcaddr = *((uint32_t*)(rec+fo.srcaddr));
record.v4.dstaddr = *((uint32_t*)(rec+fo.dstaddr));
record.srcport = *((uint16_t*)(rec+fo.srcport));
record.dstport = *((uint16_t*)(rec+fo.dstport));
record.prot = *((uint8_t*)(rec+fo.prot));
record.tcp_flags = *((uint8_t*)(rec+fo.tcp_flags));
record.tos = *((uint8_t*)(rec+fo.tos));
record.dOctets = *((uint32_t*)(rec+fo.dOctets));
record.dPkts = *((uint32_t*)(rec+fo.dPkts));
i = 0;
while ( (id = extension_info->map->ex_id[i]) != 0 ) {
switch (id) {
case EX_IO_SNMP_2:
record.input = *((uint16_t*)(rec+fo.input));
record.output = *((uint16_t*)(rec+fo.output));
break;
case EX_AS_2:
record.srcas = *((uint16_t*)(rec+fo.src_as));
record.dstas = *((uint16_t*)(rec+fo.dst_as));
break;
case EX_MULIPLE:
record.src_mask = *((uint8_t*)(rec+fo.src_mask));
record.dst_mask = *((uint8_t*)(rec+fo.dst_mask));
record.dir = 0;
record.dst_tos = 0;
break;
case EX_ROUTER_IP_v4:
record.ip_nexthop.v4 = *((uint32_t*)(rec+fo.peer_nexthop));
break;
case EX_NEXT_HOP_v4:
record.ip_router.v4 = *((uint32_t*)(rec+fo.router_sc));
break;
case EX_ROUTER_ID:
record.engine_type = *((uint8_t*)(rec+fo.engine_type));
record.engine_id = *((uint8_t*)(rec+fo.engine_id));
break;
// default: Other extensions can not be sent with v5
}
i++;
}
PackRecord(&record, nffile);
if ( extended ) {
char *string;
format_file_block_record(&record, &string, 0);
fprintf(stderr, "%s\n", string);
}
cnt++;
if ( cnt == limitflows )
break;
} /* while */
// write the last records in buffer
if ( nffile->block_header->NumRecords ) {
if ( WriteBlock(nffile) <= 0 ) {
fprintf(stderr, "Failed to write output buffer: '%s'" , strerror(errno));
}
}
free((void *)extension_info->map);
free((void *)extension_info);
DisposeFile(nffile);
return 0;
} // End of flows2nfdump
int main(int argc, char **argv) {
struct ftio ftio;
extension_info_t *extension_info;
struct stat statbuf;
uint32_t limitflows;
int i, extended, printmap, ret, fd;
char *ftfile;
/* init fterr */
fterr_setid(argv[0]);
extended = 0;
printmap = 0;
limitflows = 0;
ftfile = NULL;
while ((i = getopt(argc, argv, "EVc:hmr:?")) != -1)
switch (i) {
case 'h': /* help */
case '?':
usage(argv[0]);
exit (0);
break;
case 'V':
printf("%s: Version: %s\n",argv[0], nfdump_version);
exit(0);
break;
case 'E':
extended = 1;
break;
case 'c':
limitflows = atoi(optarg);
if ( !limitflows ) {
fprintf(stderr, "Option -c needs a number > 0\n");
exit(255);
}
break;
case 'm':
printmap = 1;
break;
case 'r':
ftfile = optarg;
if ( (stat(ftfile, &statbuf) < 0 ) || !(statbuf.st_mode & S_IFREG) ) {
fprintf(stderr, "No such file: '%s'\n", ftfile);
exit(255);
}
break;
default:
usage(argv[0]);
exit (1);
break;
} /* switch */
// End while
if (argc - optind)
fterr_errx(1, "Extra arguments starting with %s.", argv[optind]);
if ( ftfile ) {
fd = open(ftfile, O_RDONLY, 0);
if ( fd < 0 ) {
fprintf(stderr, "Can't open file '%s': %s.", ftfile, strerror(errno));
exit(255);
}
} else {
fd = 0;
}
/* read from fd */
if (ftio_init(&ftio, fd, FT_IO_FLAG_READ) < 0)
fterr_errx(1, "ftio_init(): failed");
extension_info = GenExtensionMap(&ftio);
if ( !extension_info )
exit(255);
if ( printmap ) {
PrintExtensionMap(extension_info->map);
exit(255);
}
ret = flows2nfdump(&ftio, extension_info, extended, limitflows);
return ret;
} // End of main

1129
bin/fts_compat.c Executable file
View File

File diff suppressed because it is too large Load Diff

126
bin/fts_compat.h Executable file
View File

@ -0,0 +1,126 @@
/* $Id: fts_compat.h 16 2009-06-19 09:26:19Z haag $ */
/* TNFTPD ORIGINAL: libnetbsd/ftpfts.h */
/* $NetBSD: ftpfts.h,v 1.3 2003/12/15 23:52:02 lukem Exp $ */
/*
* Copyright (c) 1989, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)fts.h 8.3 (Berkeley) 8/14/94
*/
#ifndef _FTS_H_
#define _FTS_H_ 1
typedef struct {
struct _ftsent *fts_cur; /* current node */
struct _ftsent *fts_child; /* linked list of children */
struct _ftsent **fts_array; /* sort array */
dev_t fts_dev; /* starting device # */
char *fts_path; /* path for this descent */
int fts_rfd; /* fd for root */
u_int fts_pathlen; /* sizeof(path) */
u_int fts_nitems; /* elements in the sort array */
int (*fts_compar) /* compare function */
(const struct _ftsent **, const struct _ftsent **);
#define FTS_COMFOLLOW 0x001 /* follow command line symlinks */
#define FTS_LOGICAL 0x002 /* logical walk */
#define FTS_NOCHDIR 0x004 /* don't change directories */
#define FTS_NOSTAT 0x008 /* don't get stat info */
#define FTS_PHYSICAL 0x010 /* physical walk */
#define FTS_SEEDOT 0x020 /* return dot and dot-dot */
#define FTS_XDEV 0x040 /* don't cross devices */
#define FTS_WHITEOUT 0x080 /* return whiteout information */
#define FTS_OPTIONMASK 0x0ff /* valid user option mask */
#define FTS_NAMEONLY 0x100 /* (private) child names only */
#define FTS_STOP 0x200 /* (private) unrecoverable error */
int fts_options; /* fts_open options, global flags */
} FTS;
typedef struct _ftsent {
struct _ftsent *fts_cycle; /* cycle node */
struct _ftsent *fts_parent; /* parent directory */
struct _ftsent *fts_link; /* next file in directory */
long fts_number; /* local numeric value */
void *fts_pointer; /* local address value */
char *fts_accpath; /* access path */
char *fts_path; /* root path */
int fts_errno; /* errno for this node */
int fts_symfd; /* fd for symlink */
u_short fts_pathlen; /* strlen(fts_path) */
u_short fts_namelen; /* strlen(fts_name) */
ino_t fts_ino; /* inode */
dev_t fts_dev; /* device */
nlink_t fts_nlink; /* link count */
#define FTS_ROOTPARENTLEVEL -1
#define FTS_ROOTLEVEL 0
short fts_level; /* depth (-1 to N) */
#define FTS_D 1 /* preorder directory */
#define FTS_DC 2 /* directory that causes cycles */
#define FTS_DEFAULT 3 /* none of the above */
#define FTS_DNR 4 /* unreadable directory */
#define FTS_DOT 5 /* dot or dot-dot */
#define FTS_DP 6 /* postorder directory */
#define FTS_ERR 7 /* error; errno is set */
#define FTS_F 8 /* regular file */
#define FTS_INIT 9 /* initialized only */
#define FTS_NS 10 /* stat(2) failed */
#define FTS_NSOK 11 /* no stat(2) requested */
#define FTS_SL 12 /* symbolic link */
#define FTS_SLNONE 13 /* symbolic link without target */
#define FTS_W 14 /* whiteout object */
u_short fts_info; /* user flags for FTSENT structure */
#define FTS_DONTCHDIR 0x01 /* don't chdir .. to the parent */
#define FTS_SYMFOLLOW 0x02 /* followed a symlink to get here */
#define FTS_ISW 0x04 /* this is a whiteout object */
u_short fts_flags; /* private flags for FTSENT structure */
#define FTS_AGAIN 1 /* read node again */
#define FTS_FOLLOW 2 /* follow symbolic link */
#define FTS_NOINSTR 3 /* no instructions */
#define FTS_SKIP 4 /* discard node */
u_short fts_instr; /* fts_set() instructions */
struct stat *fts_statp; /* stat(2) information */
char fts_name[1]; /* file name */
} FTSENT;
FTSENT *fts_children_compat(FTS *, int);
int fts_close_compat(FTS *);
FTS *fts_open_compat(char * const *, int,
int (*)(const FTSENT **, const FTSENT **));
FTSENT *fts_read_compat(FTS *);
int fts_set_compat(FTS *, FTSENT *, int);
#endif /* !_FTS_H_ */

2204
bin/grammar.y Executable file
View File

File diff suppressed because it is too large Load Diff

1732
bin/grammar.y.mine Executable file
View File

File diff suppressed because it is too large Load Diff

91
bin/heapsort_inline.c Executable file
View File

@ -0,0 +1,91 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: heapsort_inline.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
static void heapSort(SortElement_t *SortElement, uint32_t array_size, int topN);
static inline void siftDown(SortElement_t *SortElement, uint32_t root, uint32_t bottom);
static void heapSort(SortElement_t *SortElement, uint32_t array_size, int topN) {
int32_t i, maxindex;
for(i = array_size - 1; i >= 0; i--)
siftDown(SortElement,array_size,i);
/*
* we are only interested in the first top N => skip sorting the rest
* For topN == 0 -> all flows gets sorted
*/
if ( (topN >= (array_size - 1)) || topN == 0 )
maxindex = 0;
else
maxindex = array_size - 1 - topN;
for(i = array_size-1; i > maxindex; i-- ) {
SortElement_t temp = SortElement[0];
SortElement[0] = SortElement[i];
SortElement[i] = temp;
siftDown(SortElement,i,0);
}
} // End of heapSort
static inline void siftDown(SortElement_t *SortElement, uint32_t numbersSize, uint32_t node) {
uint32_t i, parent, child;
parent = node;
i = parent + 1;
while( i != parent ) {
i = parent;
// Compare with left child node
child = 2*i+1;
if( (child) < numbersSize && SortElement[child].count > SortElement[parent].count)
parent = child;
// Compare with right child node
child = 2*i+2;
if( (child) < numbersSize && SortElement[child].count > SortElement[parent].count)
parent = child;
if ( i != parent ) {
SortElement_t temp = SortElement[i];
SortElement[i] = SortElement[parent];
SortElement[parent] = temp;
}
}
} // End of siftDown

334
bin/inline.c Normal file
View File

@ -0,0 +1,334 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: inline.c 40 2009-12-16 10:41:44Z haag $
*
* $LastChangedRevision: 40 $
*
*/
static inline uint16_t Get_val16(void *p);
static inline uint32_t Get_val24(void *p);
static inline uint32_t Get_val32(void *p);
static inline uint64_t Get_val40(void *p);
static inline uint64_t Get_val48(void *p);
static inline uint64_t Get_val56(void *p);
static inline uint64_t Get_val64(void *p);
static inline void Put_val16(uint16_t v, void *p);
static inline void Put_val24(uint32_t v, void *p);
static inline void Put_val32(uint32_t v, void *p);
// static inline void Put_val40(uint64_t v, void *p);
static inline void Put_val48(uint64_t v, void *p);
// static inline void Put_val56(uint64_t v, void *p);
static inline void Put_val64(uint64_t v, void *p);
static inline uint16_t Get_val16(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = in[0];
mask.val.val8[1] = in[1];
#else
mask.val.val8[0] = in[1];
mask.val.val8[1] = in[0];
#endif
return mask.val.val16[0];
} // End of Get_val16
static inline uint32_t Get_val24(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = 0;
mask.val.val8[1] = in[0];
mask.val.val8[2] = in[1];
mask.val.val8[3] = in[2];
#else
mask.val.val8[0] = in[2];
mask.val.val8[1] = in[1];
mask.val.val8[2] = in[0];
mask.val.val8[3] = 0;
#endif
return mask.val.val32[0];
} // End of Get_val24
static inline uint32_t Get_val32(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = in[0];
mask.val.val8[1] = in[1];
mask.val.val8[2] = in[2];
mask.val.val8[3] = in[3];
#else
mask.val.val8[0] = in[3];
mask.val.val8[1] = in[2];
mask.val.val8[2] = in[1];
mask.val.val8[3] = in[0];
#endif
return mask.val.val32[0];
} // End of Get_val32
static inline uint64_t Get_val40(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = 0;
mask.val.val8[1] = 0;
mask.val.val8[2] = 0;
mask.val.val8[3] = in[0];
mask.val.val8[4] = in[1];
mask.val.val8[5] = in[2];
mask.val.val8[6] = in[3];
mask.val.val8[7] = in[4];
#else
mask.val.val8[0] = in[4];
mask.val.val8[1] = in[3];
mask.val.val8[2] = in[2];
mask.val.val8[3] = in[1];
mask.val.val8[4] = in[0];
mask.val.val8[5] = 0;
mask.val.val8[6] = 0;
mask.val.val8[7] = 0;
#endif
return mask.val.val64;
} // End of Get_val40
static inline uint64_t Get_val48(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = 0;
mask.val.val8[1] = 0;
mask.val.val8[2] = in[0];
mask.val.val8[3] = in[1];
mask.val.val8[4] = in[2];
mask.val.val8[5] = in[3];
mask.val.val8[6] = in[4];
mask.val.val8[7] = in[5];
#else
mask.val.val8[0] = in[5];
mask.val.val8[1] = in[4];
mask.val.val8[2] = in[3];
mask.val.val8[3] = in[2];
mask.val.val8[4] = in[1];
mask.val.val8[5] = in[0];
mask.val.val8[6] = 0;
mask.val.val8[7] = 0;
#endif
return mask.val.val64;
} // End of Get_val48
static inline uint64_t Get_val56(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = 0;
mask.val.val8[1] = in[0];
mask.val.val8[2] = in[1];
mask.val.val8[3] = in[2];
mask.val.val8[4] = in[3];
mask.val.val8[5] = in[4];
mask.val.val8[6] = in[5];
mask.val.val8[7] = in[6];
#else
mask.val.val8[0] = in[6];
mask.val.val8[1] = in[5];
mask.val.val8[2] = in[4];
mask.val.val8[3] = in[3];
mask.val.val8[4] = in[2];
mask.val.val8[5] = in[1];
mask.val.val8[6] = in[0];
mask.val.val8[7] = 0;
#endif
return mask.val.val64;
} // End of Get_val56
static inline uint64_t Get_val64(void *p) {
uint8_t *in = (uint8_t *)p;
type_mask_t mask;
#ifdef WORDS_BIGENDIAN
mask.val.val8[0] = in[0];
mask.val.val8[1] = in[1];
mask.val.val8[2] = in[2];
mask.val.val8[3] = in[3];
mask.val.val8[4] = in[4];
mask.val.val8[5] = in[5];
mask.val.val8[6] = in[6];
mask.val.val8[7] = in[7];
#else
mask.val.val8[0] = in[7];
mask.val.val8[1] = in[6];
mask.val.val8[2] = in[5];
mask.val.val8[3] = in[4];
mask.val.val8[4] = in[3];
mask.val.val8[5] = in[2];
mask.val.val8[6] = in[1];
mask.val.val8[7] = in[0];
#endif
return mask.val.val64;
} // End of Get_val64
static inline void Put_val16(uint16_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val16[0] = v;
out[0] = mask.val.val8[0];
out[1] = mask.val.val8[1];
} // End of Put_val16
static inline void Put_val24(uint32_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val32[0] = v;
out[0] = mask.val.val8[1];
out[1] = mask.val.val8[2];
out[2] = mask.val.val8[3];
} // End of Put_val24
static inline void Put_val32(uint32_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val32[0] = v;
out[0] = mask.val.val8[0];
out[1] = mask.val.val8[1];
out[2] = mask.val.val8[2];
out[3] = mask.val.val8[3];
} // End of Put_val32
/*
* not yet used
*
static inline void Put_val40(uint64_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val64 = v;
out[0] = mask.val.val8[3];
out[1] = mask.val.val8[4];
out[2] = mask.val.val8[5];
out[3] = mask.val.val8[6];
out[4] = mask.val.val8[7];
} // End of Put_val40
*
*/
static inline void Put_val48(uint64_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val64 = v;
out[0] = mask.val.val8[2];
out[1] = mask.val.val8[3];
out[2] = mask.val.val8[4];
out[3] = mask.val.val8[5];
out[4] = mask.val.val8[6];
out[5] = mask.val.val8[7];
} // End of Put_val48
/*
* not yet used
*
static inline void Put_val56(uint64_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val64 = v;
out[0] = mask.val.val8[1];
out[1] = mask.val.val8[2];
out[2] = mask.val.val8[3];
out[3] = mask.val.val8[4];
out[4] = mask.val.val8[5];
out[5] = mask.val.val8[6];
out[6] = mask.val.val8[7];
} // End of Put_val56
*
*/
static inline void Put_val64(uint64_t v, void *p) {
uint8_t *out = (uint8_t *)p;
type_mask_t mask;
mask.val.val64 = v;
out[0] = mask.val.val8[0];
out[1] = mask.val.val8[1];
out[2] = mask.val.val8[2];
out[3] = mask.val.val8[3];
out[4] = mask.val.val8[4];
out[5] = mask.val.val8[5];
out[6] = mask.val.val8[6];
out[7] = mask.val.val8[7];
} // End of Put_val64

361
bin/ipconv.c Normal file
View File

@ -0,0 +1,361 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2008-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* Copyright (c) 1996 by Internet Software Consortium.
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND INTERNET SOFTWARE CONSORTIUM DISCLAIMS
* ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL INTERNET SOFTWARE
* CONSORTIUM BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
* DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
* PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
* ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
* SOFTWARE.
*
*
* $Author: haag $
*
* $Id: ipconv.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <arpa/nameser.h>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#ifdef HAVE_RESOLV_H
#include <resolv.h>
#endif
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "util.h"
#include "ipconv.h"
static int parse_ipv4(const char *src, uint32_t *dst, int *bytes);
static int parse_ipv6(const char *src, uint64_t *dst, int *bytes);
static int lookup_host(const char *hostname, uint64_t *iplist, uint32_t *num_ip );
int parse_ip(int *af, const char *src, uint64_t *dst, int *bytes, int lookup, uint32_t *num_ip ) {
char *alpha = "abcdefghijklmnopqrstuvwxzyABCDEFGHIJKLMNOPQRSTUVWXZY";
uint32_t v4addr;
int ret;
// check for IPv6 address
if ( strchr(src, ':') != NULL ) {
*af = PF_INET6;
// check for alpha chars -> hostname -> lookup
} else if ( strpbrk(src, alpha)) {
*af = 0;
if ( lookup == STRICT_IP )
return -1;
else
return lookup_host(src, dst, num_ip );
// it's IPv4
} else
*af = PF_INET;
*num_ip = 1;
switch (*af) {
case AF_INET:
ret = (parse_ipv4(src, &v4addr, bytes));
dst[0] = 0;
dst[1] = ntohl(v4addr) & 0xffffffffLL ;
return ret;
break;
case AF_INET6:
ret = (parse_ipv6(src, dst, bytes));
dst[0] = ntohll(dst[0]);
dst[1] = ntohll(dst[1]);
return ret;
break;
}
/* NOTREACHED */
return 0;
}
static int parse_ipv4(const char *src, uint32_t *dst, int *bytes) {
static const char digits[] = "0123456789";
int saw_digit, ch;
uint8_t tmp[4], *tp;
saw_digit = 0;
*bytes = 0;
*(tp = tmp) = 0;
memset(tmp, 0, sizeof(tmp));
while ((ch = *src++) != '\0') {
const char *pch;
if ((pch = strchr(digits, ch)) != NULL) {
unsigned int new = *tp * 10 + (pch - digits);
if (new > 255)
return (0);
if (! saw_digit) {
if (++(*bytes) > 4)
return (0);
saw_digit = 1;
}
*tp = new;
} else if (ch == '.' && saw_digit) {
if (*bytes == 4)
return (0);
*++tp = 0;
saw_digit = 0;
if ( !(*src) )
return 0;
} else
return (0);
}
memcpy(dst, tmp, sizeof(tmp));
return (1);
}
static int parse_ipv6(const char *src, uint64_t *dst, int *bytes) {
static const char xdigits_l[] = "0123456789abcdef",
xdigits_u[] = "0123456789ABCDEF";
uint8_t tmp[16], *tp, *endp, *colonp;
const char *xdigits, *curtok;
int ch, saw_xdigit;
u_int val;
memset((tp = tmp), '\0', sizeof(tmp));
endp = tp + sizeof(tmp);
colonp = NULL;
/* Leading :: requires some special handling. */
if (*src == ':')
if (*++src != ':')
return (0);
curtok = src;
saw_xdigit = 0;
val = 0;
while ((ch = *src++) != '\0') {
const char *pch;
if ((pch = strchr((xdigits = xdigits_l), ch)) == NULL)
pch = strchr((xdigits = xdigits_u), ch);
if (pch != NULL) {
val <<= 4;
val |= (pch - xdigits);
if (val > 0xffff)
return (0);
saw_xdigit = 1;
continue;
}
if (ch == ':') {
curtok = src;
if (!saw_xdigit) {
if (colonp)
return (0);
colonp = tp;
continue;
} else if (*src == '\0') {
return (0);
}
if (tp + sizeof(uint16_t) > endp)
return (0);
*tp++ = (u_char) (val >> 8) & 0xff;
*tp++ = (u_char) val & 0xff;
saw_xdigit = 0;
val = 0;
continue;
}
if (ch == '.' && ((tp + 4) <= endp) &&
parse_ipv4(curtok, (uint32_t *)tp, bytes) > 0) {
tp += 4;
saw_xdigit = 0;
break; /* '\0' was seen by parse_ipv4(). */
}
return (0);
}
if (saw_xdigit) {
if (tp + sizeof(uint16_t) > endp)
return (0);
*tp++ = (u_char) (val >> 8) & 0xff;
*tp++ = (u_char) val & 0xff;
}
if (colonp != NULL) {
/*
* Since some memmove()'s erroneously fail to handle
* overlapping regions, we'll do the shift by hand.
*/
const int n = tp - colonp;
int i;
for (i = 1; i <= n; i++) {
endp[- i] = colonp[n - i];
colonp[n - i] = 0;
}
tp = endp;
}
*bytes = 16 - ( endp - tp );
memcpy(dst, tmp, sizeof(tmp));
return (1);
}
static int lookup_host(const char *hostname, uint64_t *iplist, uint32_t *num_ip ) {
struct addrinfo hints, *res, *r;
int errcode, i, len;
char addrstr[128];
char reverse[256];
void *ptr;
printf("Resolving %s ...\n", hostname);
memset (&hints, 0, sizeof (hints));
hints.ai_family = PF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags |= AI_CANONNAME;
errcode = getaddrinfo (hostname, NULL, &hints, &res);
if (errcode != 0) {
fprintf(stderr, "Failed to resolve IP address for %s: %s\n", hostname, gai_strerror(errno));
return 0;
}
// count the number of records found
*num_ip = 0;
// remember res for later free()
r = res;
i = 0;
while (res) {
if ( *num_ip >= MAXHOSTS ) {
printf ("Too man IP addresses in DNS response\n");
return 1;
}
switch (res->ai_family) {
case PF_INET:
ptr = &(((struct sockaddr_in *) res->ai_addr)->sin_addr);
iplist[i++] = 0;
iplist[i++] = ntohl(*(uint32_t *)ptr) & 0xffffffffLL ;
len = sizeof(struct sockaddr_in);
break;
case AF_INET6:
ptr = &((struct sockaddr_in6 *) res->ai_addr)->sin6_addr;
iplist[i++] = ntohll(((uint64_t *)ptr)[0]);
iplist[i++] = ntohll(((uint64_t *)ptr)[1]);
len = sizeof(struct sockaddr_in6);
break;
default: {
// not handled
res = res->ai_next;
continue;
}
}
inet_ntop (res->ai_family, ptr, addrstr, 100);
addrstr[99] = '\0';
if ( (errcode = getnameinfo(res->ai_addr, len, reverse, sizeof(reverse), NULL,0,0)) != 0 ) {
snprintf(reverse, sizeof(reverse)-1, "<reverse lookup failed>");
// fprintf(stderr, "Failed to reverse lookup %s: %s\n", addrstr, gai_strerror(errcode));
}
printf ("IPv%d address: %s (%s)\n", res->ai_family == PF_INET6 ? 6 : 4, addrstr, reverse );
res = res->ai_next;
(*num_ip)++;
}
freeaddrinfo(r);
return 1;
} // End of lookup_host
int set_nameserver(char *ns) {
struct hostent *host;
res_init();
host = gethostbyname(ns);
if (host == NULL) {
(void) fprintf(stderr,"Can not resolv nameserver %s: %s\n", ns, hstrerror(h_errno));
return 0;
}
(void) memcpy((void *)&_res.nsaddr_list[0].sin_addr, (void *)host->h_addr_list[0], (size_t)host->h_length);
_res.nscount = 1;
return 1;
} // End of set_nameserver
/*
int main( int argc, char **argv ) {
char *s, t[64];
uint64_t anyaddr[2];
uint32_t num_ip;
int af, ret, bytes;
s = argv[1];
if (argc == 3 && !set_nameserver(argv[2]) )
return 0;
lookup_host(s, &num_ip);
return 0;
ret = parse_ip(&af, s, anyaddr, &bytes);
if ( ret != 1 ) {
printf("Parse failed!\n");
return 0;
}
if ( af == PF_INET )
inet_ntop(af, &(((uint32_t *)anyaddr)[3]), t, 64);
else
inet_ntop(af, anyaddr, t, 64);
printf("Convert back: %s => %s %i bytes\n", s, t, bytes);
}
*/

52
bin/ipconv.h Executable file
View File

@ -0,0 +1,52 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author:$
*
* $Id:$
*
* $LastChangedRevision:$
*
*
*/
#ifndef _IPCONV_H
#define _IPCONV_H 1
int parse_ip(int *af, const char *src, uint64_t *dst, int *bytes, int lookup, uint32_t *num_ip );
int set_nameserver(char *ns);
#define MAXHOSTS 512
#define STRICT_IP 0
#define ALLOW_LOOKUP 1
#endif //_IPCONV_H

1742
bin/ipfix.c Normal file
View File

File diff suppressed because it is too large Load Diff

268
bin/ipfix.h Normal file
View File

@ -0,0 +1,268 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author:$
*
* $Id:$
*
* $LastChangedRevision:$
*
*/
#ifndef _IPFIX_H
#define _IPFIX_H 1
/* reference: http://tools.ietf.org/html/draft-ietf-ipfix-protocol-rfc5101bis-00 */
typedef struct ipfix_header {
uint16_t Version; // set to 10 for IPFIX
uint16_t Length; // Total length incl. this header. up to 65535 bytes
uint32_t ExportTime; // UNIC epoch export Time of flow.
uint32_t LastSequence; // Incremental sequence counter modulo 2^32 of all IPFIX Data Records
uint32_t ObservationDomain; // identifier , unique to the exporter
} ipfix_header_t;
#define IPFIX_HEADER_LENGTH sizeof(ipfix_header_t)
/*
Message Header Field Descriptions:
Version
Version of Flow Record format exported in this message. The value
of this field is 0x000a for the current version, incrementing by
one the version used in the NetFlow services export version 9
[RFC3954].
Length
Total length of the IPFIX Message, measured in octets, including
Message Header and Set(s).
Export Time
Time at which the IPFIX Message Header leaves the Exporter,
expressed in seconds since the UNIX epoch of 1 January 1970 at
00:00 UTC, encoded as an unsigned 32-bit integer.
Sequence Number
Incremental sequence counter modulo 2^32 of all IPFIX Data Records
sent on this PR-SCTP stream from the current Observation Domain by
the Exporting Process. Check the specific meaning of this field
in the subsections of Section 10 when UDP or TCP is selected as
the transport protocol. This value SHOULD be used by the
Collecting Process to identify whether any IPFIX Data Records have
been missed. Template and Options Template Records do not
increase the Sequence Number.
Observation Domain ID
A 32-bit identifier of the Observation Domain that is locally
unique to the Exporting Process. The Exporting Process uses the
Observation Domain ID to uniquely identify to the Collecting
Process the Observation Domain that metered the Flows. It is
RECOMMENDED that this identifier also be unique per IPFIX Device.
Collecting Processes SHOULD use the Transport Session and the
Observation Domain ID field to separate different export streams
originating from the same Exporter. The Observation Domain ID
SHOULD be 0 when no specific Observation Domain ID is relevant for
the entire IPFIX Message, for example, when exporting the
Exporting Process Statistics, or in case of a hierarchy of
Collectors when aggregated Data Records are exported.
*/
/* set format:
A Set has the format shown in Figure H. The record types can be
either Template Records, Options Template Records, or Data Records.
The record types MUST NOT be mixed within a Set.
+--------------------------------------------------+
| Set Header |
+--------------------------------------------------+
| record |
+--------------------------------------------------+
| record |
+--------------------------------------------------+
...
+--------------------------------------------------+
| record |
+--------------------------------------------------+
| Padding (opt.) |
+--------------------------------------------------+
*/
typedef struct set_header_s {
uint16_t SetID; // SetIDs:
// 0, 1: not used
// 2: Template Set
// 3: Options Template Set
#define IPFIX_TEMPLATE_FLOWSET_ID 2
#define IPFIX_OPTIONS_FLOWSET_ID 3
#define IPFIX_MIN_RECORD_FLOWSET_ID 256
uint16_t Length; // Length of bytes incl. this header
uint32_t records[1]; // pointer to records
} set_header_t;
/* Template Record Format
The format of the Template Record is shown in Figure J. It consists
of a Template Record Header and one or more Field Specifiers. The
definition of the Field Specifiers is given in Figure G above.
+--------------------------------------------------+
| Template Record Header |
+--------------------------------------------------+
| Field Specifier |
+--------------------------------------------------+
| Field Specifier |
+--------------------------------------------------+
...
+--------------------------------------------------+
| Field Specifier |
+--------------------------------------------------+
*/
typedef struct ipfix_template_record_s {
uint16_t TemplateID; // Template ID:
// 0-255 reserved for Template Sets, Options Template Sets,
// and other reserved Sets yet to be created.
// 256-65535 Template IDs of Data Sets
uint16_t FieldCount;
uint32_t elements[1];
} ipfix_template_record_t;
/* Standard Information Elements
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| Information Element ident. | Field Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*/
typedef struct ipfix_template_elements_std_s {
uint16_t Type;
uint16_t Length;
} ipfix_template_elements_std_t;
/* enterprise-specific Information Elements
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| Information Element ident. | Field Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*/
typedef struct ipfix_template_elements_e_s {
uint16_t Type;
uint16_t Length;
uint32_t EnterpriseNumber;
} ipfix_template_elements_e_t;
#define _1byte 1
#define _2bytes 2
#define _3bytes 3
#define _4bytes 4
#define _6bytes 6
#define _8bytes 8
#define _16bytes 16
// IPFIX std element definitios
// Flowset record types
#define IPFIX_octetDeltaCount 1
#define IPFIX_packetDeltaCount 2
// reserved 3
#define IPFIX_FLOWS_AGGR 3
#define IPFIX_protocolIdentifier 4
#define IPFIX_ipClassOfService 5
#define IPFIX_tcpControlBits 6
#define IPFIX_SourceTransportPort 7
#define IPFIX_SourceIPv4Address 8
#define IPFIX_SourceIPv4PrefixLength 9
#define IPFIX_ingressInterface 10
#define IPFIX_DestinationTransportPort 11
#define IPFIX_DestinationIPv4Address 12
#define IPFIX_DestinationIPv4PrefixLength 13
#define IPFIX_egressInterface 14
#define IPFIX_ipNextHopIPv4Address 15
#define IPFIX_bgpSourceAsNumber 16
#define IPFIX_bgpDestinationAsNumber 17
#define IPFIX_bgpNextHopIPv4Address 18
#define IPFIX_flowEndSysUpTime 21
#define IPFIX_flowStartSysUpTime 22
#define IPFIX_postOctetDeltaCount 23
#define IPFIX_postPacketDeltaCount 24
#define IPFIX_SourceIPv6Address 27
#define IPFIX_DestinationIPv6Address 28
#define IPFIX_SourceIPv6PrefixLength 29
#define IPFIX_DestinationIPv6PrefixLength 30
#define IPFIX_flowLabelIPv6 31
#define IPFIX_icmpTypeCodeIPv4 32
// reserved 34, 35
// reserved 38, 39
// reserved 48, 49, 50, 51
// #define IPFIX_MIN_TTL 52
// #define IPFIX_MAX_TTL 53
// #define IPFIX_IPV4_IDENT 54
#define IPFIX_postIpClassOfService 55
#define IPFIX_SourceMacAddress 56
#define IPFIX_postDestinationMacAddress 57
#define IPFIX_vlanId 58
#define IPFIX_postVlanId 59
#define IPFIX_flowDirection 61
#define IPFIX_ipNextHopIPv6Address 62
#define IPFIX_bgpNextHopIPv6Address 63
#define IPFIX_mplsTopLabelStackSection 70
#define IPFIX_mplsLabelStackSection2 71
#define IPFIX_mplsLabelStackSection3 72
#define IPFIX_mplsLabelStackSection4 73
#define IPFIX_mplsLabelStackSection5 74
#define IPFIX_mplsLabelStackSection6 75
#define IPFIX_mplsLabelStackSection7 76
#define IPFIX_mplsLabelStackSection8 77
#define IPFIX_mplsLabelStackSection9 78
#define IPFIX_mplsLabelStackSection10 79
#define IPFIX_DestinationMacAddress 80
#define IPFIX_postSourceMacAddress 81
#define IPFIX_octetTotalCount 85
#define IPFIX_packetTotalCount 86
#define IPFIX_flowStartMilliseconds 152
#define IPFIX_flowEndMilliseconds 153
// reserved 89
/* prototypes */
int Init_IPFIX(void);
void Process_IPFIX(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs);
#endif //_IPFIX_H 1

302
bin/ipfrag.c Normal file
View File

@ -0,0 +1,302 @@
/*
* Copyright (c) 2014, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: phaag $
*
* $Id: ipfrag.c 40874 2014-03-06 09:58:20Z phaag $
*
* $LastChangedRevision: 40874 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <netinet/ip.h>
#include <unistd.h>
#include <stdint.h>
#include "util.h"
#include "rbtree.h"
#include "ipfrag.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#define KEYLEN (offsetof(IPFragNode_t,data_size) - offsetof(IPFragNode_t, src_addr))
static int IPFragNodeCMP(struct IPFragNode *e1, struct IPFragNode *e2);
static struct IPFragNode *New_node(void);
static void Free_node(struct IPFragNode *node, int free_data);
static void Remove_node(struct IPFragNode *node);
// Insert the IP RB tree code here
RB_GENERATE(IPFragTree, IPFragNode, entry, IPFragNodeCMP);
static IPFragTree_t *IPFragTree;
static int IPFragNodeCMP(struct IPFragNode *e1, struct IPFragNode *e2) {
uint32_t *a = &e1->src_addr;
uint32_t *b = &e2->src_addr;
int i;
// 2 x sizeof(uint32_t) (8) + frag_offset == 12
i = memcmp((void *)a, (void *)b, KEYLEN );
return i;
} // End of IPFragNodeCMP
static struct IPFragNode *New_node(void) {
struct IPFragNode *node;
node = malloc(sizeof(struct IPFragNode));
if ( !node ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
memset((void *)node, 0, sizeof(struct IPFragNode));
node->data = malloc(IP_MAXPACKET);
if ( !node->data ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
free(node);
return NULL;
}
memset(node->data, 0, IP_MAXPACKET);
node->eod = node->data;
node->data_size = 0;
node->holes = malloc(sizeof(hole_t));
if ( !node->holes ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
free(node);
return NULL;
}
node->holes->next = NULL;
node->holes->first = 0;
node->holes->last = IP_MAXPACKET;
return node;
} // End of New_node
static void Free_node(struct IPFragNode *node, int free_data) {
hole_t *hole, *h;
hole = node->holes;
while (hole) {
h = hole->next;
free(hole);
hole = h;
}
if ( free_data)
free(node->data);
free(node);
} // End of Free_node
static void Remove_node(struct IPFragNode *node) {
struct IPFragNode *n;
n = RB_REMOVE(IPFragTree, IPFragTree, node);
if ( n ) {
Free_node(n, 0);
} // else - node not in tree
} // End of Remove_node
int IPFragTree_init(void) {
IPFragTree = malloc(sizeof(IPFragTree_t));
if ( !IPFragTree ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return 0;
}
RB_INIT(IPFragTree);
dbg_printf("IPFrag key len: %lu\n", KEYLEN);
return 1;
} // End of IPFragTree_init
void IPFragTree_free(void) {
struct IPFragNode *node, *nxt;
// Dump all incomplete flows to the file
for (node = RB_MIN(IPFragTree, IPFragTree); node != NULL; node = nxt) {
nxt = RB_NEXT(IPFragTree, IPFragTree, node);
RB_REMOVE(IPFragTree, IPFragTree, node);
Free_node(node, 1);
}
free(IPFragTree);
IPFragTree = NULL;
} // End of IPFragTree_free
void *IPFrag_tree_Update(uint32_t src, uint32_t dst, uint32_t ident, uint32_t *length, uint32_t ip_off, void *data) {
struct IPFragNode FindNode, *n;
hole_t *hole, *h, *hole_parent;
uint16_t more_fragments, first, last, max;
int found_hole;
FindNode.src_addr = src;
FindNode.dst_addr = dst;
FindNode.ident = ident;
n = RB_FIND(IPFragTree, IPFragTree, &FindNode);
if ( !n ) {
n = New_node();
n->src_addr = src;
n->dst_addr = dst;
n->ident = ident;
if ( RB_INSERT(IPFragTree, IPFragTree, n) ) {
// must never happen
LogError("Node insert returned existing node - Software error in %s line %d", __FILE__, __LINE__);
}
}
hole = n->holes;
hole_parent = NULL;
first = (ip_off & IP_OFFMASK) << 3;
more_fragments = (ip_off & IP_MF) != 0 ? 1 : 0;
last = first + *length - 1;
if ( last > IP_MAXPACKET ) {
LogError("Fragment assembly error: last > IP_MAXPACKET");
LogError("Fraget assembly: first: %u, last: %u, MF: %u\n", first, last, more_fragments);
return NULL;
}
// last fragment - sets max offset
found_hole = 0;
max = more_fragments == 0 ? last : 0;
dbg_printf("Fraget assembly: first: %u, last: %u, MF: %u\n", first, last, more_fragments);
while (hole) {
uint16_t hole_last;
if ( max ) {
dbg_printf("max in last fragment: %u\n", max);
// last fragment offset/length
if ( hole->last == IP_MAXPACKET ) {
// last fragment has max size
if ( max > hole->first ) {
dbg_printf("set max of last fragment: %u\n", max);
hole->last = max;
} else {
LogError("last fragment offset error - teardrop attack??");
}
} else {
// last fragment must always be max offset
if ( max < hole->last ) {
LogError("last fragment offset error - teardrop attack??");
}
}
}
dbg_printf("Check Hole: first: %u, last: %u\n", hole->first, hole->last);
if ( first > hole->last ) {
hole_parent = hole;
hole = hole->next;
dbg_printf("Fragment right outside hole\n");
continue;
}
if ( last < hole->first ) {
hole_parent = hole;
hole = hole->next;
dbg_printf("Fragment left outside hole\n");
continue;
}
// fragment fits into hole
found_hole = 1;
if ( last > n->data_size )
n->data_size = last;
hole_last = hole->last;
if ( first == hole->first ) {
dbg_printf("Fragment matches first\n");
// fragment fits at beginning of hole
if ( last == hole->last ) {
dbg_printf("Fragment matches last\n");
// fragment fits completly into hole - delete hole
if ( hole_parent ) {
hole_parent->next = hole->next;
} else {
n->holes = NULL;
}
free(hole);
hole = NULL;
} else {
// fragment smaller than hole
dbg_printf("Fragment smaller than hole\n");
hole->first = last+1;
}
} else {
// fragment start within hole
dbg_printf("Fragment inside hole\n");
hole->last = first - 1;
if ( last < hole_last ) {
// fragment ends within hole - add another hole
h = malloc(sizeof(hole_t));
if ( !h ) {
LogError("malloc() error in %s line %d: %s", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
h->first = last + 1;
h->last = hole_last;
h->next = n->holes;
n->holes = h;
}
}
memcpy(n->data + first, data, *length);
break;
}
if ( !found_hole )
LogError("No space - Fragment overlap: first: %u, last: %u\n", first, last);
if ( n->holes == NULL ) {
void *data = n->data;
n->data_size++;
*length = n->data_size;
Remove_node(n);
dbg_printf("Datagramm complete - size: %u\n", n->data_size);
return data;
} else {
return NULL;
}
} // End of IPFrag_tree_Update

74
bin/ipfrag.h Normal file
View File

@ -0,0 +1,74 @@
/*
* Copyright (c) 2014, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: phaag $
*
* $Id: ipfrag.h 40691 2014-03-03 11:24:22Z phaag $
*
* $LastChangedRevision: 40691 $
*
*/
typedef struct hole_s {
struct hole_s *next;
uint32_t first;
uint32_t last;
} hole_t;
struct IPFragNode {
// tree
RB_ENTRY(IPFragNode) entry;
// flow key
// IP addr
uint32_t src_addr;
uint32_t dst_addr;
uint32_t ident;
// End of flow key
uint32_t data_size;
// packet data
void *data;
void *eod;
hole_t *holes;
};
typedef struct IPFragNode IPFragNode_t;
/* flow tree type */
typedef RB_HEAD(IPFragTree, IPFragNode) IPFragTree_t;
// Insert the RB prototypes here
RB_PROTOTYPE(IPFragTree, IPFragNode, entry, IPFragNodeCMP);
int IPFragTree_init(void);
void IPFragTree_free(void);
void *IPFrag_tree_Update(uint32_t src, uint32_t dst, uint32_t ident, uint32_t *length, uint32_t ip_off, void *data);

440
bin/launch.c Normal file
View File

@ -0,0 +1,440 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: launch.c 69 2010-09-09 07:17:43Z haag $
*
* $LastChangedRevision: 69 $
*
*
*/
#include "config.h"
#include <signal.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <sys/socket.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <syslog.h>
#include <stdarg.h>
#include <errno.h>
#include <unistd.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nfstatfile.h"
#include "bookkeeper.h"
#ifdef HAVE_FTS_H
# include <fts.h>
#else
# include "fts_compat.h"
#define fts_children fts_children_compat
#define fts_close fts_close_compat
#define fts_open fts_open_compat
#define fts_read fts_read_compat
#define fts_set fts_set_compat
#endif
#include "expire.h"
#include "nffile.h"
#include "nfxstat.h"
#include "collector.h"
static int done, launch, child_exit;
static void SignalHandler(int signal);
static char *cmd_expand(srecord_t *InfoRecord, char *ident, char *datadir, char *process);
static void cmd_parse(char *buf, char **args);
static void cmd_execute(char **args);
static void do_expire(char *datadir);
#define MAXARGS 256
#define MAXCMDLEN 4096
static void SignalHandler(int signal) {
switch (signal) {
case SIGTERM:
// in case the process will not terminate, we
// kill the process directly after the 2nd TERM signal
if ( done > 1 )
exit(234);
done++;
break;
case SIGHUP:
launch = 1;
break;
case SIGCHLD:
child_exit++;
break;
}
} /* End of IntHandler */
/*
* Expand % placeholders in command string
* expand the memory needed in the command string and replace placeholders
* prevent endless expansion
*/
static char *cmd_expand(srecord_t *InfoRecord, char *ident, char *datadir, char *process) {
char *q, *s, tmp[16];
int i;
q = strdup(process);
if ( !q ) {
perror("Process cmdline");
return NULL;
}
i = 0;
while ( q[i] ) {
if ( (q[i] == '%') && q[i+1] ) {
// replace the %x var
switch ( q[i+1] ) {
case 'd' :
s = datadir;
break;
case 'f' :
s = InfoRecord->fname;
break;
case 't' :
s = InfoRecord->tstring;
break;
case 'u' :
#if defined __OpenBSD__ || defined __FreeBSD__
snprintf(tmp, 16, "%i", InfoRecord->tstamp);
#else
snprintf(tmp, 16, "%li", InfoRecord->tstamp);
#endif
tmp[15] = 0;
s = tmp;
break;
case 'i' :
s = ident;
break;
default:
syslog(LOG_ERR, "Unknown format token '%%%c'\n", q[i+1]);
s = NULL;
}
if ( s ) {
q = realloc(q, strlen(q) + strlen(s));
if ( !q ) {
perror("Process cmdline");
return NULL;
}
// be a bit paranoid and prevent endless expansion
if ( strlen(q) > MAXCMDLEN ) {
// this is fishy
syslog(LOG_ERR, "Error: cmdline too long!\n");
return NULL;
}
memmove(&q[i] + strlen(s), &q[i+2], strlen(&q[i+2]) + 1); // include trailing '0' in memmove
memcpy(&q[i], s, strlen(s));
}
}
i++;
}
return q;
} // End of cmd_expand
/*
* split the command in buf into individual arguments.
*/
static void cmd_parse(char *buf, char **args) {
int i, argnum;
i = argnum = 0;
while ( (i < MAXCMDLEN) && (buf[i] != 0) ) {
/*
* Strip whitespace. Use nulls, so
* that the previous argument is terminated
* automatically.
*/
while ( (i < MAXCMDLEN) && ((buf[i] == ' ') || (buf[i] == '\t')))
buf[i++] = 0;
/*
* Save the argument.
*/
if ( argnum < MAXARGS )
args[argnum++] = &(buf[i]);
/*
* Skip over the argument.
*/
while ( (i < MAXCMDLEN) && ((buf[i] != 0) && (buf[i] != ' ') && (buf[i] != '\t')))
i++;
}
if ( argnum < MAXARGS )
args[argnum] = NULL;
if ( (i >= MAXCMDLEN) || (argnum >= MAXARGS) ) {
// for safety reason, disable the command
args[0] = NULL;
syslog(LOG_ERR, "Launcher: Unable to parse command: '%s'", buf);
}
} // End of cmd_parse
/*
* cmd_execute
* spawn a child process and execute the program.
*/
static void cmd_execute(char **args) {
int pid;
// Get a child process.
syslog(LOG_DEBUG, "Launcher: fork child.");
if ((pid = fork()) < 0) {
syslog(LOG_ERR, "Can't fork: %s", strerror(errno));
return;
}
if (pid == 0) { // we are the child
execvp(*args, args);
syslog(LOG_ERR, "Can't execvp: %s: %s", args[0], strerror(errno));
_exit(1);
}
// we are the parent
syslog(LOG_DEBUG, "Launcher: child exec done.");
/* empty */
} // End of cmd_execute
static void do_expire(char *datadir) {
bookkeeper_t *books;
dirstat_t *dirstat, oldstat;
int ret, bookkeeper_stat, do_rescan;
syslog(LOG_INFO, "Run expire on '%s'", datadir);
do_rescan = 0;
ret = ReadStatInfo(datadir, &dirstat, CREATE_AND_LOCK);
switch (ret) {
case STATFILE_OK:
break;
case ERR_NOSTATFILE:
dirstat->low_water = 95;
case FORCE_REBUILD:
syslog(LOG_INFO, "Force rebuild stat record");
do_rescan = 1;
break;
case ERR_FAIL:
syslog(LOG_ERR, "expire failed: can't read stat record");
return;
/* not reached */
break;
default:
syslog(LOG_ERR, "expire failed: unexpected return code %i reading stat record", ret);
return;
/* not reached */
}
bookkeeper_stat = AccessBookkeeper(&books, datadir);
if ( do_rescan ) {
RescanDir(datadir, dirstat);
if ( bookkeeper_stat == BOOKKEEPER_OK ) {
ClearBooks(books, NULL);
// release the books below
}
}
if ( bookkeeper_stat == BOOKKEEPER_OK ) {
bookkeeper_t tmp_books;
ClearBooks(books, &tmp_books);
UpdateBookStat(dirstat, &tmp_books);
ReleaseBookkeeper(books, DETACH_ONLY);
} else {
syslog(LOG_ERR, "Error %i: can't access book keeping records", ret);
}
syslog(LOG_INFO, "Limits: Filesize %s, Lifetime %s, Watermark: %llu%%\n",
dirstat->max_size ? ScaleValue(dirstat->max_size) : "<none>",
dirstat->max_lifetime ? ScaleTime(dirstat->max_lifetime) : "<none>",
(unsigned long long)dirstat->low_water);
syslog(LOG_INFO, "Current size: %s, Current lifetime: %s, Number of files: %llu",
ScaleValue(dirstat->filesize),
ScaleTime(dirstat->last - dirstat->first),
(unsigned long long)dirstat->numfiles);
oldstat = *dirstat;
if ( dirstat->max_size || dirstat->max_lifetime )
ExpireDir(datadir, dirstat, dirstat->max_size, dirstat->max_lifetime, 0);
WriteStatInfo(dirstat);
if ( (oldstat.numfiles - dirstat->numfiles) > 0 ) {
syslog(LOG_INFO, "expire completed");
syslog(LOG_INFO, " expired files: %llu", (unsigned long long)(oldstat.numfiles - dirstat->numfiles));
syslog(LOG_INFO, " expired time slot: %s", ScaleTime(dirstat->first - oldstat.first));
syslog(LOG_INFO, " expired file size: %s", ScaleValue(oldstat.filesize - dirstat->filesize));
syslog(LOG_INFO, "New size: %s, New lifetime: %s, Number of files: %llu",
ScaleValue(dirstat->filesize),
ScaleTime(dirstat->last - dirstat->first),
(unsigned long long)dirstat->numfiles);
} else {
syslog(LOG_INFO, "expire completed - nothing to expire.");
}
ReleaseStatInfo(dirstat);
} // End of do_expire
void launcher (char *commbuff, FlowSource_t *FlowSource, char *process, int expire) {
FlowSource_t *fs;
struct sigaction act;
char *args[MAXARGS];
int pid, stat;
srecord_t *InfoRecord;
InfoRecord = (srecord_t *)commbuff;
syslog(LOG_INFO, "Launcher: Startup. auto-expire %s", expire ? "enabled" : "off" );
done = launch = child_exit = 0;
// process may be NULL, if we only expire data files
if ( process ) {
char *cmd = NULL;
srecord_t TestRecord;
// check for valid command expansion
strncpy(TestRecord.fname, "test", FNAME_SIZE-1);
TestRecord.fname[FNAME_SIZE-1] = 0;
strncpy(TestRecord.tstring, "200407110845", 15);
TestRecord.tstring[15] = 0;
TestRecord.tstamp = 1;
fs = FlowSource;
while ( fs ) {
cmd = cmd_expand(&TestRecord, fs->Ident, fs->datadir, process);
if ( cmd == NULL ) {
syslog(LOG_ERR, "Launcher: ident: %s, Unable to expand command: '%s'", fs->Ident, process);
exit(255);
}
fs = fs->next;
}
}
/* Signal handling */
memset((void *)&act,0,sizeof(struct sigaction));
act.sa_handler = SignalHandler;
sigemptyset(&act.sa_mask);
act.sa_flags = 0;
sigaction(SIGCHLD, &act, NULL); // child process terminated
sigaction(SIGTERM, &act, NULL); // we are done
sigaction(SIGINT, &act, NULL); // we are done
sigaction(SIGHUP, &act, NULL); // run command
while ( !done ) {
// sleep until we get signaled
syslog(LOG_DEBUG, "Launcher: Sleeping");
select(0, NULL, NULL, NULL, NULL);
syslog(LOG_DEBUG, "Launcher: Wakeup");
if ( launch ) { // SIGHUP
launch = 0;
if ( process ) {
char *cmd = NULL;
fs = FlowSource;
while ( fs ) {
// Expand % placeholders
cmd = cmd_expand(InfoRecord, fs->Ident, fs->datadir, process);
if ( cmd == NULL ) {
syslog(LOG_ERR, "Launcher: ident: %s, Unable to expand command: '%s'", fs->Ident, process);
continue;
}
// printf("Launcher: run command: '%s'\n", cmd);
syslog(LOG_DEBUG, "Launcher: ident: %s run command: '%s'", fs->Ident, cmd);
// prepare args array
cmd_parse(cmd, args);
if ( args[0] )
cmd_execute(args);
// do not flood the system with new processes
sleep(1);
// else cmd_parse already reported the error
free(cmd);
fs = fs->next;
}
}
fs = FlowSource;
while ( fs ) {
if ( expire )
do_expire(fs->datadir);
fs = fs->next;
}
}
if ( child_exit ) {
syslog(LOG_INFO, "laucher child exit %d childs.", child_exit);
while ( (pid = waitpid (-1, &stat, WNOHANG)) > 0 ) {
if ( WIFEXITED(stat) ) {
syslog(LOG_DEBUG, "launcher child %i exit status: %i", pid, WEXITSTATUS(stat));
}
if ( WIFSIGNALED(stat) ) {
syslog(LOG_WARNING, "laucher child %i died due to signal %i", pid, WTERMSIG(stat));
}
child_exit--;
}
syslog(LOG_INFO, "laucher waiting childs done. %d childs", child_exit);
child_exit = 0;
}
if ( done ) {
syslog(LOG_INFO, "Launcher: Terminating.");
}
}
waitpid (-1, &stat, 0);
// we are done
syslog(LOG_INFO, "Launcher: exit.");
} // End of launcher

56
bin/launch.h Normal file
View File

@ -0,0 +1,56 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: launch.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
#ifndef _LAUNCH_H
#define _LAUNCH_H 1
#define FNAME_SIZE 256
#define IDENT_SIZE 32
typedef struct srecord_s {
char fname[FNAME_SIZE]; // file name
char subdir[FNAME_SIZE]; // subdir name
char ident[IDENT_SIZE]; // -I ident string
char tstring[16]; // actually 12 needed e.g. 200411011230
time_t tstamp; // UNIX time stamp
} srecord_t;
void launcher (char *commbuff, char *datadir, char *process, int expire);
#endif //_LAUNCH_H

444
bin/lzoconf.h Executable file
View File

@ -0,0 +1,444 @@
/* lzoconf.h -- configuration of the LZO data compression library
This file is part of the LZO real-time data compression library.
Copyright (C) 1996-2014 Markus Franz Xaver Johannes Oberhumer
All Rights Reserved.
The LZO library is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
The LZO library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with the LZO library; see the file COPYING.
If not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
Markus F.X.J. Oberhumer
<markus@oberhumer.com>
http://www.oberhumer.com/opensource/lzo/
*/
#ifndef __LZOCONF_H_INCLUDED
#define __LZOCONF_H_INCLUDED 1
#define LZO_VERSION 0x2080
#define LZO_VERSION_STRING "2.08"
#define LZO_VERSION_DATE "Jun 29 2014"
/* internal Autoconf configuration file - only used when building LZO */
#if defined(LZO_HAVE_CONFIG_H)
# include <config.h>
#endif
#include <limits.h>
#include <stddef.h>
/***********************************************************************
// LZO requires a conforming <limits.h>
************************************************************************/
#if !defined(CHAR_BIT) || (CHAR_BIT != 8)
# error "invalid CHAR_BIT"
#endif
#if !defined(UCHAR_MAX) || !defined(USHRT_MAX) || !defined(UINT_MAX) || !defined(ULONG_MAX)
# error "check your compiler installation"
#endif
#if (USHRT_MAX < 1) || (UINT_MAX < 1) || (ULONG_MAX < 1)
# error "your limits.h macros are broken"
#endif
/* get OS and architecture defines */
#ifndef __LZODEFS_H_INCLUDED
#include "lzodefs.h"
#endif
#ifdef __cplusplus
extern "C" {
#endif
/***********************************************************************
// some core defines
************************************************************************/
/* memory checkers */
#if !defined(__LZO_CHECKER)
# if defined(__BOUNDS_CHECKING_ON)
# define __LZO_CHECKER 1
# elif defined(__CHECKER__)
# define __LZO_CHECKER 1
# elif defined(__INSURE__)
# define __LZO_CHECKER 1
# elif defined(__PURIFY__)
# define __LZO_CHECKER 1
# endif
#endif
/***********************************************************************
// integral and pointer types
************************************************************************/
/* lzo_uint must match size_t */
#if !defined(LZO_UINT_MAX)
# if (LZO_ABI_LLP64)
# if (LZO_OS_WIN64)
typedef unsigned __int64 lzo_uint;
typedef __int64 lzo_int;
# else
typedef lzo_ullong_t lzo_uint;
typedef lzo_llong_t lzo_int;
# endif
# define LZO_SIZEOF_LZO_UINT 8
# define LZO_UINT_MAX 0xffffffffffffffffull
# define LZO_INT_MAX 9223372036854775807LL
# define LZO_INT_MIN (-1LL - LZO_INT_MAX)
# elif (LZO_ABI_IP32L64) /* MIPS R5900 */
typedef unsigned int lzo_uint;
typedef int lzo_int;
# define LZO_SIZEOF_LZO_UINT LZO_SIZEOF_INT
# define LZO_UINT_MAX UINT_MAX
# define LZO_INT_MAX INT_MAX
# define LZO_INT_MIN INT_MIN
# elif (ULONG_MAX >= LZO_0xffffffffL)
typedef unsigned long lzo_uint;
typedef long lzo_int;
# define LZO_SIZEOF_LZO_UINT LZO_SIZEOF_LONG
# define LZO_UINT_MAX ULONG_MAX
# define LZO_INT_MAX LONG_MAX
# define LZO_INT_MIN LONG_MIN
# else
# error "lzo_uint"
# endif
#endif
/* The larger type of lzo_uint and lzo_uint32_t. */
#if (LZO_SIZEOF_LZO_UINT >= 4)
# define lzo_xint lzo_uint
#else
# define lzo_xint lzo_uint32_t
#endif
typedef int lzo_bool;
/* sanity checks */
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == LZO_SIZEOF_LZO_UINT)
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_xint) >= sizeof(lzo_uint))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_xint) >= sizeof(lzo_uint32_t))
#ifndef __LZO_MMODEL
#define __LZO_MMODEL /*empty*/
#endif
/* no typedef here because of const-pointer issues */
#define lzo_bytep unsigned char __LZO_MMODEL *
#define lzo_charp char __LZO_MMODEL *
#define lzo_voidp void __LZO_MMODEL *
#define lzo_shortp short __LZO_MMODEL *
#define lzo_ushortp unsigned short __LZO_MMODEL *
#define lzo_intp lzo_int __LZO_MMODEL *
#define lzo_uintp lzo_uint __LZO_MMODEL *
#define lzo_xintp lzo_xint __LZO_MMODEL *
#define lzo_voidpp lzo_voidp __LZO_MMODEL *
#define lzo_bytepp lzo_bytep __LZO_MMODEL *
#define lzo_int8_tp lzo_int8_t __LZO_MMODEL *
#define lzo_uint8_tp lzo_uint8_t __LZO_MMODEL *
#define lzo_int16_tp lzo_int16_t __LZO_MMODEL *
#define lzo_uint16_tp lzo_uint16_t __LZO_MMODEL *
#define lzo_int32_tp lzo_int32_t __LZO_MMODEL *
#define lzo_uint32_tp lzo_uint32_t __LZO_MMODEL *
#if defined(lzo_int64_t)
#define lzo_int64_tp lzo_int64_t __LZO_MMODEL *
#define lzo_uint64_tp lzo_uint64_t __LZO_MMODEL *
#endif
/* Older LZO versions used to support ancient systems and memory models
* like 16-bit MSDOS with __huge pointers and Cray PVP, but these
* obsolete configurations are not supported any longer.
*/
#if defined(__LZO_MMODEL_HUGE)
#error "__LZO_MMODEL_HUGE is unsupported"
#endif
#if (LZO_MM_PVP)
#error "LZO_MM_PVP is unsupported"
#endif
#if (LZO_SIZEOF_INT < 4)
#error "LZO_SIZEOF_INT < 4 is unsupported"
#endif
#if (__LZO_UINTPTR_T_IS_POINTER)
#error "__LZO_UINTPTR_T_IS_POINTER is unsupported"
#endif
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(int) >= 4)
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) >= 4)
/* Strange configurations where sizeof(lzo_uint) != sizeof(size_t) should
* work but have not received much testing lately, so be strict here.
*/
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(size_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(ptrdiff_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(lzo_uint) == sizeof(lzo_uintptr_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(void *) == sizeof(lzo_uintptr_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(char *) == sizeof(lzo_uintptr_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(long *) == sizeof(lzo_uintptr_t))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(void *) == sizeof(lzo_voidp))
LZO_COMPILE_TIME_ASSERT_HEADER(sizeof(char *) == sizeof(lzo_bytep))
/***********************************************************************
// function types
************************************************************************/
/* name mangling */
#if !defined(__LZO_EXTERN_C)
# ifdef __cplusplus
# define __LZO_EXTERN_C extern "C"
# else
# define __LZO_EXTERN_C extern
# endif
#endif
/* calling convention */
#if !defined(__LZO_CDECL)
# define __LZO_CDECL __lzo_cdecl
#endif
/* DLL export information */
#if !defined(__LZO_EXPORT1)
# define __LZO_EXPORT1 /*empty*/
#endif
#if !defined(__LZO_EXPORT2)
# define __LZO_EXPORT2 /*empty*/
#endif
/* __cdecl calling convention for public C and assembly functions */
#if !defined(LZO_PUBLIC)
# define LZO_PUBLIC(_rettype) __LZO_EXPORT1 _rettype __LZO_EXPORT2 __LZO_CDECL
#endif
#if !defined(LZO_EXTERN)
# define LZO_EXTERN(_rettype) __LZO_EXTERN_C LZO_PUBLIC(_rettype)
#endif
#if !defined(LZO_PRIVATE)
# define LZO_PRIVATE(_rettype) static _rettype __LZO_CDECL
#endif
/* function types */
typedef int
(__LZO_CDECL *lzo_compress_t) ( const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem );
typedef int
(__LZO_CDECL *lzo_decompress_t) ( const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem );
typedef int
(__LZO_CDECL *lzo_optimize_t) ( lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem );
typedef int
(__LZO_CDECL *lzo_compress_dict_t)(const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem,
const lzo_bytep dict, lzo_uint dict_len );
typedef int
(__LZO_CDECL *lzo_decompress_dict_t)(const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem,
const lzo_bytep dict, lzo_uint dict_len );
/* Callback interface. Currently only the progress indicator ("nprogress")
* is used, but this may change in a future release. */
struct lzo_callback_t;
typedef struct lzo_callback_t lzo_callback_t;
#define lzo_callback_p lzo_callback_t __LZO_MMODEL *
/* malloc & free function types */
typedef lzo_voidp (__LZO_CDECL *lzo_alloc_func_t)
(lzo_callback_p self, lzo_uint items, lzo_uint size);
typedef void (__LZO_CDECL *lzo_free_func_t)
(lzo_callback_p self, lzo_voidp ptr);
/* a progress indicator callback function */
typedef void (__LZO_CDECL *lzo_progress_func_t)
(lzo_callback_p, lzo_uint, lzo_uint, int);
struct lzo_callback_t
{
/* custom allocators (set to 0 to disable) */
lzo_alloc_func_t nalloc; /* [not used right now] */
lzo_free_func_t nfree; /* [not used right now] */
/* a progress indicator callback function (set to 0 to disable) */
lzo_progress_func_t nprogress;
/* INFO: the first parameter "self" of the nalloc/nfree/nprogress
* callbacks points back to this struct, so you are free to store
* some extra info in the following variables. */
lzo_voidp user1;
lzo_xint user2;
lzo_xint user3;
};
/***********************************************************************
// error codes and prototypes
************************************************************************/
/* Error codes for the compression/decompression functions. Negative
* values are errors, positive values will be used for special but
* normal events.
*/
#define LZO_E_OK 0
#define LZO_E_ERROR (-1)
#define LZO_E_OUT_OF_MEMORY (-2) /* [lzo_alloc_func_t failure] */
#define LZO_E_NOT_COMPRESSIBLE (-3) /* [not used right now] */
#define LZO_E_INPUT_OVERRUN (-4)
#define LZO_E_OUTPUT_OVERRUN (-5)
#define LZO_E_LOOKBEHIND_OVERRUN (-6)
#define LZO_E_EOF_NOT_FOUND (-7)
#define LZO_E_INPUT_NOT_CONSUMED (-8)
#define LZO_E_NOT_YET_IMPLEMENTED (-9) /* [not used right now] */
#define LZO_E_INVALID_ARGUMENT (-10)
#define LZO_E_INVALID_ALIGNMENT (-11) /* pointer argument is not properly aligned */
#define LZO_E_OUTPUT_NOT_CONSUMED (-12)
#define LZO_E_INTERNAL_ERROR (-99)
#ifndef lzo_sizeof_dict_t
# define lzo_sizeof_dict_t ((unsigned)sizeof(lzo_bytep))
#endif
/* lzo_init() should be the first function you call.
* Check the return code !
*
* lzo_init() is a macro to allow checking that the library and the
* compiler's view of various types are consistent.
*/
#define lzo_init() __lzo_init_v2(LZO_VERSION,(int)sizeof(short),(int)sizeof(int),\
(int)sizeof(long),(int)sizeof(lzo_uint32_t),(int)sizeof(lzo_uint),\
(int)lzo_sizeof_dict_t,(int)sizeof(char *),(int)sizeof(lzo_voidp),\
(int)sizeof(lzo_callback_t))
LZO_EXTERN(int) __lzo_init_v2(unsigned,int,int,int,int,int,int,int,int,int);
/* version functions (useful for shared libraries) */
LZO_EXTERN(unsigned) lzo_version(void);
LZO_EXTERN(const char *) lzo_version_string(void);
LZO_EXTERN(const char *) lzo_version_date(void);
LZO_EXTERN(const lzo_charp) _lzo_version_string(void);
LZO_EXTERN(const lzo_charp) _lzo_version_date(void);
/* string functions */
LZO_EXTERN(int)
lzo_memcmp(const lzo_voidp a, const lzo_voidp b, lzo_uint len);
LZO_EXTERN(lzo_voidp)
lzo_memcpy(lzo_voidp dst, const lzo_voidp src, lzo_uint len);
LZO_EXTERN(lzo_voidp)
lzo_memmove(lzo_voidp dst, const lzo_voidp src, lzo_uint len);
LZO_EXTERN(lzo_voidp)
lzo_memset(lzo_voidp buf, int c, lzo_uint len);
/* checksum functions */
LZO_EXTERN(lzo_uint32_t)
lzo_adler32(lzo_uint32_t c, const lzo_bytep buf, lzo_uint len);
LZO_EXTERN(lzo_uint32_t)
lzo_crc32(lzo_uint32_t c, const lzo_bytep buf, lzo_uint len);
LZO_EXTERN(const lzo_uint32_tp)
lzo_get_crc32_table(void);
/* misc. */
LZO_EXTERN(int) _lzo_config_check(void);
typedef union {
lzo_voidp a00; lzo_bytep a01; lzo_uint a02; lzo_xint a03; lzo_uintptr_t a04;
void *a05; unsigned char *a06; unsigned long a07; size_t a08; ptrdiff_t a09;
#if defined(lzo_int64_t)
lzo_uint64_t a10;
#endif
} lzo_align_t;
/* align a char pointer on a boundary that is a multiple of 'size' */
LZO_EXTERN(unsigned) __lzo_align_gap(const lzo_voidp p, lzo_uint size);
#define LZO_PTR_ALIGN_UP(p,size) \
((p) + (lzo_uint) __lzo_align_gap((const lzo_voidp)(p),(lzo_uint)(size)))
/***********************************************************************
// deprecated macros - only for backward compatibility
************************************************************************/
/* deprecated - use 'lzo_bytep' instead of 'lzo_byte *' */
#define lzo_byte unsigned char
/* deprecated type names */
#define lzo_int32 lzo_int32_t
#define lzo_uint32 lzo_uint32_t
#define lzo_int32p lzo_int32_t __LZO_MMODEL *
#define lzo_uint32p lzo_uint32_t __LZO_MMODEL *
#define LZO_INT32_MAX LZO_INT32_C(2147483647)
#define LZO_UINT32_MAX LZO_UINT32_C(4294967295)
#if defined(lzo_int64_t)
#define lzo_int64 lzo_int64_t
#define lzo_uint64 lzo_uint64_t
#define lzo_int64p lzo_int64_t __LZO_MMODEL *
#define lzo_uint64p lzo_uint64_t __LZO_MMODEL *
#define LZO_INT64_MAX LZO_INT64_C(9223372036854775807)
#define LZO_UINT64_MAX LZO_UINT64_C(18446744073709551615)
#endif
/* deprecated types */
typedef union { lzo_bytep a; lzo_uint b; } __lzo_pu_u;
typedef union { lzo_bytep a; lzo_uint32_t b; } __lzo_pu32_u;
#if defined(LZO_CFG_COMPAT)
#define __LZOCONF_H 1
#if defined(LZO_ARCH_I086)
# define __LZO_i386 1
#elif defined(LZO_ARCH_I386)
# define __LZO_i386 1
#endif
#if defined(LZO_OS_DOS16)
# define __LZO_DOS 1
# define __LZO_DOS16 1
#elif defined(LZO_OS_DOS32)
# define __LZO_DOS 1
#elif defined(LZO_OS_WIN16)
# define __LZO_WIN 1
# define __LZO_WIN16 1
#elif defined(LZO_OS_WIN32)
# define __LZO_WIN 1
#endif
#define __LZO_CMODEL /*empty*/
#define __LZO_DMODEL /*empty*/
#define __LZO_ENTRY __LZO_CDECL
#define LZO_EXTERN_CDECL LZO_EXTERN
#define LZO_ALIGN LZO_PTR_ALIGN_UP
#define lzo_compress_asm_t lzo_compress_t
#define lzo_decompress_asm_t lzo_decompress_t
#endif /* LZO_CFG_COMPAT */
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* already included */
/* vim:set ts=4 sw=4 et: */

2998
bin/lzodefs.h Executable file
View File

File diff suppressed because it is too large Load Diff

6053
bin/minilzo.c Executable file
View File

File diff suppressed because it is too large Load Diff

94
bin/minilzo.h Executable file
View File

@ -0,0 +1,94 @@
/* minilzo.h -- mini subset of the LZO real-time data compression library
This file is part of the LZO real-time data compression library.
Copyright (C) 1996-2014 Markus Franz Xaver Johannes Oberhumer
All Rights Reserved.
The LZO library is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
The LZO library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with the LZO library; see the file COPYING.
If not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
Markus F.X.J. Oberhumer
<markus@oberhumer.com>
http://www.oberhumer.com/opensource/lzo/
*/
/*
* NOTE:
* the full LZO package can be found at
* http://www.oberhumer.com/opensource/lzo/
*/
#ifndef __MINILZO_H
#define __MINILZO_H 1
#define MINILZO_VERSION 0x2080
#ifdef __LZOCONF_H
# error "you cannot use both LZO and miniLZO"
#endif
#undef LZO_HAVE_CONFIG_H
#include "lzoconf.h"
#if !defined(LZO_VERSION) || (LZO_VERSION != MINILZO_VERSION)
# error "version mismatch in header files"
#endif
#ifdef __cplusplus
extern "C" {
#endif
/***********************************************************************
//
************************************************************************/
/* Memory required for the wrkmem parameter.
* When the required size is 0, you can also pass a NULL pointer.
*/
#define LZO1X_MEM_COMPRESS LZO1X_1_MEM_COMPRESS
#define LZO1X_1_MEM_COMPRESS ((lzo_uint32_t) (16384L * lzo_sizeof_dict_t))
#define LZO1X_MEM_DECOMPRESS (0)
/* compression */
LZO_EXTERN(int)
lzo1x_1_compress ( const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem );
/* decompression */
LZO_EXTERN(int)
lzo1x_decompress ( const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem /* NOT USED */ );
/* safe decompression with overrun testing */
LZO_EXTERN(int)
lzo1x_decompress_safe ( const lzo_bytep src, lzo_uint src_len,
lzo_bytep dst, lzo_uintp dst_len,
lzo_voidp wrkmem /* NOT USED */ );
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* already included */

353
bin/netflow_pcap.c Normal file
View File

@ -0,0 +1,353 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2013, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author:$
*
* $Id:$
*
* $LastChangedRevision:$
*
*
*/
#include "config.h"
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdlib.h>
#include <syslog.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>
#include <pthread.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "nffile.h"
#include "nfx.h"
#include "nfnet.h"
#include "nf_common.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "flowtree.h"
#include "netflow_pcap.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
extern int verbose;
extern extension_descriptor_t extension_descriptor[];
/* module limited globals */
static extension_info_t pcap_extension_info; // common for all pcap records
static extension_map_t *pcap_extension_map;
static uint32_t pcap_output_record_size_v4;
static uint32_t pcap_output_record_size_v6;
typedef struct pcap_v4_block_s {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t dPkts;
uint32_t dOctets;
uint32_t data[1]; // link to next record
} __attribute__((__packed__ )) pcap_v4_block_t;
#define PCAP_V4_BLOCK_DATA_SIZE (sizeof(pcap_v4_block_t) - sizeof(uint32_t))
typedef struct pcap_v6_block_s {
uint64_t srcaddr[2];
uint64_t dstaddr[2];
uint32_t dPkts;
uint32_t dOctets;
uint32_t data[1]; // link to next record
} __attribute__((__packed__ )) pcap_v6_block_t;
#define PCAP_V6_BLOCK_DATA_SIZE (sizeof(pcap_v6_block_t) - sizeof(uint32_t))
// All required extension to save full pcap records
static uint16_t pcap_full_map[] = { 0 };
#include "nffile_inline.c"
int Init_pcap2nf(void) {
int i, id, map_index;
int extension_size;
uint16_t map_size;
// prepare pcap extension map
pcap_extension_info.map = NULL;
extension_size = 0;
map_size = 0;
i=0;
while ( (id = pcap_full_map[i]) != 0 ) {
if ( extension_descriptor[id].enabled ) {
extension_size += extension_descriptor[id].size;
map_size += sizeof(uint16_t);
}
i++;
}
// extension_size contains the sum of all optional extensions
// caculate the record size
pcap_output_record_size_v4 = COMMON_RECORD_DATA_SIZE + PCAP_V4_BLOCK_DATA_SIZE + extension_size;
pcap_output_record_size_v6 = COMMON_RECORD_DATA_SIZE + PCAP_V6_BLOCK_DATA_SIZE + extension_size;
// now the full extension map size
map_size += sizeof(extension_map_t);
// align 32 bits
if ( ( map_size & 0x3 ) != 0 )
map_size += 2;
// Create a generic pcap extension map
pcap_extension_info.map = (extension_map_t *)malloc((size_t)map_size);
if ( !pcap_extension_info.map ) {
syslog(LOG_ERR, "Process_pcap: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
pcap_extension_info.map->type = ExtensionMapType;
pcap_extension_info.map->size = map_size;
pcap_extension_info.map->map_id = INIT_ID;
pcap_extension_info.map->extension_size = extension_size;
// see netflow_pcap.h for extension map description
map_index = 0;
i=0;
while ( (id = pcap_full_map[i]) != 0 ) {
if ( extension_descriptor[id].enabled )
pcap_extension_info.map->ex_id[map_index++] = id;
i++;
}
pcap_extension_info.map->ex_id[map_index] = 0;
pcap_extension_map = NULL;
return 1;
} // End of Init_pcap2nf
int StorePcapFlow(FlowSource_t *fs, struct FlowNode *Node) {
common_record_t *common_record;
uint32_t packets, bytes, pcap_output_record_size;
uint64_t start_time, end_time;
int j, id;
char *string;
void *data_ptr;
if ( !pcap_extension_map ) {
pcap_extension_map = (extension_map_t *)malloc(pcap_extension_info.map->size);
if ( !pcap_extension_map ) {
LogError("Process_pcap: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
memcpy((void *)pcap_extension_map, (void *)pcap_extension_info.map, pcap_extension_info.map->size);
if ( !AddExtensionMap(fs, pcap_extension_map) ) {
LogError("Process_pcap: Fatal: AddExtensionMap() failed in %s line %d\n", __FILE__, __LINE__);
return 0;
}
}
if ( Node->version == AF_INET6 ) {
pcap_output_record_size = pcap_output_record_size_v6;
dbg_printf("Store Flow v6 node: size: %u\n", pcap_output_record_size);
} else if ( Node->version == AF_INET ) {
pcap_output_record_size = pcap_output_record_size_v4;
dbg_printf("Store Flow v4 node: size: %u\n", pcap_output_record_size);
} else {
LogError("Process_pcap: Unexpected version in %s line %d: %u\n", __FILE__, __LINE__, Node->version);
return 0;
}
// output buffer size check for all expected records
if ( !CheckBufferSpace(fs->nffile, pcap_output_record_size) ) {
// fishy! - should never happen. maybe disk full?
LogError("Process_pcap: output buffer size error. Abort pcap record processing");
return 0;
}
// map output record to memory buffer
common_record = (common_record_t *)fs->nffile->buff_ptr;
// header data
common_record->flags = 0;
common_record->type = CommonRecordType;
common_record->exporter_sysid = 0;
common_record->ext_map = pcap_extension_map->map_id;
common_record->size = pcap_output_record_size;
// pcap common fields
common_record->srcport = Node->src_port;
common_record->dstport = Node->dst_port;
common_record->tcp_flags = Node->flags;
common_record->prot = Node->proto;
common_record->tos = 0;
common_record->fwd_status = 0;
if ( Node->version == AF_INET6 ) {
SetFlag(common_record->flags, FLAG_IPV6_ADDR);
pcap_v6_block_t *pcap_v6_block = (pcap_v6_block_t *)common_record->data;
pcap_v6_block->srcaddr[0] = Node->src_addr.v6[0];
pcap_v6_block->srcaddr[1] = Node->src_addr.v6[1];
pcap_v6_block->dstaddr[0] = Node->dst_addr.v6[0];
pcap_v6_block->dstaddr[1] = Node->dst_addr.v6[1];
pcap_v6_block->dPkts = packets = Node->packets;
pcap_v6_block->dOctets = bytes = Node->bytes;
data_ptr = (void *)pcap_v6_block->data;
} else {
pcap_v4_block_t *pcap_v4_block = (pcap_v4_block_t *)common_record->data;
pcap_v4_block->srcaddr = Node->src_addr.v4;
pcap_v4_block->dstaddr = Node->dst_addr.v4;
pcap_v4_block->dPkts = packets = Node->packets;
pcap_v4_block->dOctets = bytes = Node->bytes;
data_ptr = (void *)pcap_v4_block->data;
}
// process optional extensions
j = 0;
while ( (id = pcap_extension_map->ex_id[j]) != 0 ) {
switch (id) {
case EX_IO_SNMP_2: { // 2 byte input/output interface index
tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr;
tpl->input = 0;
tpl->output = 0;
data_ptr = (void *)tpl->data;
} break;
default:
// this should never happen, as pcap has no other extensions
LogError("Process_pcap: Unexpected extension %i for pcap record. Skip extension", id);
}
j++;
}
common_record->first = Node->t_first.tv_sec;
common_record->msec_first = Node->t_first.tv_usec / 1000;
common_record->last = Node->t_last.tv_sec;
common_record->msec_last = Node->t_last.tv_usec / 1000;
start_time = (1000LL * (uint64_t)common_record->first) + (uint64_t)common_record->msec_first;
end_time = (1000LL * (uint64_t)common_record->last) + (uint64_t)common_record->msec_last;
// update first_seen, last_seen
if ( start_time < fs->first_seen )
fs->first_seen = start_time;
if ( end_time > fs->last_seen )
fs->last_seen = end_time;
// Update stats
switch (common_record->prot) {
case IPPROTO_ICMP:
case IPPROTO_ICMPV6:
fs->nffile->stat_record->numflows_icmp++;
fs->nffile->stat_record->numpackets_icmp += packets;
fs->nffile->stat_record->numbytes_icmp += bytes;
// fix odd CISCO behaviour for ICMP port/type in src port
if ( common_record->srcport != 0 ) {
uint8_t *s1, *s2;
s1 = (uint8_t *)&(common_record->srcport);
s2 = (uint8_t *)&(common_record->dstport);
s2[0] = s1[1];
s2[1] = s1[0];
common_record->srcport = 0;
}
break;
case IPPROTO_TCP:
fs->nffile->stat_record->numflows_tcp++;
fs->nffile->stat_record->numpackets_tcp += packets;
fs->nffile->stat_record->numbytes_tcp += bytes;
break;
case IPPROTO_UDP:
fs->nffile->stat_record->numflows_udp++;
fs->nffile->stat_record->numpackets_udp += packets;
fs->nffile->stat_record->numbytes_udp += bytes;
break;
default:
fs->nffile->stat_record->numflows_other++;
fs->nffile->stat_record->numpackets_other += packets;
fs->nffile->stat_record->numbytes_other += bytes;
}
fs->nffile->stat_record->numflows++;
fs->nffile->stat_record->numpackets += packets;
fs->nffile->stat_record->numbytes += bytes;
if ( fs->xstat ) {
uint32_t bpp = packets ? bytes/packets : 0;
if ( bpp > MAX_BPP )
bpp = MAX_BPP;
if ( common_record->prot == IPPROTO_TCP ) {
fs->xstat->bpp_histogram->tcp.bpp[bpp]++;
fs->xstat->bpp_histogram->tcp.count++;
fs->xstat->port_histogram->src_tcp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_tcp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_tcp.count++;
fs->xstat->port_histogram->dst_tcp.count++;
} else if ( common_record->prot == IPPROTO_UDP ) {
fs->xstat->bpp_histogram->udp.bpp[bpp]++;
fs->xstat->bpp_histogram->udp.count++;
fs->xstat->port_histogram->src_udp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_udp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_udp.count++;
fs->xstat->port_histogram->dst_udp.count++;
}
}
if ( verbose ) {
master_record_t master_record;
ExpandRecord_v2((common_record_t *)common_record, &pcap_extension_info, NULL, &master_record);
format_file_block_record(&master_record, &string, 0);
printf("%s\n", string);
}
// update file record size ( -> output buffer size )
fs->nffile->block_header->NumRecords += 1;
fs->nffile->block_header->size += pcap_output_record_size;
fs->nffile->buff_ptr = data_ptr;
return 1;
} /* End of StorePcapFlow */

42
bin/netflow_pcap.h Normal file
View File

@ -0,0 +1,42 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2013, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author:$
*
* $Id:$
*
* $LastChangedRevision:$
*
*
*/
int Init_pcap2nf(void);
int StorePcapFlow(FlowSource_t *fs, struct FlowNode *Node);

538
bin/netflow_v1.c Normal file
View File

@ -0,0 +1,538 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: netflow_v1.c 30 2011-07-18 11:19:46Z peter $
*
* $LastChangedRevision: 30 $
*
*/
#include "config.h"
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdlib.h>
#include <syslog.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "nffile.h"
#include "nfx.h"
#include "nfnet.h"
#include "nf_common.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "netflow_v1.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
extern int verbose;
extern extension_descriptor_t extension_descriptor[];
/* module limited globals */
static extension_info_t v1_extension_info; // common for all v1 records
static uint16_t v1_output_record_size;
// All required extension to save full v1 records
static uint16_t v1_full_map[] = { EX_IO_SNMP_2, EX_NEXT_HOP_v4, EX_ROUTER_IP_v4, EX_RECEIVED, 0 };
typedef struct v1_block_s {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t dPkts;
uint32_t dOctets;
uint32_t data[1]; // link to next record
} v1_block_t;
#define V1_BLOCK_DATA_SIZE (sizeof(v1_block_t) - sizeof(uint32_t))
typedef struct exporter_v1_s {
// identical to generic_exporter_t
struct exporter_v1_s *next;
// generic exporter information
exporter_info_record_t info;
uint64_t packets; // number of packets sent by this exporter
uint64_t flows; // number of flow records sent by this exporter
uint32_t sequence_failure; // number of sequence failues
generic_sampler_t *sampler;
// End of generic_exporter_t
// extension map
extension_map_t *extension_map;
} exporter_v1_t;
static inline exporter_v1_t *GetExporter(FlowSource_t *fs, netflow_v1_header_t *header);
/* functions */
#include "nffile_inline.c"
int Init_v1(void) {
int i, id, map_index;
int extension_size;
uint16_t map_size;
// prepare v1 extension map
v1_extension_info.map = NULL;
v1_extension_info.next = NULL;
v1_extension_info.offset_cache = NULL;
v1_extension_info.ref_count = 0;
extension_size = 0;
// default map - 0 extensions
map_size = sizeof(extension_map_t);
i=0;
dbg_printf("v1 map: map size start: %u\n", map_size);
while ( (id = v1_full_map[i]) != 0 ) {
if ( extension_descriptor[id].enabled ) {
extension_size += extension_descriptor[id].size;
map_size += sizeof(uint16_t);
dbg_printf("v1 map: enabled extension %u\n", id);
}
i++;
}
dbg_printf("v1 map: map size so far: %u\n", map_size);
// extension_size contains the sum of all optional extensions
// caculate the record size
v1_output_record_size = COMMON_RECORD_DATA_SIZE + V1_BLOCK_DATA_SIZE + extension_size;
// align 32 bits
if ( ( map_size & 0x3 ) != 0 )
map_size += 2;
// Create a generic netflow v1 extension map
v1_extension_info.map = (extension_map_t *)malloc((size_t)map_size);
if ( !v1_extension_info.map ) {
syslog(LOG_ERR, "Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
v1_extension_info.map->type = ExtensionMapType;
v1_extension_info.map->size = map_size;
v1_extension_info.map->map_id = INIT_ID;
v1_extension_info.map->extension_size = extension_size;
// see netflow_v1.h for extension map description
map_index = 0;
i=0;
while ( (id = v1_full_map[i]) != 0 ) {
if ( extension_descriptor[id].enabled )
v1_extension_info.map->ex_id[map_index++] = id;
i++;
}
v1_extension_info.map->ex_id[map_index] = 0;
return 1;
} // End of Init_v1
/*
* functions used for receiving netflow v1 records
*/
static inline exporter_v1_t *GetExporter(FlowSource_t *fs, netflow_v1_header_t *header) {
exporter_v1_t **e = (exporter_v1_t **)&(fs->exporter_data);
uint16_t version = ntohs(header->version);
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
// search the appropriate exporter engine
while ( *e ) {
if ( (*e)->info.version == version &&
(*e)->info.ip.v6[0] == fs->ip.v6[0] && (*e)->info.ip.v6[1] == fs->ip.v6[1])
return *e;
e = &((*e)->next);
}
// nothing found
*e = (exporter_v1_t *)malloc(sizeof(exporter_v1_t));
if ( !(*e)) {
syslog(LOG_ERR, "Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return NULL;
}
memset((void *)(*e), 0, sizeof(exporter_v1_t));
(*e)->info.header.type = ExporterInfoRecordType;
(*e)->info.header.size = sizeof(exporter_info_record_t);
(*e)->info.version = version;
(*e)->info.id = 0;
(*e)->info.ip = fs->ip;
(*e)->info.sa_family = fs->sa_family;
(*e)->next = NULL;
(*e)->packets = 0;
(*e)->flows = 0;
(*e)->sequence_failure = 0;
(*e)->sampler = NULL;
// copy the v1 generic extension map
(*e)->extension_map = (extension_map_t *)malloc(v1_extension_info.map->size);
if ( !(*e)->extension_map ) {
syslog(LOG_ERR, "Process_v1: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
free(*e);
*e = NULL;
return NULL;
}
memcpy((void *)(*e)->extension_map, (void *)v1_extension_info.map, v1_extension_info.map->size);
if ( !AddExtensionMap(fs, (*e)->extension_map) ) {
// bad - we must free this map and fail - otherwise data can not be read any more
free((*e)->extension_map);
free(*e);
*e = NULL;
return NULL;
}
(*e)->info.sysid = 0;
FlushInfoExporter(fs, &((*e)->info));
if ( fs->sa_family == AF_INET ) {
uint32_t _ip = htonl(fs->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
} else if ( fs->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(fs->ip.v6[0]);
_ip[1] = htonll(fs->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
}
dbg_printf("New Exporter: v1 SysID: %u, Extension ID: %i, IP: %s, \n",
(*e)->info.sysid, (*e)->extension_map->map_id, ipstr);
syslog(LOG_INFO, "Process_v1: SysID: %u, New exporter: IP: %s\n", (*e)->info.sysid, ipstr);
return (*e);
} // End of GetExporter
void Process_v1(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) {
netflow_v1_header_t *v1_header;
netflow_v1_record_t *v1_record;
exporter_v1_t *exporter;
extension_map_t *extension_map;
common_record_t *common_record;
uint64_t start_time, end_time, boot_time;
uint32_t First, Last;
uint16_t count;
uint8_t flags;
int i, done, flow_record_length;
ssize_t size_left;
char *string;
// map v1 data structure to input buffer
v1_header = (netflow_v1_header_t *)in_buff;
exporter = GetExporter(fs, v1_header);
if ( !exporter ) {
syslog(LOG_ERR,"Process_v1: Exporter NULL: Abort v1 record processing");
return;
}
flags = 0;
exporter->packets++;
extension_map = exporter->extension_map;
flow_record_length = NETFLOW_V1_RECORD_LENGTH;
// this many data to process
size_left = in_buff_cnt;
common_record = fs->nffile->buff_ptr;
done = 0;
while ( !done ) {
v1_block_t *v1_block;
/* Process header */
// count check
count = ntohs(v1_header->count);
if ( count > NETFLOW_V1_MAX_RECORDS ) {
syslog(LOG_ERR,"Process_v1: Unexpected record count in header: %i. Abort v1 record processing", count);
fs->nffile->buff_ptr = (void *)common_record;
return;
}
// input buffer size check for all expected records
if ( size_left < ( NETFLOW_V1_HEADER_LENGTH + count * flow_record_length) ) {
syslog(LOG_ERR,"Process_v1: Not enough data to process v1 record. Abort v1 record processing");
fs->nffile->buff_ptr = (void *)common_record;
return;
}
// output buffer size check for all expected records
if ( !CheckBufferSpace(fs->nffile, count * v1_output_record_size) ) {
// fishy! - should never happen. maybe disk full?
syslog(LOG_ERR,"Process_v1: output buffer size error. Abort v1 record processing");
return;
}
// map output record to memory buffer
common_record = (common_record_t *)fs->nffile->buff_ptr;
v1_block = (v1_block_t *)common_record->data;
v1_header->SysUptime = ntohl(v1_header->SysUptime);
v1_header->unix_secs = ntohl(v1_header->unix_secs);
v1_header->unix_nsecs = ntohl(v1_header->unix_nsecs);
/* calculate boot time in msec */
boot_time = ((uint64_t)(v1_header->unix_secs)*1000 +
((uint64_t)(v1_header->unix_nsecs) / 1000000) ) - (uint64_t)(v1_header->SysUptime);
// process all records
v1_record = (netflow_v1_record_t *)((pointer_addr_t)v1_header + NETFLOW_V1_HEADER_LENGTH);
/* loop over each records associated with this header */
for (i = 0; i < count; i++) {
pointer_addr_t bsize;
void *data_ptr;
uint8_t *s1, *s2;
int j, id;
// header data
common_record->flags = flags;
common_record->type = CommonRecordType;
common_record->exporter_sysid = exporter->info.sysid;
common_record->ext_map = extension_map->map_id;
common_record->size = v1_output_record_size;
// v1 common fields
common_record->srcport = ntohs(v1_record->srcport);
common_record->dstport = ntohs(v1_record->dstport);
common_record->tcp_flags = v1_record->tcp_flags;
common_record->prot = v1_record->prot;
common_record->tos = v1_record->tos;
common_record->fwd_status = 0;
common_record->reserved = 0;
// v1 typed data as fixed struct v1_block
v1_block->srcaddr = ntohl(v1_record->srcaddr);
v1_block->dstaddr = ntohl(v1_record->dstaddr);
v1_block->dPkts = ntohl(v1_record->dPkts);
v1_block->dOctets = ntohl(v1_record->dOctets);
// process optional extensions
data_ptr = (void *)v1_block->data;
j = 0;
while ( (id = extension_map->ex_id[j]) != 0 ) {
switch (id) {
case EX_IO_SNMP_2: { // 2 byte input/output interface index
tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr;
tpl->input = ntohs(v1_record->input);
tpl->output = ntohs(v1_record->output);
data_ptr = (void *)tpl->data;
} break;
case EX_NEXT_HOP_v4: { // IPv4 next hop
tpl_ext_9_t *tpl = (tpl_ext_9_t *)data_ptr;
tpl->nexthop = ntohl(v1_record->nexthop);
data_ptr = (void *)tpl->data;
} break;
case EX_ROUTER_IP_v4: { // IPv4 router address
tpl_ext_23_t *tpl = (tpl_ext_23_t *)data_ptr;
tpl->router_ip = fs->ip.v4;
data_ptr = (void *)tpl->data;
ClearFlag(common_record->flags, FLAG_IPV6_EXP);
} break;
case EX_RECEIVED: {
tpl_ext_27_t *tpl = (tpl_ext_27_t *)data_ptr;
tpl->received = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL);
data_ptr = (void *)tpl->data;
} break;
default:
// this should never happen, as v1 has no other extensions
syslog(LOG_ERR,"Process_v1: Unexpected extension %i for v1 record. Skip extension", id);
}
j++;
}
// Time issues
First = ntohl(v1_record->First);
Last = ntohl(v1_record->Last);
if ( First > Last ) {
/* First in msec, in case of msec overflow, between start and end */
start_time = boot_time - 0x100000000LL + (uint64_t)First;
} else {
start_time = boot_time + (uint64_t)First;
}
/* end time in msecs */
end_time = (uint64_t)Last + boot_time;
// if overflow happened after flow ended but before got exported
if ( Last > v1_header->SysUptime ) {
start_time -= 0x100000000LL;
end_time -= 0x100000000LL;
}
common_record->first = start_time/1000;
common_record->msec_first = start_time - common_record->first*1000;
common_record->last = end_time/1000;
common_record->msec_last = end_time - common_record->last*1000;
// update first_seen, last_seen
if ( start_time < fs->first_seen )
fs->first_seen = start_time;
if ( end_time > fs->last_seen )
fs->last_seen = end_time;
// Update stats
switch (common_record->prot) {
case IPPROTO_ICMP:
fs->nffile->stat_record->numflows_icmp++;
fs->nffile->stat_record->numpackets_icmp += v1_block->dPkts;
fs->nffile->stat_record->numbytes_icmp += v1_block->dOctets;
// fix odd CISCO behaviour for ICMP port/type in src port
if ( common_record->srcport != 0 ) {
s1 = (uint8_t *)&(common_record->srcport);
s2 = (uint8_t *)&(common_record->dstport);
s2[0] = s1[1];
s2[1] = s1[0];
common_record->srcport = 0;
}
break;
case IPPROTO_TCP:
fs->nffile->stat_record->numflows_tcp++;
fs->nffile->stat_record->numpackets_tcp += v1_block->dPkts;
fs->nffile->stat_record->numbytes_tcp += v1_block->dOctets;
break;
case IPPROTO_UDP:
fs->nffile->stat_record->numflows_udp++;
fs->nffile->stat_record->numpackets_udp += v1_block->dPkts;
fs->nffile->stat_record->numbytes_udp += v1_block->dOctets;
break;
default:
fs->nffile->stat_record->numflows_other++;
fs->nffile->stat_record->numpackets_other += v1_block->dPkts;
fs->nffile->stat_record->numbytes_other += v1_block->dOctets;
}
exporter->flows++;
fs->nffile->stat_record->numflows++;
fs->nffile->stat_record->numpackets += v1_block->dPkts;
fs->nffile->stat_record->numbytes += v1_block->dOctets;
if ( fs->xstat ) {
uint32_t bpp = v1_block->dPkts ? v1_block->dOctets/v1_block->dPkts : 0;
if ( bpp > MAX_BPP )
bpp = MAX_BPP;
if ( common_record->prot == IPPROTO_TCP ) {
fs->xstat->bpp_histogram->tcp.bpp[bpp]++;
fs->xstat->bpp_histogram->tcp.count++;
fs->xstat->port_histogram->src_tcp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_tcp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_tcp.count++;
fs->xstat->port_histogram->dst_tcp.count++;
} else if ( common_record->prot == IPPROTO_UDP ) {
fs->xstat->bpp_histogram->udp.bpp[bpp]++;
fs->xstat->bpp_histogram->udp.count++;
fs->xstat->port_histogram->src_udp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_udp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_udp.count++;
fs->xstat->port_histogram->dst_udp.count++;
}
}
if ( verbose ) {
master_record_t master_record;
ExpandRecord_v2((common_record_t *)common_record, &v1_extension_info, &(exporter->info), &master_record);
format_file_block_record(&master_record, &string, 0);
printf("%s\n", string);
}
// advance to next input flow record
v1_record = (netflow_v1_record_t *)((pointer_addr_t)v1_record + flow_record_length);
if ( ((pointer_addr_t)data_ptr - (pointer_addr_t)common_record) != v1_output_record_size ) {
printf("Panic size check: ptr diff: %llu, record size: %u\n", (unsigned long long)((pointer_addr_t)data_ptr - (pointer_addr_t)common_record), v1_output_record_size );
abort();
}
// advance to next output record
common_record = (common_record_t *)data_ptr;
v1_block = (v1_block_t *)common_record->data;
// buffer size sanity check - should never happen, but check it anyway
bsize = (pointer_addr_t)common_record - (pointer_addr_t)fs->nffile->block_header - sizeof(data_block_header_t);
if ( bsize > BUFFSIZE ) {
syslog(LOG_ERR,"### Software error ###: %s line %d", __FILE__, __LINE__);
syslog(LOG_ERR,"Process_v1: Output buffer overflow! Flush buffer and skip records.");
syslog(LOG_ERR,"Buffer size: size: %u, bsize: %llu > %u", fs->nffile->block_header->size, (unsigned long long)bsize, BUFFSIZE);
// reset buffer
fs->nffile->block_header->size = 0;
fs->nffile->block_header->NumRecords = 0;
fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) );
return;
}
} // End of foreach v1 record
// update file record size ( -> output buffer size )
fs->nffile->block_header->NumRecords += count;
fs->nffile->block_header->size += count * v1_output_record_size;
fs->nffile->buff_ptr = (void *)common_record;
// still to go for this many input bytes
size_left -= NETFLOW_V1_HEADER_LENGTH + count * flow_record_length;
// next header
v1_header = (netflow_v1_header_t *)v1_record;
// should never be < 0
done = size_left <= 0;
} // End of while !done
return;
} /* End of Process_v1 */

97
bin/netflow_v1.h Normal file
View File

@ -0,0 +1,97 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: netflow_v1.h 26 2011-07-05 18:51:25Z peter $
*
* $LastChangedRevision: 26 $
*
*/
#ifndef _NETFLOW_V1_H
#define _NETFLOW_V1_H 1
#define NETFLOW_V1_HEADER_LENGTH 16
#define NETFLOW_V1_RECORD_LENGTH 48
#define NETFLOW_V1_MAX_RECORDS 24
/* v1 structures */
typedef struct netflow_v1_header {
uint16_t version;
uint16_t count;
uint32_t SysUptime;
uint32_t unix_secs;
uint32_t unix_nsecs;
} netflow_v1_header_t;
typedef struct netflow_v1_record {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t nexthop;
uint16_t input;
uint16_t output;
uint32_t dPkts;
uint32_t dOctets;
uint32_t First;
uint32_t Last;
uint16_t srcport;
uint16_t dstport;
uint16_t pad1;
uint8_t prot;
uint8_t tos;
uint8_t tcp_flags;
uint8_t pad2[7];
} netflow_v1_record_t;
/* prototypes */
int Init_v1(void);
void Process_v1(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs);
/*
* Extension map for v1
*
* Required extensions:
*
* 4 byte byte counter
* | 4byte packet counter
* | | IPv4
* | | |
* xxxx x0 0 0
*
* Optional extensions:
*
* 4 : 2 byte input/output interface id
* 9 : IPv4 next hop
*/
#endif //_NETFLOW_V1_H

796
bin/netflow_v5_v7.c Normal file
View File

@ -0,0 +1,796 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: netflow_v5_v7.c 69 2010-09-09 07:17:43Z haag $
*
* $LastChangedRevision: 69 $
*
*/
#include "config.h"
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdlib.h>
#include <syslog.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "nffile.h"
#include "nfx.h"
#include "nfnet.h"
#include "nf_common.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "netflow_v5_v7.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
extern int verbose;
extern extension_descriptor_t extension_descriptor[];
extern uint32_t default_sampling;
extern uint32_t overwrite_sampling;
/* module limited globals */
static extension_info_t v5_extension_info; // common for all v5 records
static uint16_t v5_output_record_size, v5_output_record_base_size;
// All required extension to save full v5 records
static uint16_t v5_full_mapp[] = { EX_IO_SNMP_2, EX_AS_2, EX_MULIPLE, EX_NEXT_HOP_v4, EX_ROUTER_IP_v4, EX_ROUTER_ID, EX_RECEIVED, 0 };
// to simplify, assume blocks with 64 bit counters to check for enough buffer space
// regardless if 32 or 64 bit packet/byte counters
#define V5_BLOCK_DATA_SIZE (sizeof(ipv4_block_t) - sizeof(uint32_t) + 2 * sizeof(uint64_t))
typedef struct exporter_v5_s {
// identical to generic_exporter_t
struct exporter_v5_s *next;
// generic exporter information
exporter_info_record_t info;
uint64_t packets; // number of packets sent by this exporter
uint64_t flows; // number of flow records sent by this exporter
uint32_t sequence_failure; // number of sequence failues
// generic sampler
generic_sampler_t *sampler;
// end of generic_exporter_t
// sequence vars
int64_t last_sequence;
int64_t sequence, distance;
int64_t last_count;
int first;
// extension map
extension_map_t *extension_map;
} exporter_v5_t;
// for sending netflow v5
static netflow_v5_header_t *v5_output_header;
static netflow_v5_record_t *v5_output_record;
static exporter_v5_t output_engine;
static inline exporter_v5_t *GetExporter(FlowSource_t *fs, netflow_v5_header_t *header);
static inline int CheckBufferSpace(nffile_t *nffile, size_t required);
/* functions */
#include "nffile_inline.c"
int Init_v5_v7_input(void) {
int i, id, map_index;
int extension_size;
uint16_t map_size;
extension_size = 0;
// prepare v5 extension map
v5_extension_info.map = NULL;
v5_extension_info.next = NULL;
v5_extension_info.offset_cache = NULL;
v5_extension_info.ref_count = 0;
// default map - 0 extensions
map_size = sizeof(extension_map_t);
i=0;
while ( (id = v5_full_mapp[i]) != 0 ) {
if ( extension_descriptor[id].enabled ) {
extension_size += extension_descriptor[id].size;
map_size += sizeof(uint16_t);
}
i++;
}
// extension_size contains the sum of all optional extensions
// caculate the record size without counters!
v5_output_record_base_size = COMMON_RECORD_DATA_SIZE + 8 + extension_size; // + 8 for 2 x IPv4 addr
// align 32 bits
if ( ( map_size & 0x3 ) != 0 )
map_size += 2;
// Create a generic v5 extension map
v5_extension_info.map = (extension_map_t *)malloc((size_t)map_size);
if ( !v5_extension_info.map ) {
syslog(LOG_ERR, "Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
v5_extension_info.map->type = ExtensionMapType;
v5_extension_info.map->size = map_size;
v5_extension_info.map->map_id = INIT_ID;
v5_extension_info.map->extension_size = extension_size;
// see netflow_v5_v7.h for extension map description
map_index = 0;
i=0;
while ( (id = v5_full_mapp[i]) != 0 ) {
if ( extension_descriptor[id].enabled )
v5_extension_info.map->ex_id[map_index++] = id;
i++;
}
v5_extension_info.map->ex_id[map_index] = 0;
return 1;
} // End of Init_v5_input
/*
* functions used for receiving netflow v5 records
*/
static inline exporter_v5_t *GetExporter(FlowSource_t *fs, netflow_v5_header_t *header) {
exporter_v5_t **e = (exporter_v5_t **)&(fs->exporter_data);
generic_sampler_t *sampler;
uint16_t engine_tag = ntohs(header->engine_tag);
uint16_t version = ntohs(header->version);
#define IP_STRING_LEN 40
char ipstr[IP_STRING_LEN];
// search the appropriate exporter engine
while ( *e ) {
if ( (*e)->info.version == version && (*e)->info.id == engine_tag &&
(*e)->info.ip.v6[0] == fs->ip.v6[0] && (*e)->info.ip.v6[1] == fs->ip.v6[1])
return *e;
e = &((*e)->next);
}
// nothing found
*e = (exporter_v5_t *)malloc(sizeof(exporter_v5_t));
if ( !(*e)) {
syslog(LOG_ERR, "Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return NULL;
}
memset((void *)(*e), 0, sizeof(exporter_v5_t));
(*e)->next = NULL;
(*e)->info.header.type = ExporterInfoRecordType;
(*e)->info.header.size = sizeof(exporter_info_record_t);
(*e)->info.version = version;
(*e)->info.id = engine_tag;
(*e)->info.ip = fs->ip;
(*e)->info.sa_family = fs->sa_family;
(*e)->sequence_failure = 0;
(*e)->packets = 0;
(*e)->flows = 0;
(*e)->first = 1;
sampler = (generic_sampler_t *)malloc(sizeof(generic_sampler_t));
if ( !sampler ) {
syslog(LOG_ERR, "Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return NULL;
}
(*e)->sampler = sampler;
sampler->info.header.type = SamplerInfoRecordype;
sampler->info.header.size = sizeof(sampler_info_record_t);
sampler->info.id = -1;
sampler->info.mode = (0xC000 & ntohs(header->sampling_interval)) >> 14;
sampler->info.interval = 0x3fff & ntohs(header->sampling_interval);
sampler->next = NULL;
// default is global default_sampling ( user defined or unsampled => 1 )
if ( sampler->info.interval == 0 )
sampler->info.interval = default_sampling;
// copy the v5 generic extension map
(*e)->extension_map = (extension_map_t *)malloc(v5_extension_info.map->size);
if ( !(*e)->extension_map ) {
syslog(LOG_ERR, "Process_v5: malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
free(*e);
*e = NULL;
return NULL;
}
memcpy((void *)(*e)->extension_map, (void *)v5_extension_info.map, v5_extension_info.map->size);
if ( !AddExtensionMap(fs, (*e)->extension_map) ) {
// bad - we must free this map and fail - otherwise data can not be read any more
free((*e)->extension_map);
free(*e);
*e = NULL;
return NULL;
}
(*e)->info.sysid = 0;
FlushInfoExporter(fs, &((*e)->info));
sampler->info.exporter_sysid = (*e)->info.sysid;
FlushInfoSampler(fs, &(sampler->info));
if ( fs->sa_family == AF_INET ) {
uint32_t _ip = htonl(fs->ip.v4);
inet_ntop(AF_INET, &_ip, ipstr, sizeof(ipstr));
} else if ( fs->sa_family == AF_INET6 ) {
uint64_t _ip[2];
_ip[0] = htonll(fs->ip.v6[0]);
_ip[1] = htonll(fs->ip.v6[1]);
inet_ntop(AF_INET6, &_ip, ipstr, sizeof(ipstr));
} else {
strncpy(ipstr, "<unknown>", IP_STRING_LEN);
}
dbg_printf("New Exporter: v5 SysID: %u, Extension ID: %i, IP: %s, Sampling Mode: %i, Sampling Interval: %u\n",
(*e)->info.sysid, (*e)->extension_map->map_id, ipstr, sampler->info.mode ,sampler->info.interval);
syslog(LOG_INFO, "Process_v5: New exporter: SysID: %u, engine id %u, type %u, IP: %s, Sampling Mode: %i, Sampling Interval: %u\n",
(*e)->info.sysid, ( engine_tag & 0xFF ),( (engine_tag >> 8) & 0xFF ), ipstr, sampler->info.mode ,sampler->info.interval );
if ( overwrite_sampling > 0 ) {
sampler->info.interval = overwrite_sampling;
syslog(LOG_INFO, "Process_v5: Hard overwrite sampling rate: %u\n", sampler->info.interval);
}
return (*e);
} // End of GetExporter
void Process_v5_v7(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs) {
netflow_v5_header_t *v5_header;
netflow_v5_record_t *v5_record;
exporter_v5_t *exporter;
extension_map_t *extension_map;
common_record_t *common_record;
uint64_t start_time, end_time, boot_time;
uint32_t First, Last;
uint16_t count;
uint8_t flags;
int i, done, version, flow_record_length;
ssize_t size_left;
char *string;
/*
* v7 is treated as v5. It differes only in the record length, for what we process.
*/
// map v5 data structure to input buffer
v5_header = (netflow_v5_header_t *)in_buff;
exporter = GetExporter(fs, v5_header);
if ( !exporter ) {
syslog(LOG_ERR,"Process_v5: Exporter NULL: Abort v5/v7 record processing");
return;
}
exporter->packets++;
// calculate record size depending on counter size
// sigh .. one day I should fix switch to 64bits
if ( exporter->sampler->info.interval == 1 ) {
flags = 0;
v5_output_record_size = v5_output_record_base_size + 8; // 2 x 4 byte counters
} else {
flags = 0;
SetFlag(flags, FLAG_SAMPLED);
SetFlag(flags, FLAG_PKG_64);
SetFlag(flags, FLAG_BYTES_64);
v5_output_record_size = v5_output_record_base_size + 16; // 2 x 8 byte counters
}
extension_map = exporter->extension_map;
version = ntohs(v5_header->version);
flow_record_length = version == 5 ? NETFLOW_V5_RECORD_LENGTH : NETFLOW_V7_RECORD_LENGTH;
// this many data to process
size_left = in_buff_cnt;
common_record = fs->nffile->buff_ptr;
done = 0;
while ( !done ) {
ipv4_block_t *ipv4_block;
/* Process header */
// count check
count = ntohs(v5_header->count);
if ( count > NETFLOW_V5_MAX_RECORDS ) {
syslog(LOG_ERR,"Process_v5: Unexpected record count in header: %i. Abort v5/v7 record processing", count);
fs->nffile->buff_ptr = (void *)common_record;
return;
}
// input buffer size check for all expected records
if ( size_left < ( NETFLOW_V5_HEADER_LENGTH + count * flow_record_length) ) {
syslog(LOG_ERR,"Process_v5: Not enough data to process v5 record. Abort v5/v7 record processing");
fs->nffile->buff_ptr = (void *)common_record;
return;
}
// output buffer size check for all expected records
if ( !CheckBufferSpace(fs->nffile, count * v5_output_record_size) ) {
// fishy! - should never happen. maybe disk full?
syslog(LOG_ERR,"Process_v5: output buffer size error. Abort v5/v7 record processing");
return;
}
// map output record to memory buffer
common_record = (common_record_t *)fs->nffile->buff_ptr;
ipv4_block = (ipv4_block_t *)common_record->data;
// sequence check
if ( exporter->first ) {
exporter->last_sequence = ntohl(v5_header->flow_sequence);
exporter->sequence = exporter->last_sequence;
exporter->first = 0;
} else {
exporter->last_sequence = exporter->sequence;
exporter->sequence = ntohl(v5_header->flow_sequence);
exporter->distance = exporter->sequence - exporter->last_sequence;
// handle overflow
if (exporter->distance < 0) {
exporter->distance = 0xffffffff + exporter->distance +1;
}
if (exporter->distance != exporter->last_count) {
#define delta(a,b) ( (a)>(b) ? (a)-(b) : (b)-(a) )
fs->nffile->stat_record->sequence_failure++;
exporter->sequence_failure++;
/*
syslog(LOG_ERR,"Flow v%d sequence last:%llu now:%llu mismatch. Missing: dist:%lu flows",
version, exporter->last_sequence, exporter->sequence, exporter->distance);
*/
}
}
exporter->last_count = count;
v5_header->SysUptime = ntohl(v5_header->SysUptime);
v5_header->unix_secs = ntohl(v5_header->unix_secs);
v5_header->unix_nsecs = ntohl(v5_header->unix_nsecs);
/* calculate boot time in msec */
boot_time = ((uint64_t)(v5_header->unix_secs)*1000 +
((uint64_t)(v5_header->unix_nsecs) / 1000000) ) - (uint64_t)(v5_header->SysUptime);
// process all records
v5_record = (netflow_v5_record_t *)((pointer_addr_t)v5_header + NETFLOW_V5_HEADER_LENGTH);
/* loop over each records associated with this header */
for (i = 0; i < count; i++) {
pointer_addr_t bsize;
uint64_t packets, bytes;
void *data_ptr;
uint8_t *s1, *s2;
int j, id;
// header data
common_record->flags = flags;
common_record->type = CommonRecordType;
common_record->exporter_sysid = exporter->info.sysid;;
common_record->ext_map = extension_map->map_id;
common_record->size = v5_output_record_size;
// v5 common fields
common_record->srcport = ntohs(v5_record->srcport);
common_record->dstport = ntohs(v5_record->dstport);
common_record->tcp_flags = v5_record->tcp_flags;
common_record->prot = v5_record->prot;
common_record->tos = v5_record->tos;
common_record->fwd_status = 0;
common_record->reserved = 0;
// v5 typed data as fixed struct v5_block
ipv4_block->srcaddr = ntohl(v5_record->srcaddr);
ipv4_block->dstaddr = ntohl(v5_record->dstaddr);
if ( exporter->sampler->info.interval == 1 ) {
value32_t *v = (value32_t *)ipv4_block->data;
packets = (uint64_t)ntohl(v5_record->dPkts);
bytes = (uint64_t)ntohl(v5_record->dOctets);
v->val = packets;
v = (value32_t *)v->data;
v->val = bytes;
data_ptr = (void *)v->data;
} else {
value64_t *v = (value64_t *)ipv4_block->data;
uint32_t *ptr = (uint32_t *)&packets;
packets = (uint64_t)ntohl(v5_record->dPkts) * (uint64_t)exporter->sampler->info.interval;
bytes = (uint64_t)ntohl(v5_record->dOctets) * (uint64_t)exporter->sampler->info.interval;
// pack packets in 32bit chunks
v->val.val32[0] = ptr[0];
v->val.val32[1] = ptr[1];
// pack bytes in 32bit chunks
v = (value64_t *)v->data;
ptr = (uint32_t *)&bytes;
v->val.val32[0] = ptr[0];
v->val.val32[1] = ptr[1];
data_ptr = (void *)v->data;
}
// process optional extensions
j = 0;
while ( (id = extension_map->ex_id[j]) != 0 ) {
switch (id) {
case EX_IO_SNMP_2: { // 2 byte input/output interface index
tpl_ext_4_t *tpl = (tpl_ext_4_t *)data_ptr;
tpl->input = ntohs(v5_record->input);
tpl->output = ntohs(v5_record->output);
data_ptr = (void *)tpl->data;
} break;
case EX_AS_2: { // 2 byte src/dst AS number
tpl_ext_6_t *tpl = (tpl_ext_6_t *)data_ptr;
tpl->src_as = ntohs(v5_record->src_as);
tpl->dst_as = ntohs(v5_record->dst_as);
data_ptr = (void *)tpl->data;
} break;
case EX_MULIPLE: { // dst tos, direction, src/dst mask
tpl_ext_8_t *tpl = (tpl_ext_8_t *)data_ptr;
tpl->dst_tos = 0;
tpl->dir = 0;
tpl->src_mask = v5_record->src_mask;
tpl->dst_mask = v5_record->dst_mask;
data_ptr = (void *)tpl->data;
} break;
case EX_NEXT_HOP_v4: { // IPv4 next hop
tpl_ext_9_t *tpl = (tpl_ext_9_t *)data_ptr;
tpl->nexthop = ntohl(v5_record->nexthop);
data_ptr = (void *)tpl->data;
} break;
case EX_ROUTER_IP_v4: { // IPv4 router address
tpl_ext_23_t *tpl = (tpl_ext_23_t *)data_ptr;
tpl->router_ip = fs->ip.v4;
data_ptr = (void *)tpl->data;
ClearFlag(common_record->flags, FLAG_IPV6_EXP);
} break;
case EX_ROUTER_ID: { // engine type, engine ID
tpl_ext_25_t *tpl = (tpl_ext_25_t *)data_ptr;
uint16_t engine_tag = ntohs(v5_header->engine_tag);
tpl->engine_type = (engine_tag >> 8) & 0xFF;
tpl->engine_id = (engine_tag & 0xFF);
data_ptr = (void *)tpl->data;
} break;
case EX_RECEIVED: {
tpl_ext_27_t *tpl = (tpl_ext_27_t *)data_ptr;
tpl->received = (uint64_t)((uint64_t)fs->received.tv_sec * 1000LL) + (uint64_t)((uint64_t)fs->received.tv_usec / 1000LL);
data_ptr = (void *)tpl->data;
} break;
default:
// this should never happen, as v5 has no other extensions
syslog(LOG_ERR,"Process_v5: Unexpected extension %i for v5 record. Skip extension", id);
}
j++;
}
// Time issues
First = ntohl(v5_record->First);
Last = ntohl(v5_record->Last);
#ifdef FIXTIMEBUG
/*
* Some users report, that they see flows, which have duration time of about 40days
* which is almost the overflow value. Investigating this, it cannot be an overflow
* and the difference is always 15160 or 15176 msec too little for a classical
* overflow. Therefore assume this must be an exporter bug
*/
if ( First > Last && ( (First - Last) < 20000) ) {
uint32_t _t;
syslog(LOG_ERR,"Process_v5: Unexpected time swap: First 0x%llx smaller than boot time: 0x%llx", start_time, boot_time);
_t= First;
First = Last;
Last = _t;
}
#endif
if ( First > Last ) {
/* First in msec, in case of msec overflow, between start and end */
start_time = boot_time - 0x100000000LL + (uint64_t)First;
} else {
start_time = boot_time + (uint64_t)First;
}
/* end time in msecs */
end_time = (uint64_t)Last + boot_time;
// if overflow happened after flow ended but before got exported
// the additional check > 100000 is required due to a CISCO IOS bug
// CSCei12353 - thanks to Bojan
if ( Last > v5_header->SysUptime && (( Last - v5_header->SysUptime) > 100000)) {
start_time -= 0x100000000LL;
end_time -= 0x100000000LL;
}
common_record->first = start_time/1000;
common_record->msec_first = start_time - common_record->first*1000;
common_record->last = end_time/1000;
common_record->msec_last = end_time - common_record->last*1000;
// update first_seen, last_seen
if ( start_time < fs->first_seen )
fs->first_seen = start_time;
if ( end_time > fs->last_seen )
fs->last_seen = end_time;
// Update stats
switch (common_record->prot) {
case IPPROTO_ICMP:
fs->nffile->stat_record->numflows_icmp++;
fs->nffile->stat_record->numpackets_icmp += packets;
fs->nffile->stat_record->numbytes_icmp += bytes;
// fix odd CISCO behaviour for ICMP port/type in src port
if ( common_record->srcport != 0 ) {
s1 = (uint8_t *)&(common_record->srcport);
s2 = (uint8_t *)&(common_record->dstport);
s2[0] = s1[1];
s2[1] = s1[0];
common_record->srcport = 0;
}
break;
case IPPROTO_TCP:
fs->nffile->stat_record->numflows_tcp++;
fs->nffile->stat_record->numpackets_tcp += packets;
fs->nffile->stat_record->numbytes_tcp += bytes;
break;
case IPPROTO_UDP:
fs->nffile->stat_record->numflows_udp++;
fs->nffile->stat_record->numpackets_udp += packets;
fs->nffile->stat_record->numbytes_udp += bytes;
break;
default:
fs->nffile->stat_record->numflows_other++;
fs->nffile->stat_record->numpackets_other += packets;
fs->nffile->stat_record->numbytes_other += bytes;
}
exporter->flows++;
fs->nffile->stat_record->numflows++;
fs->nffile->stat_record->numpackets += packets;
fs->nffile->stat_record->numbytes += bytes;
if ( fs->xstat ) {
uint32_t bpp = packets ? (bytes/packets) : 0;
if ( bpp > MAX_BPP )
bpp = MAX_BPP;
if ( common_record->prot == IPPROTO_TCP ) {
fs->xstat->bpp_histogram->tcp.bpp[bpp]++;
fs->xstat->bpp_histogram->tcp.count++;
fs->xstat->port_histogram->src_tcp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_tcp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_tcp.count++;
fs->xstat->port_histogram->dst_tcp.count++;
} else if ( common_record->prot == IPPROTO_UDP ) {
fs->xstat->bpp_histogram->udp.bpp[bpp]++;
fs->xstat->bpp_histogram->udp.count++;
fs->xstat->port_histogram->src_udp.port[common_record->srcport]++;
fs->xstat->port_histogram->dst_udp.port[common_record->dstport]++;
fs->xstat->port_histogram->src_udp.count++;
fs->xstat->port_histogram->dst_udp.count++;
}
}
if ( verbose ) {
master_record_t master_record;
ExpandRecord_v2((common_record_t *)common_record, &v5_extension_info, &(exporter->info), &master_record);
format_file_block_record(&master_record, &string, 0);
printf("%s\n", string);
}
// advance to next input flow record
v5_record = (netflow_v5_record_t *)((pointer_addr_t)v5_record + flow_record_length);
if ( ((pointer_addr_t)data_ptr - (pointer_addr_t)common_record) != v5_output_record_size ) {
printf("Panic size check: ptr diff: %llu, record size: %u\n",
(unsigned long long)((pointer_addr_t)data_ptr - (pointer_addr_t)common_record), v5_output_record_size );
abort();
}
// advance to next output record
common_record = (common_record_t *)data_ptr;
ipv4_block = (ipv4_block_t *)common_record->data;
// buffer size sanity check - should never happen, but check it anyway
bsize = (pointer_addr_t)common_record - (pointer_addr_t)fs->nffile->block_header - sizeof(data_block_header_t);
if ( bsize >= BUFFSIZE ) {
syslog(LOG_ERR,"### Software error ###: %s line %d", __FILE__, __LINE__);
syslog(LOG_ERR,"Process_v5: Output buffer overflow! Flush buffer and skip records.");
syslog(LOG_ERR,"Buffer size: size: %u, bsize: %llu > %u", fs->nffile->block_header->size, (unsigned long long)bsize, BUFFSIZE);
// reset buffer
fs->nffile->block_header->size = 0;
fs->nffile->block_header->NumRecords = 0;
fs->nffile->buff_ptr = (void *)((pointer_addr_t)fs->nffile->block_header + sizeof(data_block_header_t) );
return;
}
} // End of foreach v5 record
// update file record size ( -> output buffer size )
fs->nffile->block_header->NumRecords += count;
fs->nffile->block_header->size += count * v5_output_record_size;
fs->nffile->buff_ptr = (void *)common_record;
// still to go for this many input bytes
size_left -= NETFLOW_V5_HEADER_LENGTH + count * flow_record_length;
// next header
v5_header = (netflow_v5_header_t *)v5_record;
// should never be < 0
done = size_left <= 0;
} // End of while !done
return;
} /* End of Process_v5 */
/*
* functions used for sending netflow v5 records
*/
void Init_v5_v7_output(send_peer_t *peer) {
v5_output_header = (netflow_v5_header_t *)peer->send_buffer;
v5_output_header->version = htons(5);
v5_output_header->SysUptime = 0;
v5_output_header->unix_secs = 0;
v5_output_header->unix_nsecs = 0;
v5_output_header->count = 0;
output_engine.first = 1;
output_engine.sequence = 0;
output_engine.last_sequence = 0;
output_engine.last_count = 0;
output_engine.sequence_failure = 0;
v5_output_record = (netflow_v5_record_t *)((pointer_addr_t)v5_output_header + (pointer_addr_t)sizeof(netflow_v5_header_t));
} // End of Init_v5_v7_output
int Add_v5_output_record(master_record_t *master_record, send_peer_t *peer) {
static uint64_t boot_time; // in msec
static int cnt;
extension_map_t *extension_map = master_record->map_ref;
uint32_t i, id, t1, t2;
// Skip IPv6 records
if ( (master_record->flags & FLAG_IPV6_ADDR ) != 0 )
return 0;
if ( output_engine.first ) { // first time a record is added
// boot time is set one day back - assuming that the start time of every flow does not start ealier
boot_time = (uint64_t)(master_record->first - 86400)*1000;
v5_output_header->unix_secs = htonl(master_record->first - 86400);
cnt = 0;
output_engine.first = 0;
}
if ( cnt == 0 ) {
peer->buff_ptr = (void *)((pointer_addr_t)peer->send_buffer + NETFLOW_V5_HEADER_LENGTH);
v5_output_record = (netflow_v5_record_t *)((pointer_addr_t)v5_output_header + (pointer_addr_t)sizeof(netflow_v5_header_t));
output_engine.sequence = output_engine.last_sequence + output_engine.last_count;
v5_output_header->flow_sequence = htonl(output_engine.sequence);
output_engine.last_sequence = output_engine.sequence;
}
t1 = (uint32_t)(1000LL * (uint64_t)master_record->first + (uint64_t)master_record->msec_first - boot_time);
t2 = (uint32_t)(1000LL * (uint64_t)master_record->last + (uint64_t)master_record->msec_last - boot_time);
v5_output_record->First = htonl(t1);
v5_output_record->Last = htonl(t2);
v5_output_record->srcaddr = htonl(master_record->v4.srcaddr);
v5_output_record->dstaddr = htonl(master_record->v4.dstaddr);
v5_output_record->srcport = htons(master_record->srcport);
v5_output_record->dstport = htons(master_record->dstport);
v5_output_record->tcp_flags = master_record->tcp_flags;
v5_output_record->prot = master_record->prot;
v5_output_record->tos = master_record->tos;
// the 64bit counters are cut down to 32 bits for v5
v5_output_record->dPkts = htonl((uint32_t)master_record->dPkts);
v5_output_record->dOctets = htonl((uint32_t)master_record->dOctets);
v5_output_record->input = 0;
v5_output_record->output = 0;
v5_output_record->src_as = 0;
v5_output_record->dst_as = 0;
v5_output_record->src_mask = 0;
v5_output_record->dst_mask = 0;
v5_output_record->pad1 = 0;
v5_output_record->pad2 = 0;
v5_output_record->nexthop = 0;
i = 0;
while ( (id = extension_map->ex_id[i]) != 0 ) {
switch (id) {
case EX_IO_SNMP_2:
v5_output_record->input = htons(master_record->input);
v5_output_record->output = htons(master_record->output);
break;
case EX_AS_2:
v5_output_record->src_as = htons(master_record->srcas);
v5_output_record->dst_as = htons(master_record->dstas);
break;
case EX_MULIPLE:
v5_output_record->src_mask = master_record->src_mask;
v5_output_record->dst_mask = master_record->dst_mask;
break;
case EX_NEXT_HOP_v4:
v5_output_record->nexthop = htonl(master_record->ip_nexthop.v4);
break;
// default: Other extensions can not be sent with v5
}
i++;
}
cnt++;
v5_output_header->count = htons(cnt);
peer->buff_ptr = (void *)((pointer_addr_t)peer->buff_ptr + NETFLOW_V5_RECORD_LENGTH);
v5_output_record++;
if ( cnt == NETFLOW_V5_MAX_RECORDS ) {
peer->flush = 1;
output_engine.last_count = cnt;
cnt = 0;
}
return 0;
} // End of Add_v5_output_record

149
bin/netflow_v5_v7.h Normal file
View File

@ -0,0 +1,149 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: netflow_v5_v7.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NETFLOW_V5_V7_H
#define _NETFLOW_V5_V7_H 1
#define NETFLOW_V5_HEADER_LENGTH 24
#define NETFLOW_V5_RECORD_LENGTH 48
#define NETFLOW_V5_MAX_RECORDS 30
#define NETFLOW_V7_HEADER_LENGTH 24
#define NETFLOW_V7_RECORD_LENGTH 52
#define NETFLOW_V7_MAX_RECORDS 28
/* v5 structures */
typedef struct netflow_v5_header {
uint16_t version;
uint16_t count;
uint32_t SysUptime;
uint32_t unix_secs;
uint32_t unix_nsecs;
uint32_t flow_sequence;
uint16_t engine_tag;
uint16_t sampling_interval;
} netflow_v5_header_t;
typedef struct netflow_v5_record {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t nexthop;
uint16_t input;
uint16_t output;
uint32_t dPkts;
uint32_t dOctets;
uint32_t First;
uint32_t Last;
uint16_t srcport;
uint16_t dstport;
uint8_t pad1;
uint8_t tcp_flags;
uint8_t prot;
uint8_t tos;
uint16_t src_as;
uint16_t dst_as;
uint8_t src_mask;
uint8_t dst_mask;
uint16_t pad2;
} netflow_v5_record_t;
/* v7 structures */
typedef struct netflow_v7_header {
uint16_t version;
uint16_t count;
uint32_t SysUptime;
uint32_t unix_secs;
uint32_t unix_nsecs;
uint32_t flow_sequence;
uint32_t reserved;
} netflow_v7_header_t;
typedef struct netflow_v7_record {
uint32_t srcaddr;
uint32_t dstaddr;
uint32_t nexthop;
uint16_t input;
uint16_t output;
uint32_t dPkts;
uint32_t dOctets;
uint32_t First;
uint32_t Last;
uint16_t srcport;
uint16_t dstport;
uint8_t flags;
uint8_t tcp_flags;
uint8_t prot;
uint8_t tos;
uint16_t src_as;
uint16_t dst_as;
uint8_t src_mask;
uint8_t dst_mask;
uint16_t pad;
uint32_t router_sc;
} netflow_v7_record_t;
/* prototypes */
int Init_v5_v7_input(void);
void Process_v5_v7(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs);
void Init_v5_v7_output(send_peer_t *peer);
int Add_v5_output_record(master_record_t *master_record, send_peer_t *peer);
/*
* Extension map for v5/v7
*
* Required extensions:
*
* 4 byte byte counter
* | 4byte packet counter
* | | IPv4
* | | |
* xxxx x0 0 0
*
* Optional extensions:
*
* 4 : 2 byte input/output interface id
* 6 : 2 byte src/dst as
* 8 : srcmask/dst mask dst tos = 0, dir = 0
* 9 : IPv4 next hop
*/
#endif //_NETFLOW_V5_V7_H

3064
bin/netflow_v9.c Normal file
View File

File diff suppressed because it is too large Load Diff

309
bin/netflow_v9.h Normal file
View File

@ -0,0 +1,309 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: netflow_v9.h 51 2010-01-29 09:01:54Z haag $
*
* $LastChangedRevision: 51 $
*
*/
/* v9 structures */
/* Packet Header Field Descriptions
*
* Version
* Version of Flow Record format exported in this packet. The
* value of this field is 9 for the current version.
*
* Count
* The total number of records in the Export Packet, which is the
* sum of Options FlowSet records, Template FlowSet records, and
* Data FlowSet records.
*
* sysUpTime
* Time in milliseconds since this device was first booted.
*
* UNIX Secs
* Time in seconds since 0000 UTC 1970, at which the Export Packet
* leaves the Exporter.
*
* Sequence Number
* Incremental sequence counter of all Export Packets sent from
* the current Observation Domain by the Exporter. This value
* MUST be cumulative, and SHOULD be used by the Collector to
* identify whether any Export Packets have been missed.
*
* Source ID
* A 32-bit value that identifies the Exporter Observation Domain.
* NetFlow Collectors SHOULD use the combination of the source IP
* address and the Source ID field to separate different export
* streams originating from the same Exporter.
*/
#ifndef _NETFLOW_V9_H
#define _NETFLOW_V9_H 1
typedef struct netflow_v9_header {
uint16_t version;
uint16_t count;
uint32_t SysUptime;
uint32_t unix_secs;
uint32_t sequence;
uint32_t source_id;
} netflow_v9_header_t;
#define NETFLOW_V9_HEADER_LENGTH sizeof(netflow_v9_header_t)
/* FlowSet ID
* FlowSet ID value of 0 is reserved for the Template FlowSet.
* Length
* Total length of this FlowSet. Because an individual Template
* FlowSet MAY contain multiple Template Records, the Length value
* MUST be used to determine the position of the next FlowSet
* record, which could be any type of FlowSet. Length is the sum
* of the lengths of the FlowSet ID, the Length itself, and all
* Template Records within this FlowSet.
*
* Template ID
* Each of the newly generated Template Records is given a unique
* Template ID. This uniqueness is local to the Observation
* Domain that generated the Template ID. Template IDs 0-255 are
* reserved for Template FlowSets, Options FlowSets, and other
* reserved FlowSets yet to be created. Template IDs of Data
* FlowSets are numbered from 256 to 65535.
*
* Field Count
* Number of fields in this Template Record. Because a Template
* FlowSet usually contains multiple Template Records, this field
* allows the Collector to determine the end of the current
* Template Record and the start of the next.
*
* Field Type
* A numeric value that represents the type of the field. Refer
* to the "Field Type Definitions" section.
*
* Field Length
* The length of the corresponding Field Type, in bytes. Refer to
* the "Field Type Definitions" section.
*/
typedef struct template_record_s {
uint16_t template_id;
uint16_t count;
struct {
uint16_t type;
uint16_t length;
} record[1];
} template_record_t;
typedef struct template_flowset_s {
uint16_t flowset_id;
uint16_t length;
template_record_t fields[1];
} template_flowset_t;
typedef struct data_flowset_s {
uint16_t flowset_id;
uint16_t length;
uint8_t data[4];
} data_flowset_t;
typedef struct option_template_flowset_s {
uint16_t flowset_id;
uint16_t length;
uint16_t template_id;
uint16_t option_scope_length;
uint16_t option_length;
struct {
uint16_t type;
uint16_t length;
} record[1];
} option_template_flowset_t;
typedef struct common_header_s {
uint16_t flowset_id;
uint16_t length;
} common_header_t;
#define _1byte 1
#define _2bytes 2
#define _3bytes 3
#define _4bytes 4
#define _6bytes 6
#define _8bytes 8
#define _12bytes 12
#define _16bytes 16
#define _20bytes 20
#define _24bytes 24
#define _65bytes 65
#define _72bytes 72
#define NF9_TEMPLATE_FLOWSET_ID 0
#define NF9_OPTIONS_FLOWSET_ID 1
#define NF9_MIN_RECORD_FLOWSET_ID 256
// Flowset record types
#define NF9_IN_BYTES 1
#define NF9_IN_PACKETS 2
#define NF9_FLOWS_AGGR 3
#define NF9_IN_PROTOCOL 4
#define NF9_SRC_TOS 5
#define NF9_TCP_FLAGS 6
#define NF9_L4_SRC_PORT 7
#define NF9_IPV4_SRC_ADDR 8
#define NF9_SRC_MASK 9
#define NF9_INPUT_SNMP 10
#define NF9_L4_DST_PORT 11
#define NF9_IPV4_DST_ADDR 12
#define NF9_DST_MASK 13
#define NF9_OUTPUT_SNMP 14
#define NF9_V4_NEXT_HOP 15
#define NF9_SRC_AS 16
#define NF9_DST_AS 17
#define NF9_BGP_V4_NEXT_HOP 18
#define NF9_LAST_SWITCHED 21
#define NF9_FIRST_SWITCHED 22
#define NF9_OUT_BYTES 23
#define NF9_OUT_PKTS 24
#define NF9_IPV6_SRC_ADDR 27
#define NF9_IPV6_DST_ADDR 28
#define NF9_IPV6_SRC_MASK 29
#define NF9_IPV6_DST_MASK 30
#define NF9_IPV6_FLOW_LABEL 31
#define NF9_ICMP_TYPE 32
#define NF9_SAMPLING_INTERVAL 34
#define NF9_SAMPLING_ALGORITHM 35
#define NF9_ENGINE_TYPE 38
#define NF9_ENGINE_ID 39
#define NF9_FLOW_SAMPLER_ID 48
#define FLOW_SAMPLER_MODE 49
#define NF9_FLOW_SAMPLER_RANDOM_INTERVAL 50
// #define NF9_MIN_TTL 52
// #define NF9_MAX_TTL 53
// #define NF9_IPV4_IDENT 54
#define NF9_DST_TOS 55
#define NF9_IN_SRC_MAC 56
#define NF9_OUT_DST_MAC 57
#define NF9_SRC_VLAN 58
#define NF9_DST_VLAN 59
#define NF9_DIRECTION 61
#define NF9_V6_NEXT_HOP 62
#define NF9_BPG_V6_NEXT_HOP 63
// #define NF9_V6_OPTION_HEADERS 64
#define NF9_MPLS_LABEL_1 70
#define NF9_MPLS_LABEL_2 71
#define NF9_MPLS_LABEL_3 72
#define NF9_MPLS_LABEL_4 73
#define NF9_MPLS_LABEL_5 74
#define NF9_MPLS_LABEL_6 75
#define NF9_MPLS_LABEL_7 76
#define NF9_MPLS_LABEL_8 77
#define NF9_MPLS_LABEL_9 78
#define NF9_MPLS_LABEL_10 79
#define NF9_IN_DST_MAC 80
#define NF9_OUT_SRC_MAC 81
#define NF9_FORWARDING_STATUS 89
#define NF9_BGP_ADJ_NEXT_AS 128
#define NF9_BGP_ADJ_PREV_AS 129
// CISCO ASA NSEL extension - Network Security Event Logging
#define NF_F_FLOW_BYTES 85
#define NF_F_CONN_ID 148
#define NF_F_FLOW_CREATE_TIME_MSEC 152
#define NF_F_FLOW_END_TIME_MSEC 153
#define NF_F_ICMP_TYPE 176
#define NF_F_ICMP_CODE 177
#define NF_F_ICMP_TYPE_IPV6 178
#define NF_F_ICMP_CODE_IPV6 179
#define NF_F_FWD_FLOW_DELTA_BYTES 231
#define NF_F_REV_FLOW_DELTA_BYTES 232
#define NF_F_EVENT_TIME_MSEC 323
#define NF_F_INGRESS_ACL_ID 33000
#define NF_F_EGRESS_ACL_ID 33001
#define NF_F_FW_EXT_EVENT 33002
#define NF_F_USERNAME 40000
#define NF_F_XLATE_SRC_ADDR_IPV4 225
#define NF_F_XLATE_DST_ADDR_IPV4 226
#define NF_F_XLATE_SRC_PORT 227
#define NF_F_XLATE_DST_PORT 228
#define NF_F_XLATE_SRC_ADDR_IPV6 281
#define NF_F_XLATE_DST_ADDR_IPV6 282
#define NF_F_FW_EVENT 233
// ASA 8.4 compat elements
#define NF_F_XLATE_SRC_ADDR_84 40001
#define NF_F_XLATE_DST_ADDR_84 40002
#define NF_F_XLATE_SRC_PORT_84 40003
#define NF_F_XLATE_DST_PORT_84 40004
#define NF_F_FW_EVENT_84 40005
// Cisco ASR 1000 series NEL extension - Nat Event Logging
#define NF_N_NAT_EVENT 230
#define NF_N_INGRESS_VRFID 234
#define NF_N_EGRESS_VRFID 235
#define NF_F_XLATE_PORT_BLOCK_START 361
#define NF_F_XLATE_PORT_BLOCK_END 362
#define NF_F_XLATE_PORT_BLOCK_STEP 363
#define NF_F_XLATE_PORT_BLOCK_SIZE 364
// nprobe latency extensions
#define NF9_NPROBE_CLIENT_NW_DELAY_SEC 57554
#define NF9_NPROBE_CLIENT_NW_DELAY_USEC 57555
#define NF9_NPROBE_SERVER_NW_DELAY_SEC 57556
#define NF9_NPROBE_SERVER_NW_DELAY_USEC 57557
#define NF9_NPROBE_APPL_LATENCY_SEC 57558
#define NF9_NPROBE_APPL_LATENCY_USEC 57559
/* prototypes */
int Init_v9(void);
void Process_v9(void *in_buff, ssize_t in_buff_cnt, FlowSource_t *fs);
void Init_v9_output(send_peer_t *peer);
int Add_v9_output_record(master_record_t *master_record, send_peer_t *peer);
#endif //_NETFLOW_V9_H 1

2887
bin/nf_common.c Normal file
View File

File diff suppressed because it is too large Load Diff

122
bin/nf_common.h Normal file
View File

@ -0,0 +1,122 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nf_common.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
#ifndef _NF_COMMON_H
#define _NF_COMMON_H 1
typedef void (*printer_t)(void *, char **, int);
#if ( SIZEOF_VOID_P == 8 )
typedef uint64_t pointer_addr_t;
#else
typedef uint32_t pointer_addr_t;
#endif
typedef struct msec_time_s {
time_t sec;
uint16_t msec;
} msec_time_tt;
/* common minimum netflow header for all versions */
typedef struct common_flow_header {
uint16_t version;
uint16_t count;
} common_flow_header_t;
typedef struct printmap_s {
char *printmode; // name of the output format
printer_t func; // name of the function, which prints the record
char *Format; // output format definition
} printmap_t;
#define NSEL_EVENT_IGNORE 0LL
#define NSEL_EVENT_CREATE 1LL
#define NSEL_EVENT_DELETE 2LL
#define NSEL_EVENT_DENIED 3LL
#define NSEL_EVENT_ALERT 4LL
#define NSEL_EVENT_UPDATE 5LL
#define NEL_EVENT_INVALID 0LL
#define NEL_EVENT_ADD 1LL
#define NEL_EVENT_DELETE 2LL
/* prototypes */
int InitSymbols(void);
void Setv6Mode(int mode);
int Getv6Mode(void);
int Proto_num(char *protostr);
void format_file_block_header(void *header, char **s, int tag);
char *format_csv_header(void);
char *get_record_header(void);
void set_record_header(void);
void format_file_block_record(void *record, char **s, int tag);
void flow_record_to_pipe(void *record, char ** s, int tag);
void flow_record_to_csv(void *record, char ** s, int tag);
void flow_record_to_null(void *record, char ** s, int tag);
int ParseOutputFormat(char *format, int plain_numbers, printmap_t *printmap);
void format_special(void *record, char ** s, int tag);
uint32_t Get_fwd_status_id(char *status);
char *Get_fwd_status_name(uint32_t id);
void Proto_string(uint8_t protonum, char *protostr);
void condense_v6(char *s);
#define TAG_CHAR ''
#endif //_NF_COMMON_H

483
bin/nfanon.c Executable file
View File

@ -0,0 +1,483 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id:$
*
* $LastChangedRevision: 48 $
*
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdarg.h>
#include <errno.h>
#include <time.h>
#include <string.h>
#include <ctype.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/param.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <fcntl.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "nfx.h"
#include "util.h"
#include "flist.h"
#include "panonymizer.h"
#if ( SIZEOF_VOID_P == 8 )
typedef uint64_t pointer_addr_t;
#else
typedef uint32_t pointer_addr_t;
#endif
// module limited globals
extension_map_list_t *extension_map_list;
/* Function Prototypes */
static void usage(char *name);
static inline void AnonRecord(master_record_t *master_record);
static void process_data(void *wfile);
/* Functions */
#define NEED_PACKRECORD 1
#include "nffile_inline.c"
#undef NEED_PACKRECORD
static void usage(char *name) {
printf("usage %s [options] \n"
"-h\t\tthis text you see right here\n"
"-K <key>\tAnonymize IP addressses using CryptoPAn with key <key>.\n"
"-r\t\tread input from file\n"
"-M <expr>\tRead input from multiple directories.\n"
"-R <expr>\tRead input from sequence of files.\n"
"-w <file>\tName of output file. Defaults to input file.\n"
, name);
} /* usage */
static inline void AnonRecord(master_record_t *master_record) {
extension_map_t *extension_map = master_record->map_ref;
int i;
// Required extension 1 - IP addresses
if ( (master_record->flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6
// IPv6
uint64_t anon_ip[2];
anonymize_v6(master_record->v6.srcaddr, anon_ip);
master_record->v6.srcaddr[0] = anon_ip[0];
master_record->v6.srcaddr[1] = anon_ip[1];
anonymize_v6(master_record->v6.dstaddr, anon_ip);
master_record->v6.dstaddr[0] = anon_ip[0];
master_record->v6.dstaddr[1] = anon_ip[1];
} else {
// IPv4
master_record->v4.srcaddr = anonymize(master_record->v4.srcaddr);
master_record->v4.dstaddr = anonymize(master_record->v4.dstaddr);
}
// Process optional extensions
i=0;
while ( extension_map->ex_id[i] ) {
switch (extension_map->ex_id[i++]) {
case EX_AS_2: // srcas/dstas 2 byte
master_record->srcas = 0;
master_record->dstas = 0;
break;
case EX_AS_4: // srcas/dstas 4 byte
master_record->srcas = 0;
master_record->dstas = 0;
break;
case EX_NEXT_HOP_v4:
master_record->ip_nexthop.v4 = anonymize(master_record->ip_nexthop.v4);
break;
case EX_NEXT_HOP_v6: {
uint64_t anon_ip[2];
anonymize_v6(master_record->ip_nexthop.v6, anon_ip);
master_record->ip_nexthop.v6[0] = anon_ip[0];
master_record->ip_nexthop.v6[1] = anon_ip[1];
} break;
case EX_NEXT_HOP_BGP_v4:
master_record->bgp_nexthop.v4 = anonymize(master_record->bgp_nexthop.v4);
break;
case EX_NEXT_HOP_BGP_v6: {
uint64_t anon_ip[2];
anonymize_v6(master_record->bgp_nexthop.v6, anon_ip);
master_record->bgp_nexthop.v6[0] = anon_ip[0];
master_record->bgp_nexthop.v6[1] = anon_ip[1];
} break;
case EX_ROUTER_IP_v4:
master_record->ip_router.v4 = anonymize(master_record->ip_router.v4);
break;
case EX_ROUTER_IP_v6: {
uint64_t anon_ip[2];
anonymize_v6(master_record->ip_router.v6, anon_ip);
master_record->ip_router.v6[0] = anon_ip[0];
master_record->ip_router.v6[1] = anon_ip[1];
} break;
#ifdef NSEL
case EX_NSEL_XLATE_IP_v4:
master_record->xlate_src_ip.v4 = anonymize(master_record->xlate_src_ip.v4);
master_record->xlate_dst_ip.v4 = anonymize(master_record->xlate_dst_ip.v4);
break;
case EX_NSEL_XLATE_IP_v6: {
uint64_t anon_ip[2];
anonymize_v6(master_record->xlate_src_ip.v6, anon_ip);
master_record->xlate_src_ip.v6[0] = anon_ip[0];
master_record->xlate_src_ip.v6[1] = anon_ip[1];
anonymize_v6(master_record->xlate_dst_ip.v6, anon_ip);
master_record->xlate_dst_ip.v6[0] = anon_ip[0];
master_record->xlate_dst_ip.v6[1] = anon_ip[1];
} break;
#endif
}
}
} // End of AnonRecord
static void process_data(void *wfile) {
master_record_t master_record;
common_record_t *flow_record;
nffile_t *nffile_r;
nffile_t *nffile_w;
int i, done, ret, cnt, verbose;
char outfile[MAXPATHLEN], *cfile;
#ifdef COMPAT15
int v1_map_done = 0;
#endif
setbuf(stderr, NULL);
cnt = 1;
verbose = 1;
// Get the first file handle
nffile_r = GetNextFile(NULL, 0, 0);
if ( !nffile_r ) {
LogError("GetNextFile() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
if ( nffile_r == EMPTY_LIST ) {
LogError("Empty file list. No files to process\n");
return;
}
cfile = GetCurrentFilename();
if ( !cfile ) {
if ( nffile_r->fd == 0 ) { // stdin
outfile[0] = '-';
outfile[1] = '\0';
verbose = 0;
} else {
LogError("(NULL) input file name error in %s line %d\n", __FILE__, __LINE__);
return;
}
} else {
// prepare output file
snprintf(outfile,MAXPATHLEN-1, "%s-tmp", cfile);
outfile[MAXPATHLEN-1] = '\0';
if ( verbose )
fprintf(stderr, " %i Processing %s\r", cnt++, cfile);
}
if ( wfile )
nffile_w = OpenNewFile(wfile, NULL, FILE_IS_COMPRESSED(nffile_r), 1, NULL);
else
nffile_w = OpenNewFile(outfile, NULL, FILE_IS_COMPRESSED(nffile_r), 1, NULL);
if ( !nffile_w ) {
if ( nffile_r ) {
CloseFile(nffile_r);
DisposeFile(nffile_r);
}
return;
}
memcpy((void *)nffile_w->stat_record, (void *)nffile_r->stat_record, sizeof(stat_record_t));
done = 0;
while ( !done ) {
// get next data block from file
ret = ReadBlock(nffile_r);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
LogError("Skip corrupt data file '%s'\n",GetCurrentFilename());
else
LogError("Read error in file '%s': %s\n",GetCurrentFilename(), strerror(errno) );
// fall through - get next file in chain
case NF_EOF: {
nffile_t *next;
if ( nffile_w->block_header->NumRecords ) {
if ( WriteBlock(nffile_w) <= 0 ) {
LogError("Failed to write output buffer to disk: '%s'" , strerror(errno));
}
}
if ( wfile == NULL ) {
CloseUpdateFile(nffile_w, nffile_r->file_header->ident);
if ( rename(outfile, cfile) < 0 ) {
LogError("\nrename() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
LogError("Abort processing.\n");
return;
}
}
next = GetNextFile(nffile_r, 0, 0);
if ( next == EMPTY_LIST ) {
done = 1;
continue;
}
if ( next == NULL ) {
LogError("Unexpected end of file list\n");
done = 1;
continue;
}
cfile = GetCurrentFilename();
if ( !cfile ) {
LogError("(NULL) input file name error in %s line %d\n", __FILE__, __LINE__);
return;
}
LogError(" %i Processing %s\r", cnt++, cfile);
if ( wfile == NULL ) {
snprintf(outfile,MAXPATHLEN-1, "%s-tmp", cfile);
outfile[MAXPATHLEN-1] = '\0';
nffile_w = OpenNewFile(outfile, nffile_w, FILE_IS_COMPRESSED(nffile_r), 1, NULL);
if ( !nffile_w ) {
if ( nffile_r ) {
CloseFile(nffile_r);
DisposeFile(nffile_r);
}
return;
}
memcpy((void *)nffile_w->stat_record, (void *)&nffile_r->stat_record, sizeof(stat_record_t));
} else {
SumStatRecords(nffile_w->stat_record, nffile_r->stat_record);
}
// continue with next file
continue;
} break; // not really needed
}
#ifdef COMPAT15
if ( nffile_r->block_header->id == DATA_BLOCK_TYPE_1 ) {
common_record_v1_t *v1_record = (common_record_v1_t *)nffile_r->buff_ptr;
// create an extension map for v1 blocks
if ( v1_map_done == 0 ) {
extension_map_t *map = malloc(sizeof(extension_map_t) + 2 * sizeof(uint16_t) );
if ( ! map ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
map->type = ExtensionMapType;
map->size = sizeof(extension_map_t) + 2 * sizeof(uint16_t);
map->map_id = 0;
map->ex_id[0] = EX_IO_SNMP_2;
map->ex_id[1] = EX_AS_2;
map->ex_id[2] = 0;
Insert_Extension_Map(extension_map_list, map);
AppendToBuffer(nffile_w, (void *)map, map->size);
v1_map_done = 1;
}
// convert the records to v2
for ( i=0; i < nffile_r->block_header->NumRecords; i++ ) {
common_record_t *v2_record = (common_record_t *)v1_record;
Convert_v1_to_v2((void *)v1_record);
// now we have a v2 record -> use size of v2_record->size
v1_record = (common_record_v1_t *)((pointer_addr_t)v1_record + v2_record->size);
}
nffile_r->block_header->id = DATA_BLOCK_TYPE_2;
}
#endif
if ( nffile_r->block_header->id == Large_BLOCK_Type ) {
// skip
continue;
}
if ( nffile_r->block_header->id != DATA_BLOCK_TYPE_2 ) {
fprintf(stderr, "Can't process block type %u. Skip block.\n", nffile_r->block_header->id);
continue;
}
flow_record = nffile_r->buff_ptr;
for ( i=0; i < nffile_r->block_header->NumRecords; i++ ) {
switch ( flow_record->type ) {
case CommonRecordV0Type:
case CommonRecordType: {
uint32_t map_id = flow_record->ext_map;
if ( extension_map_list->slot[map_id] == NULL ) {
LogError("Corrupt data file! No such extension map id: %u. Skip record", flow_record->ext_map );
} else {
ExpandRecord_v2( flow_record, extension_map_list->slot[flow_record->ext_map], NULL, &master_record);
// update number of flows matching a given map
extension_map_list->slot[map_id]->ref_count++;
AnonRecord(&master_record);
PackRecord(&master_record, nffile_w);
}
} break;
case ExtensionMapType: {
extension_map_t *map = (extension_map_t *)flow_record;
if ( Insert_Extension_Map(extension_map_list, map) ) {
// flush new map
} // else map already known and flushed
AppendToBuffer(nffile_w, (void *)map, map->size);
} break;
case ExporterRecordType:
case SamplerRecordype:
case ExporterInfoRecordType:
case ExporterStatRecordType:
case SamplerInfoRecordype:
// Silently skip exporter/sampler records
break;
default: {
fprintf(stderr, "Skip unknown record type %i\n", flow_record->type);
}
}
// Advance pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
} // for all records
} // while
PackExtensionMapList(extension_map_list);
if ( wfile != NULL )
CloseUpdateFile(nffile_w, nffile_r->file_header->ident);
if ( nffile_r ) {
CloseFile(nffile_r);
DisposeFile(nffile_r);
}
DisposeFile(nffile_w);
LogError("\n");
LogError("Processed %i files.\n", --cnt);
} // End of process_data
int main( int argc, char **argv ) {
char *rfile, *Rfile, *wfile, *Mdirs;
int c;
char CryptoPAnKey[32];
rfile = Rfile = Mdirs = wfile = NULL;
while ((c = getopt(argc, argv, "K:L:r:M:R:w:")) != EOF) {
switch (c) {
case 'h':
usage(argv[0]);
exit(0);
break;
break;
case 'K':
if ( !ParseCryptoPAnKey(optarg, CryptoPAnKey) ) {
fprintf(stderr, "Invalid key '%s' for CryptoPAn!\n", optarg);
exit(255);
}
PAnonymizer_Init((uint8_t *)CryptoPAnKey);
break;
case 'L':
if ( !InitLog("argv[0]", optarg) )
exit(255);
break;
case 'r':
rfile = optarg;
if ( strcmp(rfile, "-") == 0 )
rfile = NULL;
break;
case 'M':
Mdirs = optarg;
break;
case 'R':
Rfile = optarg;
break;
case 'w':
wfile = optarg;
break;
default:
usage(argv[0]);
exit(0);
}
}
if ( rfile && Rfile ) {
fprintf(stderr, "-r and -R are mutually exclusive. Please specify either -r or -R\n");
exit(255);
}
if ( Mdirs && !(rfile || Rfile) ) {
fprintf(stderr, "-M needs either -r or -R to specify the file or file list. Add '-R .' for all files in the directories.\n");
exit(255);
}
extension_map_list = InitExtensionMaps(NEEDS_EXTENSION_LIST);
SetupInputFileSequence(Mdirs, rfile, Rfile);
process_data(wfile);
FreeExtensionMaps(extension_map_list);
return 0;
}

1236
bin/nfcapd.c Normal file
View File

File diff suppressed because it is too large Load Diff

1236
bin/nfdump.c Normal file
View File

File diff suppressed because it is too large Load Diff

85
bin/nfdump.h Normal file
View File

@ -0,0 +1,85 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfdump.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFDUMP_H
#define _NFDUMP_H 1
#define BuffNumRecords 1024
/*
* Offset definitions for filter engine. Offsets must agree with the defined
* flow record definition data_block_record_t in nffile.h
*/
#include "config.h"
typedef struct FilterParam {
uint16_t comp;
uint16_t direction;
uint32_t data;
uint32_t inout;
uint32_t acl;
uint32_t self;
} FilterParam_t;
/* IP tree type */
typedef RB_HEAD(IPtree, IPListNode) IPlist_t;
/* Port/AS tree type */
typedef RB_HEAD(ULongtree, ULongListNode) ULongtree_t;
/* parser/scanner prototypes */
int yyparse(void);
int yylex(void);
void lex_cleanup(void);
void lex_init(char *buf);
int ScreenIPString(char *string);
int ScreenIdentString(char *string);
// Insert the RB prototypes here
RB_PROTOTYPE(IPtree, IPListNode, entry, IPNodeCMP);
RB_PROTOTYPE(ULongtree, ULongListNode, entry, ULNodeCMP);
#endif //_NFDUMP_H

188
bin/nfdump.test.diff Normal file
View File

@ -0,0 +1,188 @@
3c3
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
5c5
< size = 196
---
> size = 172
57c57
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
59c59
< size = 196
---
> size = 172
111c111
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
113c113
< size = 196
---
> size = 172
165c165
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
167c167
< size = 196
---
> size = 172
219c219
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
221c221
< size = 196
---
> size = 172
273c273
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
275c275
< size = 196
---
> size = 172
327c327
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
329c329
< size = 196
---
> size = 172
381c381
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
383c383
< size = 196
---
> size = 172
435c435
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
437c437
< size = 196
---
> size = 172
489c489
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
491c491
< size = 196
---
> size = 172
543c543
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
545c545
< size = 196
---
> size = 172
597c597
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
599c599
< size = 196
---
> size = 172
651c651
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
653c653
< size = 196
---
> size = 172
705c705
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
707c707
< size = 196
---
> size = 172
758c758
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x00 FLOW, Unsampled
760c760
< size = 196
---
> size = 172
812c812
< Flags = 0x07 FLOW, Unsampled
---
> Flags = 0x01 FLOW, Unsampled
814c814
< size = 220
---
> size = 196
866c866
< Flags = 0x07 FLOW, Unsampled
---
> Flags = 0x01 FLOW, Unsampled
868c868
< size = 220
---
> size = 196
920c920
< Flags = 0x07 FLOW, Unsampled
---
> Flags = 0x05 FLOW, Unsampled
922c922
< size = 220
---
> size = 200
974c974
< Flags = 0x07 FLOW, Unsampled
---
> Flags = 0x03 FLOW, Unsampled
976c976
< size = 220
---
> size = 200
1030c1030
< size = 220
---
> size = 204
1082c1082
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x04 FLOW, Unsampled
1084c1084
< size = 196
---
> size = 176
1136c1136
< Flags = 0x06 FLOW, Unsampled
---
> Flags = 0x02 FLOW, Unsampled
1138c1138
< size = 196
---
> size = 176
1192c1192
< size = 196
---
> size = 180
1246c1246
< size = 200
---
> size = 184
1300c1300
< size = 200
---
> size = 184
1354c1354
< size = 204
---
> size = 188

127
bin/nfdump_inline.c Executable file
View File

@ -0,0 +1,127 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfdump_inline.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
// to improve readability - separate some code blocks in functions and make them inline
// as it's called for every single flow
static inline void UpdateStat(stat_record_t *stat_record, master_record_t *master_record);
static inline void UpdateXStat(xstat_t *xstat, master_record_t *master_record);
static inline void UpdateStat(stat_record_t *stat_record, master_record_t *master_record) {
switch (master_record->prot) {
case IPPROTO_ICMP:
case IPPROTO_ICMPV6:
stat_record->numflows_icmp += master_record->aggr_flows ? master_record->aggr_flows : 1;
stat_record->numpackets_icmp += master_record->dPkts;
stat_record->numpackets_icmp += master_record->out_pkts;
stat_record->numbytes_icmp += master_record->dOctets;
stat_record->numbytes_icmp += master_record->out_bytes;
break;
case IPPROTO_TCP:
stat_record->numflows_tcp += master_record->aggr_flows ? master_record->aggr_flows : 1;
stat_record->numpackets_tcp += master_record->dPkts;
stat_record->numpackets_tcp += master_record->out_pkts;
stat_record->numbytes_tcp += master_record->dOctets;
stat_record->numbytes_tcp += master_record->out_bytes;
break;
case IPPROTO_UDP:
stat_record->numflows_udp += master_record->aggr_flows ? master_record->aggr_flows : 1;
stat_record->numpackets_udp += master_record->dPkts;
stat_record->numpackets_udp += master_record->out_pkts;
stat_record->numbytes_udp += master_record->dOctets;
stat_record->numbytes_udp += master_record->out_bytes;
break;
default:
stat_record->numflows_other += master_record->aggr_flows ? master_record->aggr_flows : 1;
stat_record->numpackets_other += master_record->dPkts;
stat_record->numpackets_other += master_record->out_pkts;
stat_record->numbytes_other += master_record->dOctets;
stat_record->numbytes_other += master_record->out_bytes;
}
stat_record->numflows += master_record->aggr_flows ? master_record->aggr_flows : 1;
stat_record->numpackets += master_record->dPkts;
stat_record->numpackets += master_record->out_pkts;
stat_record->numbytes += master_record->dOctets;
stat_record->numbytes += master_record->out_bytes;
if ( master_record->first < stat_record->first_seen ) {
stat_record->first_seen = master_record->first;
stat_record->msec_first = master_record->msec_first;
}
if ( master_record->first == stat_record->first_seen &&
master_record->msec_first < stat_record->msec_first )
stat_record->msec_first = master_record->msec_first;
if ( master_record->last > stat_record->last_seen ) {
stat_record->last_seen = master_record->last;
stat_record->msec_last = master_record->msec_last;
}
if ( master_record->last == stat_record->last_seen &&
master_record->msec_last > stat_record->msec_last )
stat_record->msec_last = master_record->msec_last;
} // End of UpdateStat
static inline void UpdateXStat(xstat_t *xstat, master_record_t *master_record) {
uint32_t bpp = master_record->dPkts ? master_record->dOctets/master_record->dPkts : 0;
if ( bpp > MAX_BPP )
bpp = MAX_BPP;
if ( master_record->prot == IPPROTO_TCP ) {
xstat->bpp_histogram->tcp.bpp[bpp]++;
xstat->bpp_histogram->tcp.count++;
xstat->port_histogram->src_tcp.port[master_record->srcport]++;
xstat->port_histogram->dst_tcp.port[master_record->dstport]++;
xstat->port_histogram->src_tcp.count++;
xstat->port_histogram->dst_tcp.count++;
} else if ( master_record->prot == IPPROTO_UDP ) {
xstat->bpp_histogram->udp.bpp[bpp]++;
xstat->bpp_histogram->udp.count++;
xstat->port_histogram->src_udp.port[master_record->srcport]++;
xstat->port_histogram->dst_udp.port[master_record->dstport]++;
xstat->port_histogram->src_udp.count++;
xstat->port_histogram->dst_udp.count++;
}
} // End of UpdateXStat

544
bin/nfexpire.c Normal file
View File

@ -0,0 +1,544 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfexpire.c 51 2010-01-29 09:01:54Z haag $
*
* $LastChangedRevision: 51 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/uio.h>
#include <unistd.h>
#include <dirent.h>
#include <sys/param.h>
#include <fcntl.h>
#include <errno.h>
#include <ctype.h>
#include <time.h>
#ifdef HAVE_FTS_H
# include <fts.h>
#else
# include "fts_compat.h"
#define fts_children fts_children_compat
#define fts_close fts_close_compat
#define fts_open fts_open_compat
#define fts_read fts_read_compat
#define fts_set fts_set_compat
#endif
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "bookkeeper.h"
#include "nfstatfile.h"
#include "expire.h"
#include "util.h"
static void usage(char *name);
void CheckDataDir( char *datadir);
channel_t *GetChannelList(char *datadir, int is_profile, int do_rescan);
static void usage(char *name) {
printf("usage %s [options] \n"
"-h\t\tThis text\n"
"-l datadir\tList stat from directory\n"
"-e datadir\tExpire data in directory\n"
"-r datadir\tRescan data directory\n"
"-u datadir\tUpdate expire params from collector logging at <datadir>\n"
"-s size\t\tmax size: scales b bytes, k kilo, m mega, g giga t tera\n"
"-t lifetime\tmaximum life time of data: scales: w week, d day, H hour, M minute\n"
"-w watermark\tlow water mark in %% for expire.\n"
, name);
} // End of usage
void CheckDataDir( char *datadir) {
if ( datadir ) {
fprintf(stderr, "Specify only one option out of -l -e -r -u or -p \n");
exit(250);
}
} // End of CheckDataDir
static char *AbsolutePath(char *dirname) {
char *path;
if ( !dirname )
return NULL;
if ( dirname[0] == '/' ) // it's already absolute path
return dirname;
path = (char *)malloc(MAXPATHLEN);
if ( !path ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
memset((void *)path, 0, MAXPATHLEN);
getcwd(path, MAXPATHLEN-strlen(dirname)-2); // -2: one for '/' and one for '\0'
strncat(path, "/", 1);
strncat(path, dirname, strlen(dirname));
path[MAXPATHLEN-1] = '\0';
return path;
} // End of AbsolutePath
channel_t *GetChannelList(char *datadir, int is_profile, int do_rescan) {
channel_t **c, *channel;
stringlist_t dirlist;
struct stat stat_buf;
int i;
// Generate list of directories
InitStringlist(&dirlist, 32);
if ( is_profile ) {
DIR *PDIR = opendir(datadir);
struct dirent *entry;
if ( !PDIR ) {
fprintf(stderr, "Can't read profiledir '%s': %s\n",datadir, strerror(errno) );
return NULL;
}
while ( ( entry = readdir(PDIR)) != NULL ) {
char stringbuf[MAXPATHLEN];
snprintf(stringbuf, MAXPATHLEN-1, "%s/%s", datadir, entry->d_name);
stringbuf[MAXPATHLEN-1] = '\0';
if ( stat(stringbuf, &stat_buf) ) {
fprintf(stderr, "Can't stat '%s': %s\n",stringbuf, strerror(errno) );
continue;
}
if ( !S_ISDIR(stat_buf.st_mode) )
continue;
// skip all '.' entries -> make .anything invisible to nfprofile
if ( entry->d_name[0] == '.' )
continue;
InsertString(&dirlist, stringbuf);
}
closedir(PDIR);
} else {
InsertString(&dirlist, datadir);
}
channel = NULL;
c = &channel;
for ( i=0; i<dirlist.num_strings; i++ ) {
int ret;
*c = (channel_t *)malloc(sizeof(channel_t));
if ( !*c ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
memset((void *)*c, 0, sizeof(channel_t));
(*c)->next = NULL;
(*c)->datadir = dirlist.list[i];
(*c)->do_rescan = do_rescan;
ret = ReadStatInfo((*c)->datadir, &(*c)->dirstat, CREATE_AND_LOCK);
switch (ret) {
case FORCE_REBUILD:
printf("Force rebuild requested by stat record in %s\n", (*c)->datadir);
(*c)->do_rescan = 1; // file corrupt - rescan
break;
case STATFILE_OK:
break;
case ERR_NOSTATFILE: // first rescan bevore expire, if no file exists
if ( do_rescan == 0 ) {
printf("Force rebuild to create stat record in %s\n", (*c)->datadir);
(*c)->do_rescan = 1;
} // else do_rescan is already set - do not report
break;
default:
exit(250);
}
(*c)->books_stat = AccessBookkeeper(&((*c)->books), (*c)->datadir);
if ( (*c)->books_stat == ERR_FAILED ) {
fprintf(stderr, "Failed to access bookkeeping record.\n");
exit(250);
}
c = &(*c)->next;
}
return channel;
} // End of GetChannelList
int main( int argc, char **argv ) {
struct stat fstat;
int c, err, maxsize_set, maxlife_set;
int do_rescan, do_expire, do_list, print_stat, do_update_param, print_books, is_profile, nfsen_format;
char *maxsize_string, *lifetime_string, *datadir;
uint64_t maxsize, lifetime, low_water;
uint32_t runtime;
channel_t *channel, *current_channel;
maxsize_string = lifetime_string = NULL;
datadir = NULL;
maxsize = lifetime = 0;
do_rescan = 0;
do_expire = 0;
do_list = 0;
do_update_param = 0;
is_profile = 0;
print_stat = 0;
print_books = 0;
maxsize_set = 0;
maxlife_set = 0;
low_water = 0;
nfsen_format = 0;
runtime = 0;
while ((c = getopt(argc, argv, "e:hl:L:T:Ypr:s:t:u:w:")) != EOF) {
switch (c) {
case 'h':
usage(argv[0]);
exit(0);
break;
case 'l':
CheckDataDir(datadir);
datadir = optarg;
do_list = 1;
print_stat = 1;
break;
case 'L':
CheckDataDir(datadir);
datadir = optarg;
print_stat = 1;
print_books = 1;
break;
case 'p':
is_profile = 1;
break;
case 'r':
CheckDataDir(datadir);
do_rescan = 1;
print_stat = 1;
datadir = optarg;
break;
case 'e':
CheckDataDir(datadir);
datadir = optarg;
do_expire = 1;
print_stat = 1;
break;
case 's':
if ( ParseSizeDef(optarg, &maxsize ) == 0 )
exit(250);
maxsize_set = 1;
break;
case 't':
if ( ParseTimeDef(optarg, &lifetime ) == 0 )
exit(250);
maxlife_set = 1;
break;
case 'u':
CheckDataDir(datadir);
datadir = optarg;
do_update_param = 1;
break;
case 'w':
low_water = strtoll(optarg, NULL, 10);
if ( low_water > 100 ) {
fprintf(stderr, "Water mark > 100%%\n");
exit(250);
}
if ( low_water == 0 )
low_water = 100;
break;
case 'T':
runtime = strtoll(optarg, NULL, 10);
if ( runtime > 3600 ) {
fprintf(stderr, "Runtime > 3600 (1h)\n");
exit(250);
}
break;
case 'Y':
nfsen_format = 1;
break;
default:
usage(argv[0]);
exit(250);
}
}
datadir = AbsolutePath(datadir);
if ( !datadir ) {
fprintf(stderr, "Missing data directory\n");
usage(argv[0]);
exit(250);
}
err = stat(datadir, &fstat);
if ( !(fstat.st_mode & S_IFDIR) ) {
fprintf(stderr, "No such directory: %s\n", datadir);
exit(250);
}
channel = GetChannelList(datadir, is_profile, do_rescan);
// GetChannelList(datadir, is_profile, do_rescan);
if ( !channel ) {
exit(250);
}
// printf("Size: %llu, time: %llu\n", maxsize, lifetime);
// update water mark only, when not listing
if ( !is_profile && !do_list && low_water )
channel->dirstat->low_water = low_water;
/* process do_list first: if the UpdateBookStat results in a FORCE_REBUILD,
* this will immediately done afterwards
* do_expire will need accurate books as well, so update the books here as well
*/
if ( do_list || do_expire ) {
current_channel = channel;
while ( current_channel ) {
if ( current_channel->books_stat == BOOKKEEPER_OK ) {
bookkeeper_t tmp_books;
printf("Include nfcapd bookeeping record in %s\n", current_channel->datadir);
ClearBooks(current_channel->books, &tmp_books);
UpdateBookStat(current_channel->dirstat, &tmp_books);
if ( current_channel->dirstat->status == FORCE_REBUILD )
current_channel->do_rescan = 1;
}
current_channel = current_channel->next;
}
}
// process do_rescan: make sure stats are up to date, if required
current_channel = channel;
while ( current_channel ) {
if ( current_channel->do_rescan ) {
int i;
uint64_t last_sequence;
/* detect new files: If nfcapd adds a new file while we are rescanning the directory
* this results in inconsistent data for the rescan. Therefore check at the begin and end
* of the rescan for the sequence number, which reflects the accesss/change to the bookkeeping record
* It's assumed, that such an event does not occure more than once. However, loop 3 times max
*/
for ( i=0; i<3; i++ ) {
last_sequence = BookSequence(current_channel->books);
printf("Scanning files in %s .. ", current_channel->datadir);
RescanDir(current_channel->datadir, current_channel->dirstat);
if ( current_channel->dirstat->numfiles == 0 ) { //nothing found
current_channel->status = NOFILES;
}
if ( BookSequence(current_channel->books) == last_sequence )
break;
printf("Rescan again, due to file changes in directory!\n");
}
if ( BookSequence(current_channel->books) != last_sequence ) {
fprintf(stderr, "Could not savely rescan the directory. Data is not consistent.\n");
ReleaseBookkeeper(current_channel->books, DETACH_ONLY);
if ( current_channel->status == OK )
WriteStatInfo(current_channel->dirstat);
exit(250);
}
printf("done.\n");
if ( current_channel->books_stat == BOOKKEEPER_OK ) {
printf("Updating nfcapd bookeeping records\n");
ClearBooks(channel->books, NULL);
}
}
current_channel = current_channel->next;
}
// now process do_expire if required
if ( do_expire ) {
dirstat_t old_stat, current_stat;
if ( is_profile ) {
current_stat.status = 0;
current_stat.max_lifetime = lifetime;
current_stat.max_size = maxsize;
current_stat.low_water = low_water ? low_water : 98;
// sum up all channels in the profile
current_channel = channel;
current_stat.numfiles = current_channel->dirstat->numfiles;
current_stat.filesize = current_channel->dirstat->filesize;
current_stat.first = current_channel->dirstat->first;
current_stat.last = current_channel->dirstat->last;
current_channel = current_channel->next;
while ( current_channel ) {
current_stat.numfiles += current_channel->dirstat->numfiles;
current_stat.filesize += current_channel->dirstat->filesize;
if ( current_channel->dirstat->first && (current_channel->dirstat->first < current_stat.first) )
current_stat.first = current_channel->dirstat->first;
if ( current_channel->dirstat->last > current_stat.last )
current_stat.last = current_channel->dirstat->last;
current_channel = current_channel->next;
}
old_stat = current_stat;
ExpireProfile(channel, &current_stat, maxsize, lifetime, runtime);
} else {
// cmd args override dirstat values
if ( maxsize_set )
channel->dirstat->max_size = maxsize;
else
maxsize = channel->dirstat->max_size;
if ( maxlife_set )
channel->dirstat->max_lifetime = lifetime;
else
lifetime = channel->dirstat->max_lifetime;
old_stat = *(channel->dirstat);
ExpireDir(channel->datadir, channel->dirstat, maxsize, lifetime, runtime);
current_stat = *(channel->dirstat);
}
// Report, what we have done
printf("Expired files: %llu\n", (unsigned long long)(old_stat.numfiles - current_stat.numfiles));
printf("Expired file size: %s\n", ScaleValue(old_stat.filesize - current_stat.filesize));
printf("Expired time range: %s\n\n", ScaleTime(current_stat.first - old_stat.first));
}
if ( !is_profile && do_update_param ) {
switch (channel->books_stat) {
case BOOKKEEPER_OK:
if ( maxsize_set )
channel->dirstat->max_size = maxsize;
else
maxsize = channel->dirstat->max_size;
if ( maxlife_set )
channel->dirstat->max_lifetime = lifetime;
else
lifetime = channel->dirstat->max_lifetime;
printf("Update collector process running for directory: '%s'\n", datadir);
UpdateBooksParam(channel->books, (time_t)lifetime, maxsize);
print_stat = 1;
break;
case ERR_NOTEXISTS:
if ( maxsize_set )
channel->dirstat->max_size = maxsize;
if ( maxlife_set )
channel->dirstat->max_lifetime = lifetime;
print_stat = 1;
break;
default:
// should never be reached as already cought earlier
printf("Error %i while connecting to collector\n", channel->books_stat);
}
if ( channel->status == OK || channel->status == NOFILES )
WriteStatInfo(channel->dirstat);
}
if ( !is_profile && print_books ) {
switch (channel->books_stat) {
case BOOKKEEPER_OK:
PrintBooks(channel->books);
break;
case ERR_NOTEXISTS:
printf("No collector process running for directory: '%s'\n", channel->datadir);
break;
default:
// should never be reached as already cought earlier
printf("Error %i while connecting to collector\n", channel->books_stat);
}
}
if ( print_stat ) {
if ( is_profile ) {
dirstat_t current_stat;
current_stat.status = 0;
current_stat.max_lifetime = lifetime;
current_stat.max_size = maxsize;
current_stat.low_water = low_water ? low_water : 98;
// sum up all channels in the profile
current_channel = channel;
current_stat.numfiles = current_channel->dirstat->numfiles;
current_stat.filesize = current_channel->dirstat->filesize;
current_stat.first = current_channel->dirstat->first;
current_stat.last = current_channel->dirstat->last;
current_channel = current_channel->next;
while ( current_channel ) {
current_stat.numfiles += current_channel->dirstat->numfiles;
current_stat.filesize += current_channel->dirstat->filesize;
if ( current_channel->dirstat->first && (current_channel->dirstat->first < current_stat.first) )
current_stat.first = current_channel->dirstat->first;
if ( current_channel->dirstat->last > current_stat.last )
current_stat.last = current_channel->dirstat->last;
current_channel = current_channel->next;
}
if ( nfsen_format ) {
printf("Stat|%llu|%llu|%llu\n",
(unsigned long long)current_stat.filesize,
(unsigned long long)current_stat.first, (unsigned long long)current_stat.last);
} else
PrintDirStat(&current_stat);
} else
if ( nfsen_format )
printf("Stat|%llu|%llu|%llu\n",
(unsigned long long)channel->dirstat->filesize,
(unsigned long long)channel->dirstat->first, (unsigned long long)channel->dirstat->last );
else
PrintDirStat(channel->dirstat);
}
current_channel = channel;
while ( current_channel ) {
ReleaseBookkeeper(current_channel->books, DETACH_ONLY);
if ( current_channel->status == OK )
WriteStatInfo(current_channel->dirstat);
current_channel = current_channel->next;
}
return 0;
}

412
bin/nfexport.c Executable file
View File

@ -0,0 +1,412 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfexport.c 54 2010-01-29 11:30:22Z haag $
*
* $LastChangedRevision: 54 $
*
*/
#include "config.h"
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/param.h>
#include <netinet/in.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "nf_common.h"
#include "nffile.h"
#include "nfx.h"
#include "nfstat.h"
#include "nfxstat.h"
#include "nflowcache.h"
#include "exporter.h"
#include "nfexport.h"
#include "nfdump_inline.c"
#define NEED_PACKRECORD 1
#include "nffile_inline.c"
#undef NEED_PACKRECORD
#include "heapsort_inline.c"
#include "applybits_inline.c"
/* global vars */
extern extension_descriptor_t extension_descriptor[];
/* local vars */
enum CntIndices { FLOWS = 0, INPACKETS, INBYTES, OUTPACKETS, OUTBYTES };
static void ExportExtensionMaps( int aggregate, int bidir, nffile_t *nffile, extension_map_list_t *extension_map_list );
static void ExportExtensionMaps( int aggregate, int bidir, nffile_t *nffile, extension_map_list_t *extension_map_list ) {
int map_id, opt_extensions, num_extensions, new_map_size, opt_align;
extension_map_t *new_map;
// no extension maps to export - nothing to do
if ( extension_map_list->max_used == 0 )
return;
new_map = NULL;
for ( map_id = 0; map_id <= extension_map_list->max_used; map_id++ ) {
extension_map_t *SourceMap = extension_map_list->slot[map_id]->map;
int i, has_aggr_flows, has_out_bytes, has_out_packets, has_nat;
// skip maps, never referenced
#ifdef DEVEL
printf("Process map id: %i\n", map_id);
printf("Ref count: %i\n", extension_map_list->slot[map_id]->ref_count);
#endif
if ( extension_map_list->slot[map_id]->ref_count == 0 ) {
#ifdef DEVEL
printf("Ref count = 0 => Skip map\n");
#endif
continue;
}
// parse Source map if it contains all required fields:
// for aggregation EX_AGGR_FLOWS_4 or _8 is required
// for bidir flows EX_OUT_PKG_4 or _8 and EX_OUT_BYTES_4 or_8 are required
has_aggr_flows = 0;
has_out_bytes = 0;
has_out_packets = 0;
// parse map for older NEL nat extension
has_nat = 0;
num_extensions = 0;
i = 0;
while ( SourceMap->ex_id[i] ) {
switch (SourceMap->ex_id[i]) {
case EX_AGGR_FLOWS_4:
case EX_AGGR_FLOWS_8:
has_aggr_flows = 1;
break;
case EX_OUT_BYTES_4:
case EX_OUT_BYTES_8:
has_out_bytes = 1;
break;
case EX_OUT_PKG_4:
case EX_OUT_PKG_8:
has_out_packets = 1;
break;
case EX_NEL_GLOBAL_IP_v4:
// Map old nat extension to common NSEL extension
SourceMap->ex_id[i] = EX_NSEL_XLATE_IP_v4;
has_nat = 1;
// default: nothing to do
}
i++;
num_extensions++;
}
#ifdef DEVEL
printf("map: num_extensions: %i, has_aggr_flows: %i, has_out_bytes: %i, has_out_packets: %i, has_nat: %i\n",
num_extensions, has_aggr_flows, has_out_bytes, has_out_packets, has_nat);
#endif
// count missing extensions
opt_extensions = 0;
if ( aggregate && !has_aggr_flows )
opt_extensions++;
if ( bidir && !has_out_bytes )
opt_extensions++;
if ( bidir && !has_out_packets )
opt_extensions++;
opt_extensions += has_nat;
// calculate new map size
new_map_size = sizeof(extension_map_t) + ( num_extensions + opt_extensions) * sizeof(uint16_t);
#ifdef DEVEL
printf("opt_extensions: %i, new_map_size: %i\n", opt_extensions,new_map_size );
PrintExtensionMap(SourceMap);
#endif
if ( opt_extensions ) {
// align 32bits
if (( new_map_size & 0x3 ) != 0 ) {
new_map_size += 4 - ( new_map_size & 0x3 );
opt_align = 1;
} else {
opt_align = 0;
}
} else {
// no missing elements in extension map - we can used the original one
// and we are done
#ifdef DEVEL
printf("New map identical => use this map:\n");
PrintExtensionMap(SourceMap);
#endif
// Flush the map to disk
AppendToBuffer(nffile, (void *)SourceMap, SourceMap->size);
continue;
}
#ifdef DEVEL
printf("Create new map:\n");
#endif
// new map is different - create the new map
new_map = (extension_map_t *)malloc((ssize_t)new_map_size);
if ( !new_map ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
// Panic check - should never happen, but we are going to copy memory
if ( new_map_size < SourceMap->size ) {
LogError("PANIC! new_map_size(%i) < SourceMap->size(%i) in %s line %d\n",
new_map_size, SourceMap->size, __FILE__, __LINE__);
exit(255);
}
// copy existing map
memcpy((void *)new_map, (void *)SourceMap, SourceMap->size);
new_map->size = new_map_size;
// add the missing extensions to the output map
// skip to end of current map
while ( new_map->ex_id[i] )
i++;
if ( has_nat ) {
new_map->ex_id[i++] = EX_NSEL_XLATE_PORTS;
new_map->extension_size += extension_descriptor[EX_NSEL_XLATE_PORTS].size;
}
// add missing map elements
if ( aggregate && !has_aggr_flows ) {
new_map->ex_id[i++] = EX_AGGR_FLOWS_4;
new_map->extension_size += extension_descriptor[EX_AGGR_FLOWS_4].size;
}
if ( bidir && !has_out_bytes ) {
new_map->ex_id[i++] = EX_OUT_BYTES_8;
new_map->extension_size += extension_descriptor[EX_OUT_BYTES_8].size;
}
if ( bidir && !has_out_packets ) {
new_map->ex_id[i++] = EX_OUT_PKG_8;
new_map->extension_size += extension_descriptor[EX_OUT_PKG_8].size;
}
// end of map tag
new_map->ex_id[i++] = 0;
if ( opt_align )
new_map->ex_id[i] = 0;
#ifdef DEVEL
PrintExtensionMap(new_map);
#endif
free(extension_map_list->slot[map_id]->map);
extension_map_list->slot[map_id]->map = new_map;
// Flush the map to disk
AppendToBuffer(nffile, (void *)new_map, new_map->size);
}
} // End of ExportExtensionMaps
int ExportFlowTable(nffile_t *nffile, int aggregate, int bidir, int date_sorted, extension_map_list_t *extension_map_list) {
hash_FlowTable *FlowTable;
FlowTableRecord_t *r;
SortElement_t *SortList;
master_record_t *aggr_record_mask;
uint32_t i;
uint32_t maxindex, c;
#ifdef DEVEL
char *string;
#endif
ExportExtensionMaps(aggregate, bidir, nffile, extension_map_list);
ExportExporterList(nffile);
aggr_record_mask = GetMasterAggregateMask();
FlowTable = GetFlowTable();
c = 0;
maxindex = FlowTable->NumRecords;
if ( date_sorted ) {
// Sort records according the date
SortList = (SortElement_t *)calloc(maxindex, sizeof(SortElement_t));
if ( !SortList ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
// preset SortList table - still unsorted
for ( i=0; i<=FlowTable->IndexMask; i++ ) {
r = FlowTable->bucket[i];
if ( !r )
continue;
// foreach elem in this bucket
while ( r ) {
SortList[c].count = 1000LL * r->flowrecord.first + r->flowrecord.msec_first; // sort according the date
SortList[c].record = (void *)r;
c++;
r = r->next;
}
}
if ( c != maxindex ) {
LogError("Abort: Missmatch %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
if ( c >= 2 )
heapSort(SortList, c, 0);
for ( i = 0; i < c; i++ ) {
master_record_t *flow_record;
common_record_t *raw_record;
extension_info_t *extension_info;
r = (FlowTableRecord_t *)(SortList[i].record);
raw_record = &(r->flowrecord);
extension_info = r->map_info_ref;
flow_record = &(extension_info->master_record);
ExpandRecord_v2( raw_record, extension_info, r->exp_ref, flow_record);
flow_record->dPkts = r->counter[INPACKETS];
flow_record->dOctets = r->counter[INBYTES];
flow_record->out_pkts = r->counter[OUTPACKETS];
flow_record->out_bytes = r->counter[OUTBYTES];
flow_record->aggr_flows = r->counter[FLOWS];
// apply IP mask from aggregation, to provide a pretty output
if ( FlowTable->has_masks ) {
flow_record->v6.srcaddr[0] &= FlowTable->IPmask[0];
flow_record->v6.srcaddr[1] &= FlowTable->IPmask[1];
flow_record->v6.dstaddr[0] &= FlowTable->IPmask[2];
flow_record->v6.dstaddr[1] &= FlowTable->IPmask[3];
}
if ( FlowTable->apply_netbits )
ApplyNetMaskBits(flow_record, FlowTable->apply_netbits);
if ( aggr_record_mask ) {
ApplyAggrMask(flow_record, aggr_record_mask);
}
// switch to output extension map
flow_record->map_ref = extension_info->map;
flow_record->ext_map = extension_info->map->map_id;
PackRecord(flow_record, nffile);
#ifdef DEVEL
format_file_block_record((void *)flow_record, &string, 0);
printf("%s\n", string);
#endif
// Update statistics
UpdateStat(nffile->stat_record, flow_record);
}
} else {
// print them as they came
for ( i=0; i<=FlowTable->IndexMask; i++ ) {
r = FlowTable->bucket[i];
while ( r ) {
master_record_t *flow_record;
common_record_t *raw_record;
extension_info_t *extension_info;
raw_record = &(r->flowrecord);
extension_info = r->map_info_ref;
flow_record = &(extension_info->master_record);
ExpandRecord_v2( raw_record, extension_info, r->exp_ref, flow_record);
flow_record->dPkts = r->counter[INPACKETS];
flow_record->dOctets = r->counter[INBYTES];
flow_record->out_pkts = r->counter[OUTPACKETS];
flow_record->out_bytes = r->counter[OUTBYTES];
flow_record->aggr_flows = r->counter[FLOWS];
// apply IP mask from aggregation, to provide a pretty output
if ( FlowTable->has_masks ) {
flow_record->v6.srcaddr[0] &= FlowTable->IPmask[0];
flow_record->v6.srcaddr[1] &= FlowTable->IPmask[1];
flow_record->v6.dstaddr[0] &= FlowTable->IPmask[2];
flow_record->v6.dstaddr[1] &= FlowTable->IPmask[3];
}
if ( FlowTable->apply_netbits )
ApplyNetMaskBits(flow_record, FlowTable->apply_netbits);
if ( aggr_record_mask ) {
ApplyAggrMask(flow_record, aggr_record_mask);
}
// switch to output extension map
flow_record->map_ref = extension_info->map;
flow_record->ext_map = extension_info->map->map_id;
PackRecord(flow_record, nffile);
#ifdef DEVEL
format_file_block_record((void *)flow_record, &string, 0);
printf("%s\n", string);
#endif
// Update statistics
UpdateStat(nffile->stat_record, flow_record);
r = r->next;
}
}
}
if ( nffile->block_header->NumRecords ) {
if ( WriteBlock(nffile) <= 0 ) {
LogError("Failed to write output buffer to disk: '%s'" , strerror(errno));
return 0;
}
}
return 1;
} // End of ExportFlowTable

45
bin/nfexport.h Executable file
View File

@ -0,0 +1,45 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfexport.h 54 2010-01-29 11:30:22Z haag $
*
* $LastChangedRevision: 54 $
*
*/
#ifndef _NFEXPORT_H
#define _NFEXPORT_H 1
int ExportFlowTable(nffile_t *nffile, int aggregate, int bidir, int date_sorted, extension_map_list_t *extension_map_list);
#endif //_NFEXPORT_H

1356
bin/nffile.c Normal file
View File

File diff suppressed because it is too large Load Diff

2203
bin/nffile.h Normal file
View File

File diff suppressed because it is too large Load Diff

835
bin/nffile_inline.c Executable file
View File

@ -0,0 +1,835 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nffile_inline.c 40 2009-12-16 10:41:44Z haag $
*
* $LastChangedRevision: 40 $
*
*/
/*
* nffile_inline.c is needed for daemon code as well as normal stdio code
* therefore a generic LogError is defined, which maps to the
* approriate logging channel - either stderr or syslog
*/
void LogError(char *format, ...);
static inline int CheckBufferSpace(nffile_t *nffile, size_t required);
static inline void AppendToBuffer(nffile_t *nffile, void *record, size_t required);
static inline void CopyV6IP(uint32_t *dst, uint32_t *src);
static inline void ExpandRecord_v2(common_record_t *input_record, extension_info_t *extension_info, exporter_info_record_t *exporter_info, master_record_t *output_record );
#ifdef NEED_PACKRECORD
static void PackRecord(master_record_t *master_record, nffile_t *nffile);
#endif
static inline int CheckBufferSpace(nffile_t *nffile, size_t required) {
#ifdef DEVEL
// printf("Buffer Size %u\n", nffile->block_header->size);
#endif
// flush current buffer to disc
if ( (nffile->block_header->size + required ) > WRITE_BUFFSIZE ) {
// this should never happen, but catch it anyway
if ( required > WRITE_BUFFSIZE ) {
LogError("Required buffer size %zu too big for output buffer!" , required);
return 0;
}
if ( WriteBlock(nffile) <= 0 ) {
LogError("Failed to write output buffer to disk: '%s'" , strerror(errno));
return 0;
}
}
return 1;
} // End of CheckBufferSpace
// Use 4 uint32_t copy cycles, as SPARC CPUs brak
static inline void CopyV6IP(uint32_t *dst, uint32_t *src) {
dst[0] = src[0];
dst[1] = src[1];
dst[2] = src[2];
dst[3] = src[3];
} // End of CopyV6IP
/*
* Expand file record into master record for further processing
* LP64 CPUs need special 32bit operations as it is not guarateed, that 64bit
* values are aligned
*/
static inline void ExpandRecord_v2(common_record_t *input_record, extension_info_t *extension_info, exporter_info_record_t *exporter_info, master_record_t *output_record ) {
extension_map_t *extension_map = extension_info->map;
uint32_t i, *u;
void *p = (void *)input_record;
#ifdef NSEL
// nasty bug work around - compat issues 1.6.10 - 1.6.12 onwards
union {
uint16_t port[2];
uint32_t vrf;
} compat_nel_bug;
compat_nel_bug.vrf = 0;
int compat_nel = 0;
#endif
// set map ref
output_record->map_ref = extension_map;
if ( input_record->type == CommonRecordType ) {
// Copy common data block
memcpy((void *)output_record, (void *)input_record, COMMON_RECORD_DATA_SIZE);
p = (void *)input_record->data;
} else {
// Compat v0 record - convert to new Common Record
common_record_v0_t *common_record_v0 = (common_record_v0_t *)input_record;
uint16_t flags = common_record_v0->flags;
uint16_t exporter_sysid = common_record_v0->exporter_sysid;
memcpy((void *)output_record, (void *)input_record, COMMON_RECORDV0_DATA_SIZE);
output_record->flags = flags;
output_record->exporter_sysid = exporter_sysid;
p = (void *)common_record_v0->data;
}
if ( exporter_info ) {
uint32_t sysid = exporter_info->sysid;
output_record->exporter_sysid = sysid;
input_record->exporter_sysid = sysid;
output_record->exp_ref = exporter_info;
} else {
output_record->exp_ref = NULL;
}
// map icmp type/code in it's own vars
output_record->icmp = output_record->dstport;
// Required extension 1 - IP addresses
if ( (input_record->flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6
// IPv6
// keep compiler happy
// memcpy((void *)output_record->v6.srcaddr, p, 4 * sizeof(uint64_t));
memcpy((void *)output_record->ip_union._ip_64.addr, p, 4 * sizeof(uint64_t));
p = (void *)((pointer_addr_t)p + 4 * sizeof(uint64_t));
} else {
// IPv4
u = (uint32_t *)p;
output_record->v6.srcaddr[0] = 0;
output_record->v6.srcaddr[1] = 0;
output_record->v4.srcaddr = u[0];
output_record->v6.dstaddr[0] = 0;
output_record->v6.dstaddr[1] = 0;
output_record->v4.dstaddr = u[1];
p = (void *)((pointer_addr_t)p + 2 * sizeof(uint32_t));
}
// Required extension 2 - packet counter
if ( (input_record->flags & FLAG_PKG_64 ) != 0 ) {
// 64bit packet counter
value64_t l, *v = (value64_t *)p;
l.val.val32[0] = v->val.val32[0];
l.val.val32[1] = v->val.val32[1];
output_record->dPkts = l.val.val64;
p = (void *)((pointer_addr_t)p + sizeof(uint64_t));
} else {
// 32bit packet counter
output_record->dPkts = *((uint32_t *)p);
p = (void *)((pointer_addr_t)p + sizeof(uint32_t));
}
// Required extension 3 - byte counter
if ( (input_record->flags & FLAG_BYTES_64 ) != 0 ) {
// 64bit byte counter
value64_t l, *v = (value64_t *)p;
l.val.val32[0] = v->val.val32[0];
l.val.val32[1] = v->val.val32[1];
output_record->dOctets = l.val.val64;
p = (void *)((pointer_addr_t)p + sizeof(uint64_t));
} else {
// 32bit bytes counter
output_record->dOctets = *((uint32_t *)p);
p = (void *)((pointer_addr_t)p + sizeof(uint32_t));
}
// preset one single flow
output_record->aggr_flows = 1;
// Process optional extensions
i=0;
while ( extension_map->ex_id[i] ) {
switch (extension_map->ex_id[i++]) {
// 0 - 3 should never be in an extension table so - ignore it
case 0:
case 1:
case 2:
case 3:
break;
case EX_IO_SNMP_2: {
tpl_ext_4_t *tpl = (tpl_ext_4_t *)p;
output_record->input = tpl->input;
output_record->output = tpl->output;
p = (void *)tpl->data;
} break;
case EX_IO_SNMP_4: {
tpl_ext_5_t *tpl = (tpl_ext_5_t *)p;
output_record->input = tpl->input;
output_record->output = tpl->output;
p = (void *)tpl->data;
} break;
case EX_AS_2: {
tpl_ext_6_t *tpl = (tpl_ext_6_t *)p;
output_record->srcas = tpl->src_as;
output_record->dstas = tpl->dst_as;
p = (void *)tpl->data;
} break;
case EX_AS_4: {
tpl_ext_7_t *tpl = (tpl_ext_7_t *)p;
output_record->srcas = tpl->src_as;
output_record->dstas = tpl->dst_as;
p = (void *)tpl->data;
} break;
case EX_MULIPLE: {
tpl_ext_8_t *tpl = (tpl_ext_8_t *)p;
// use a 32 bit int to copy all 4 fields
output_record->any = tpl->any;
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_v4: {
tpl_ext_9_t *tpl = (tpl_ext_9_t *)p;
output_record->ip_nexthop.v6[0] = 0;
output_record->ip_nexthop.v6[1] = 0;
output_record->ip_nexthop.v4 = tpl->nexthop;
p = (void *)tpl->data;
ClearFlag(output_record->flags, FLAG_IPV6_NH);
} break;
case EX_NEXT_HOP_v6: {
tpl_ext_10_t *tpl = (tpl_ext_10_t *)p;
CopyV6IP((uint32_t *)output_record->ip_nexthop.v6, (uint32_t *)tpl->nexthop);
p = (void *)tpl->data;
SetFlag(output_record->flags, FLAG_IPV6_NH);
} break;
case EX_NEXT_HOP_BGP_v4: {
tpl_ext_11_t *tpl = (tpl_ext_11_t *)p;
output_record->bgp_nexthop.v6[0] = 0;
output_record->bgp_nexthop.v6[1] = 0;
output_record->bgp_nexthop.v4 = tpl->bgp_nexthop;
ClearFlag(output_record->flags, FLAG_IPV6_NHB);
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_BGP_v6: {
tpl_ext_12_t *tpl = (tpl_ext_12_t *)p;
CopyV6IP((uint32_t *)output_record->bgp_nexthop.v6, (uint32_t *)tpl->bgp_nexthop);
p = (void *)tpl->data;
SetFlag(output_record->flags, FLAG_IPV6_NHB);
} break;
case EX_VLAN: {
tpl_ext_13_t *tpl = (tpl_ext_13_t *)p;
output_record->src_vlan = tpl->src_vlan;
output_record->dst_vlan = tpl->dst_vlan;
p = (void *)tpl->data;
} break;
case EX_OUT_PKG_4: {
tpl_ext_14_t *tpl = (tpl_ext_14_t *)p;
output_record->out_pkts = tpl->out_pkts;
p = (void *)tpl->data;
} break;
case EX_OUT_PKG_8: {
tpl_ext_15_t v, *tpl = (tpl_ext_15_t *)p;
v.v[0] = tpl->v[0];
v.v[1] = tpl->v[1];
output_record->out_pkts = v.out_pkts;
p = (void *)tpl->data;
} break;
case EX_OUT_BYTES_4: {
tpl_ext_16_t *tpl = (tpl_ext_16_t *)p;
output_record->out_bytes = tpl->out_bytes;
p = (void *)tpl->data;
} break;
case EX_OUT_BYTES_8: {
tpl_ext_17_t v,*tpl = (tpl_ext_17_t *)p;
v.v[0] = tpl->v[0];
v.v[1] = tpl->v[1];
output_record->out_bytes = v.out_bytes;
p = (void *)tpl->data;
} break;
case EX_AGGR_FLOWS_4: {
tpl_ext_18_t *tpl = (tpl_ext_18_t *)p;
output_record->aggr_flows = tpl->aggr_flows;
p = (void *)tpl->data;
} break;
case EX_AGGR_FLOWS_8: {
tpl_ext_19_t v, *tpl = (tpl_ext_19_t *)p;
v.v[0] = tpl->v[0];
v.v[1] = tpl->v[1];
output_record->aggr_flows = v.aggr_flows;
p = (void *)tpl->data;
} break;
case EX_MAC_1: {
tpl_ext_20_t v, *tpl = (tpl_ext_20_t *)p;
v.v1[0] = tpl->v1[0];
v.v1[1] = tpl->v1[1];
output_record->in_src_mac = v.in_src_mac;
v.v2[0] = tpl->v2[0];
v.v2[1] = tpl->v2[1];
output_record->out_dst_mac = v.out_dst_mac;
p = (void *)tpl->data;
} break;
case EX_MAC_2: {
tpl_ext_21_t v, *tpl = (tpl_ext_21_t *)p;
v.v1[0] = tpl->v1[0];
v.v1[1] = tpl->v1[1];
output_record->in_dst_mac = v.in_dst_mac;
v.v2[0] = tpl->v2[0];
v.v2[1] = tpl->v2[1];
output_record->out_src_mac = v.out_src_mac;
p = (void *)tpl->data;
} break;
case EX_MPLS: {
tpl_ext_22_t *tpl = (tpl_ext_22_t *)p;
int j;
for (j=0; j<10; j++ ) {
output_record->mpls_label[j] = tpl->mpls_label[j];
}
p = (void *)tpl->data;
} break;
case EX_ROUTER_IP_v4: {
tpl_ext_23_t *tpl = (tpl_ext_23_t *)p;
output_record->ip_router.v6[0] = 0;
output_record->ip_router.v6[1] = 0;
output_record->ip_router.v4 = tpl->router_ip;
p = (void *)tpl->data;
ClearFlag(output_record->flags, FLAG_IPV6_EXP);
} break;
case EX_ROUTER_IP_v6: {
tpl_ext_24_t *tpl = (tpl_ext_24_t *)p;
CopyV6IP((uint32_t *)output_record->ip_router.v6, (uint32_t *)tpl->router_ip);
p = (void *)tpl->data;
SetFlag(output_record->flags, FLAG_IPV6_EXP);
} break;
case EX_ROUTER_ID: {
tpl_ext_25_t *tpl = (tpl_ext_25_t *)p;
output_record->engine_type = tpl->engine_type;
output_record->engine_id = tpl->engine_id;
p = (void *)tpl->data;
} break;
case EX_BGPADJ: {
tpl_ext_26_t *tpl = (tpl_ext_26_t *)p;
output_record->bgpNextAdjacentAS = tpl->bgpNextAdjacentAS;
output_record->bgpPrevAdjacentAS = tpl->bgpPrevAdjacentAS;
p = (void *)tpl->data;
} break;
case EX_LATENCY: {
tpl_ext_latency_t *tpl = (tpl_ext_latency_t *)p;
output_record->client_nw_delay_usec = tpl->client_nw_delay_usec;
output_record->server_nw_delay_usec = tpl->server_nw_delay_usec;
output_record->appl_latency_usec = tpl->appl_latency_usec;
p = (void *)tpl->data;
} break;
case EX_RECEIVED: {
tpl_ext_27_t *tpl = (tpl_ext_27_t *)p;
value64_t v;
v.val.val32[0] = tpl->v[0];
v.val.val32[1] = tpl->v[1];
output_record->received = v.val.val64;
p = (void *)tpl->data;
} break;
#ifdef NSEL
case EX_NSEL_COMMON: {
tpl_ext_37_t *tpl = (tpl_ext_37_t *)p;
value64_t v;
v.val.val32[0] = tpl->v[0];
v.val.val32[1] = tpl->v[1];
output_record->event_time = v.val.val64;
output_record->conn_id = tpl->conn_id;
output_record->event = tpl->fw_event;
output_record->event_flag = FW_EVENT;
output_record->fw_xevent = tpl->fw_xevent;
output_record->icmp = tpl->nsel_icmp;
p = (void *)tpl->data;
} break;
case EX_NSEL_XLATE_PORTS: {
tpl_ext_38_t *tpl = (tpl_ext_38_t *)p;
output_record->xlate_src_port = tpl->xlate_src_port;
output_record->xlate_dst_port = tpl->xlate_dst_port;
p = (void *)tpl->data;
} break;
case EX_NSEL_XLATE_IP_v4: {
tpl_ext_39_t *tpl = (tpl_ext_39_t *)p;
output_record->xlate_src_ip.v6[0] = 0;
output_record->xlate_src_ip.v6[1] = 0;
output_record->xlate_src_ip.v4 = tpl->xlate_src_ip;
output_record->xlate_dst_ip.v6[0] = 0;
output_record->xlate_dst_ip.v6[1] = 0;
output_record->xlate_dst_ip.v4 = tpl->xlate_dst_ip;
p = (void *)tpl->data;
output_record->xlate_flags = 0;
} break;
case EX_NSEL_XLATE_IP_v6: {
tpl_ext_40_t *tpl = (tpl_ext_40_t *)p;
output_record->xlate_src_ip.v6[0] = tpl->xlate_src_ip[0];
output_record->xlate_src_ip.v6[1] = tpl->xlate_src_ip[1];
output_record->xlate_dst_ip.v6[0] = tpl->xlate_dst_ip[0];
output_record->xlate_dst_ip.v6[1] = tpl->xlate_dst_ip[1];
p = (void *)tpl->data;
output_record->xlate_flags = 1;
} break;
case EX_NSEL_ACL: {
tpl_ext_41_t *tpl = (tpl_ext_41_t *)p;
int j;
for (j=0; j<3; j++) {
output_record->ingress_acl_id[j] = tpl->ingress_acl_id[j];
output_record->egress_acl_id[j] = tpl->egress_acl_id[j];
}
p = (void *)tpl->data;
} break;
case EX_NSEL_USER: {
tpl_ext_42_t *tpl = (tpl_ext_42_t *)p;
strncpy((void *)output_record->username, (void *)tpl->username, sizeof(output_record->username));
output_record->username[sizeof(output_record->username)-1] = '\0'; // safety 0
p = (void *)tpl->data;
} break;
case EX_NSEL_USER_MAX: {
tpl_ext_43_t *tpl = (tpl_ext_43_t *)p;
strncpy((void *)output_record->username, (void *)tpl->username, sizeof(output_record->username));
output_record->username[sizeof(output_record->username)-1] = '\0'; // safety 0
p = (void *)tpl->data;
} break;
case EX_NEL_COMMON: {
tpl_ext_46_t *tpl = (tpl_ext_46_t *)p;
output_record->event = tpl->nat_event;
output_record->event_flag = FW_EVENT;
// XXX - 3 bytes unused
output_record->egress_vrfid = tpl->egress_vrfid;
output_record->ingress_vrfid = tpl->ingress_vrfid;
p = (void *)tpl->data;
// remember this value, if we read old 1.6.10 files
compat_nel_bug.vrf = tpl->egress_vrfid;
if ( compat_nel ) {
output_record->xlate_src_port = compat_nel_bug.port[0];
output_record->xlate_dst_port = compat_nel_bug.port[1];
output_record->egress_vrfid = 0;
}
} break;
// compat record v1.6.10
case EX_NEL_GLOBAL_IP_v4: {
tpl_ext_47_t *tpl = (tpl_ext_47_t *)p;
output_record->xlate_src_ip.v6[0] = 0;
output_record->xlate_src_ip.v6[1] = 0;
output_record->xlate_src_ip.v4 = tpl->nat_inside;
output_record->xlate_dst_ip.v6[0] = 0;
output_record->xlate_dst_ip.v6[1] = 0;
output_record->xlate_dst_ip.v4 = tpl->nat_outside;
p = (void *)tpl->data;
output_record->xlate_src_port = compat_nel_bug.port[0];
output_record->xlate_dst_port = compat_nel_bug.port[1];
output_record->egress_vrfid = 0;
compat_nel = 1;
} break;
case EX_PORT_BLOCK_ALLOC: {
tpl_ext_48_t *tpl = (tpl_ext_48_t *)p;
output_record->block_start = tpl->block_start;
output_record->block_end = tpl->block_end;
output_record->block_step = tpl->block_step;
output_record->block_size = tpl->block_size;
if ( output_record->block_end == 0 && output_record->block_size != 0 )
output_record->block_end = output_record->block_start + output_record->block_size - 1;
p = (void *)tpl->data;
} break;
#endif
}
}
} // End of ExpandRecord_v2
#ifdef NEED_PACKRECORD
static void PackRecord(master_record_t *master_record, nffile_t *nffile) {
extension_map_t *extension_map = master_record->map_ref;
common_record_t *common_record;
uint32_t required = COMMON_RECORD_DATA_SIZE + extension_map->extension_size;
size_t size;
void *p;
int i;
// check size of packets and bytes
if ( master_record->dPkts > 0xffffffffLL ) {
master_record->flags |= FLAG_PKG_64;
required += 8;
} else {
master_record->flags &= ~FLAG_PKG_64;
required += 4;
}
if ( master_record->dOctets > 0xffffffffLL ) {
master_record->flags |= FLAG_BYTES_64;
required += 8;
} else {
master_record->flags &= ~FLAG_BYTES_64;
required += 4;
}
if ( (master_record->flags & FLAG_IPV6_ADDR) != 0 ) // IPv6
required += 32;
else
required += 8;
master_record->size = required;
// flush current buffer to disc if not enough space
if ( !CheckBufferSpace(nffile, required) ) {
return;
}
// enough buffer space available at this point
common_record = (common_record_t *)nffile->buff_ptr;
// write common record
size = COMMON_RECORD_DATA_SIZE;
memcpy((void *)common_record, (void *)master_record, size);
common_record->reserved = 0;
p = (void *)((pointer_addr_t)common_record + size);
// Required extension 1 - IP addresses
if ( (master_record->flags & FLAG_IPV6_ADDR) != 0 ) { // IPv6
// IPv6
// keep compiler happy
// memcpy(p, (void *)master_record->v6.srcaddr, 4 * sizeof(uint64_t));
memcpy(p, (void *)master_record->ip_union._ip_64.addr, 4 * sizeof(uint64_t));
p = (void *)((pointer_addr_t)p + 4 * sizeof(uint64_t));
} else {
// IPv4
uint32_t *u = (uint32_t *)p;
u[0] = master_record->v4.srcaddr;
u[1] = master_record->v4.dstaddr;
p = (void *)((pointer_addr_t)p + 2 * sizeof(uint32_t));
}
// Required extension 2 - packet counter
if ( (master_record->flags & FLAG_PKG_64 ) != 0 ) {
// 64bit packet counter
value64_t l, *v = (value64_t *)p;
l.val.val64 = master_record->dPkts;
v->val.val32[0] = l.val.val32[0];
v->val.val32[1] = l.val.val32[1];
p = (void *)((pointer_addr_t)p + sizeof(uint64_t));
} else {
// 32bit packet counter
*((uint32_t *)p) = master_record->dPkts;
p = (void *)((pointer_addr_t)p + sizeof(uint32_t));
}
// Required extension 3 - byte counter
if ( (master_record->flags & FLAG_BYTES_64 ) != 0 ) {
// 64bit byte counter
value64_t l, *v = (value64_t *)p;
l.val.val64 = master_record->dOctets;
v->val.val32[0] = l.val.val32[0];
v->val.val32[1] = l.val.val32[1];
p = (void *)((pointer_addr_t)p + sizeof(uint64_t));
} else {
// 32bit bytes counter
*((uint32_t *)p) = master_record->dOctets;
p = (void *)((pointer_addr_t)p + sizeof(uint32_t));
}
// Process optional extensions
i=0;
while ( extension_map->ex_id[i] ) {
switch (extension_map->ex_id[i++]) {
// 0 - 3 should never be in an extension table so - ignore it
case 0:
case 1:
case 2:
case 3:
break;
case EX_IO_SNMP_2: { // input/output SNMP 2 byte
tpl_ext_4_t *tpl = (tpl_ext_4_t *)p;
tpl->input = master_record->input;
tpl->output = master_record->output;
p = (void *)tpl->data;
} break;
case EX_IO_SNMP_4: { // input/output SNMP 4 byte
tpl_ext_5_t *tpl = (tpl_ext_5_t *)p;
tpl->input = master_record->input;
tpl->output = master_record->output;
p = (void *)tpl->data;
} break;
case EX_AS_2: { // srcas/dstas 2 byte
tpl_ext_6_t *tpl = (tpl_ext_6_t *)p;
tpl->src_as = master_record->srcas;
tpl->dst_as = master_record->dstas;
p = (void *)tpl->data;
} break;
case EX_AS_4: { // srcas/dstas 4 byte
tpl_ext_7_t *tpl = (tpl_ext_7_t *)p;
tpl->src_as = master_record->srcas;
tpl->dst_as = master_record->dstas;
p = (void *)tpl->data;
} break;
case EX_MULIPLE: {
tpl_ext_8_t *tpl = (tpl_ext_8_t *)p;
// use a 32 bit int to copy all 4 fields
tpl->any = master_record->any;
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_v4: {
tpl_ext_9_t *tpl = (tpl_ext_9_t *)p;
tpl->nexthop = master_record->ip_nexthop.v4;
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_v6: {
tpl_ext_10_t *tpl = (tpl_ext_10_t *)p;
tpl->nexthop[0] = master_record->ip_nexthop.v6[0];
tpl->nexthop[1] = master_record->ip_nexthop.v6[1];
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_BGP_v4: {
tpl_ext_11_t *tpl = (tpl_ext_11_t *)p;
tpl->bgp_nexthop = master_record->bgp_nexthop.v4;
p = (void *)tpl->data;
} break;
case EX_NEXT_HOP_BGP_v6: {
tpl_ext_12_t *tpl = (tpl_ext_12_t *)p;
tpl->bgp_nexthop[0] = master_record->bgp_nexthop.v6[0];
tpl->bgp_nexthop[1] = master_record->bgp_nexthop.v6[1];
p = (void *)tpl->data;
} break;
case EX_VLAN: {
tpl_ext_13_t *tpl = (tpl_ext_13_t *)p;
tpl->src_vlan = master_record->src_vlan;
tpl->dst_vlan = master_record->dst_vlan;
p = (void *)tpl->data;
} break;
case EX_OUT_PKG_4: {
tpl_ext_14_t *tpl = (tpl_ext_14_t *)p;
tpl->out_pkts = master_record->out_pkts;
p = (void *)tpl->data;
} break;
case EX_OUT_PKG_8: {
tpl_ext_15_t v, *tpl = (tpl_ext_15_t *)p;
v.out_pkts = master_record->out_pkts;
tpl->v[0] = v.v[0];
tpl->v[1] = v.v[1];
p = (void *)tpl->data;
} break;
case EX_OUT_BYTES_4: {
tpl_ext_16_t *tpl = (tpl_ext_16_t *)p;
tpl->out_bytes = master_record->out_bytes;
p = (void *)tpl->data;
} break;
case EX_OUT_BYTES_8: {
tpl_ext_17_t v, *tpl = (tpl_ext_17_t *)p;
v.out_bytes = master_record->out_bytes;
tpl->v[0] = v.v[0];
tpl->v[1] = v.v[1];
p = (void *)tpl->data;
} break;
case EX_AGGR_FLOWS_4: {
tpl_ext_18_t *tpl = (tpl_ext_18_t *)p;
tpl->aggr_flows = master_record->aggr_flows;
p = (void *)tpl->data;
} break;
case EX_AGGR_FLOWS_8: {
tpl_ext_19_t v, *tpl = (tpl_ext_19_t *)p;
v.aggr_flows = master_record->aggr_flows;
tpl->v[0] = v.v[0];
tpl->v[1] = v.v[1];
p = (void *)tpl->data;
} break;
case EX_MAC_1: {
tpl_ext_20_t v, *tpl = (tpl_ext_20_t *)p;
v.in_src_mac = master_record->in_src_mac;
tpl->v1[0] = v.v1[0];
tpl->v1[1] = v.v1[1];
v.out_dst_mac = master_record->out_dst_mac;
tpl->v2[0] = v.v2[0];
tpl->v2[1] = v.v2[1];
p = (void *)tpl->data;
} break;
case EX_MAC_2: {
tpl_ext_21_t v, *tpl = (tpl_ext_21_t *)p;
v.in_dst_mac = master_record->in_dst_mac;
tpl->v1[0] = v.v1[0];
tpl->v1[1] = v.v1[1];
v.out_src_mac = master_record->out_src_mac;
tpl->v2[0] = v.v2[0];
tpl->v2[1] = v.v2[1];
p = (void *)tpl->data;
} break;
case EX_MPLS: {
tpl_ext_22_t *tpl = (tpl_ext_22_t *)p;
int j;
for (j=0; j<10; j++ ) {
tpl->mpls_label[j] = master_record->mpls_label[j];
}
p = (void *)tpl->data;
} break;
case EX_ROUTER_IP_v4: {
tpl_ext_23_t *tpl = (tpl_ext_23_t *)p;
tpl->router_ip = master_record->ip_router.v4;
p = (void *)tpl->data;
} break;
case EX_ROUTER_IP_v6: {
tpl_ext_24_t *tpl = (tpl_ext_24_t *)p;
tpl->router_ip[0] = master_record->ip_router.v6[0];
tpl->router_ip[1] = master_record->ip_router.v6[1];
p = (void *)tpl->data;
} break;
case EX_ROUTER_ID: {
tpl_ext_25_t *tpl = (tpl_ext_25_t *)p;
tpl->engine_type = master_record->engine_type;
tpl->engine_id = master_record->engine_id;
p = (void *)tpl->data;
} break;
case EX_BGPADJ: {
tpl_ext_26_t *tpl = (tpl_ext_26_t *)p;
tpl->bgpNextAdjacentAS = master_record->bgpNextAdjacentAS;
tpl->bgpPrevAdjacentAS = master_record->bgpPrevAdjacentAS;
p = (void *)tpl->data;
} break;
case EX_RECEIVED: {
tpl_ext_27_t *tpl = (tpl_ext_27_t *)p;
tpl->received = master_record->received;
p = (void *)tpl->data;
} break;
#ifdef NSEL
case EX_NSEL_COMMON: {
tpl_ext_37_t *tpl = (tpl_ext_37_t *)p;
tpl->event_time = master_record->event_time;
tpl->conn_id = master_record->conn_id;
tpl->fw_event = master_record->event;
tpl->nsel_icmp = master_record->icmp;
tpl->fill = 0;
tpl->fill2 = 0;
tpl->fw_xevent = master_record->fw_xevent;
p = (void *)tpl->data;
} break;
case EX_NSEL_XLATE_PORTS: {
tpl_ext_38_t *tpl = (tpl_ext_38_t *)p;
tpl->xlate_src_port = master_record->xlate_src_port;
tpl->xlate_dst_port = master_record->xlate_dst_port;
p = (void *)tpl->data;
} break;
case EX_NSEL_XLATE_IP_v4: {
tpl_ext_39_t *tpl = (tpl_ext_39_t *)p;
tpl->xlate_src_ip = master_record->xlate_src_ip.v4;
tpl->xlate_dst_ip = master_record->xlate_dst_ip.v4;
p = (void *)tpl->data;
} break;
case EX_NSEL_XLATE_IP_v6: {
tpl_ext_40_t *tpl = (tpl_ext_40_t *)p;
tpl->xlate_src_ip[0] = master_record->xlate_src_ip.v6[0];
tpl->xlate_src_ip[1] = master_record->xlate_src_ip.v6[1];
p = (void *)tpl->data;
tpl->xlate_dst_ip[0] = master_record->xlate_dst_ip.v6[0];
tpl->xlate_dst_ip[1] = master_record->xlate_dst_ip.v6[1];
p = (void *)tpl->data;
} break;
case EX_NSEL_ACL: {
tpl_ext_41_t *tpl = (tpl_ext_41_t *)p;
int j;
for (j=0; j<3; j++) {
tpl->ingress_acl_id[j] = master_record->ingress_acl_id[j];
tpl->egress_acl_id[j] = master_record->egress_acl_id[j];
}
p = (void *)tpl->data;
} break;
case EX_NSEL_USER: {
tpl_ext_42_t *tpl = (tpl_ext_42_t *)p;
strncpy((void *)tpl->username, (void *)master_record->username, sizeof(tpl->username));
tpl->username[sizeof(tpl->username)-1] = '\0'; // safety 0
p = (void *)tpl->data;
} break;
case EX_NSEL_USER_MAX: {
tpl_ext_43_t *tpl = (tpl_ext_43_t *)p;
strncpy((void *)tpl->username, (void *)master_record->username, sizeof(tpl->username));
tpl->username[sizeof(tpl->username)-1] = '\0'; // safety 0
p = (void *)tpl->data;
} break;
case EX_NEL_COMMON: {
tpl_ext_46_t *tpl = (tpl_ext_46_t *)p;
tpl->nat_event = master_record->event;
tpl->fill = 0;
tpl->flags = 0;
tpl->egress_vrfid = master_record->egress_vrfid;
tpl->ingress_vrfid = master_record->ingress_vrfid;
p = (void *)tpl->data;
} break;
case EX_PORT_BLOCK_ALLOC: {
tpl_ext_48_t *tpl = (tpl_ext_48_t *)p;
tpl->block_start = master_record->block_start;
tpl->block_end = master_record->block_end;
tpl->block_step = master_record->block_step;
tpl->block_size = master_record->block_size;
p = (void *)tpl->data;
} break;
#endif
}
}
nffile->block_header->size += required;
nffile->block_header->NumRecords++;
#ifdef DEVEL
if ( ((pointer_addr_t)p - (pointer_addr_t)nffile->buff_ptr) != required ) {
fprintf(stderr, "Packrecord: size missmatch: required: %i, written: %li!\n",
required, (long)((ptrdiff_t)p - (ptrdiff_t)nffile->buff_ptr));
exit(255);
}
#endif
nffile->buff_ptr = p;
} // End of PackRecord
#endif
static inline void AppendToBuffer(nffile_t *nffile, void *record, size_t required) {
// flush current buffer to disc
if ( !CheckBufferSpace(nffile, required)) {
return;
}
// enough buffer space available at this point
memcpy(nffile->buff_ptr, record, required);
// update stat
nffile->block_header->NumRecords++;
nffile->block_header->size += required;
// advance write pointer
nffile->buff_ptr = (void *)((pointer_addr_t)nffile->buff_ptr + required);
} // End of AppendToBuffer

550
bin/nfgen.c Normal file
View File

@ -0,0 +1,550 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfgen.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdarg.h>
#include <errno.h>
#include <time.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "nfx.h"
#include "nfnet.h"
#include "nf_common.h"
#include "util.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "netflow_v5_v7.h"
extern extension_descriptor_t extension_descriptor[];
static time_t when;
uint32_t offset = 10;
uint32_t msecs = 10;
static extension_info_t extension_info;
#define NEED_PACKRECORD 1
#include "nffile_inline.c"
#undef NEED_PACKRECORD
void *GenRecord(int af, void *buff_ptr, char *src_ip, char *dst_ip, int src_port, int dst_port,
int proto, int tcp_flags, int tos, uint64_t packets, uint64_t bytes, int src_as, int dst_as);
static void SetIPaddress(master_record_t *record, int af, char *src_ip, char *dst_ip);
static void SetNextIPaddress(master_record_t *record, int af, char *next_ip);
static void SetRouterIPaddress(master_record_t *record, int af, char *next_ip);
static void SetBGPNextIPaddress(master_record_t *record, int af, char *next_ip);
static void UpdateRecord(master_record_t *record);
static void SetIPaddress(master_record_t *record, int af, char *src_ip, char *dst_ip) {
if ( af == PF_INET6 ) {
SetFlag(record->flags, FLAG_IPV6_ADDR);
inet_pton(PF_INET6, src_ip, &(record->v6.srcaddr[0]));
inet_pton(PF_INET6, dst_ip, &(record->v6.dstaddr[0]));
record->v6.srcaddr[0] = ntohll(record->v6.srcaddr[0]);
record->v6.srcaddr[1] = ntohll(record->v6.srcaddr[1]);
record->v6.dstaddr[0] = ntohll(record->v6.dstaddr[0]);
record->v6.dstaddr[1] = ntohll(record->v6.dstaddr[1]);
} else {
ClearFlag(record->flags, FLAG_IPV6_ADDR);
inet_pton(PF_INET, src_ip, &record->v4.srcaddr);
inet_pton(PF_INET, dst_ip, &record->v4.dstaddr);
record->v4.srcaddr = ntohl(record->v4.srcaddr);
record->v4.dstaddr = ntohl(record->v4.dstaddr);
}
} // End of SetIPaddress
static void SetNextIPaddress(master_record_t *record, int af, char *next_ip) {
if ( af == PF_INET6 ) {
SetFlag(record->flags, FLAG_IPV6_NH);
inet_pton(PF_INET6, next_ip, &(record->ip_nexthop.v6[0]));
record->ip_nexthop.v6[0] = ntohll(record->ip_nexthop.v6[0]);
record->ip_nexthop.v6[1] = ntohll(record->ip_nexthop.v6[1]);
} else {
ClearFlag(record->flags, FLAG_IPV6_NH);
inet_pton(PF_INET, next_ip, &record->ip_nexthop.v4);
record->ip_nexthop.v4 = ntohl(record->ip_nexthop.v4);
}
} // End of SetNextIPaddress
static void SetRouterIPaddress(master_record_t *record, int af, char *next_ip) {
if ( af == PF_INET6 ) {
SetFlag(record->flags, FLAG_IPV6_NH);
inet_pton(PF_INET6, next_ip, &(record->ip_router.v6[0]));
record->ip_router.v6[0] = ntohll(record->ip_router.v6[0]);
record->ip_router.v6[1] = ntohll(record->ip_router.v6[1]);
} else {
ClearFlag(record->flags, FLAG_IPV6_NH);
inet_pton(PF_INET, next_ip, &record->ip_router.v4);
record->ip_router.v4 = ntohl(record->ip_router.v4);
}
} // End of SetRouterIPaddress
static void SetBGPNextIPaddress(master_record_t *record, int af, char *next_ip) {
if ( af == PF_INET6 ) {
SetFlag(record->flags, FLAG_IPV6_NHB);
inet_pton(PF_INET6, next_ip, &(record->bgp_nexthop.v6[0]));
record->bgp_nexthop.v6[0] = ntohll(record->bgp_nexthop.v6[0]);
record->bgp_nexthop.v6[1] = ntohll(record->bgp_nexthop.v6[1]);
} else {
ClearFlag(record->flags, FLAG_IPV6_NHB);
inet_pton(PF_INET, next_ip, &record->bgp_nexthop.v4);
record->bgp_nexthop.v4 = ntohl(record->bgp_nexthop.v4);
}
} // End of SetBGPNextIPaddress
static void UpdateRecord(master_record_t *record) {
record->first = when;
record->last = when + offset;
record->msec_first = msecs;
record->msec_last = msecs + 10;
when += 10;
offset += 10;
msecs += 100;
if ( msecs > 1000 )
msecs = msecs - 1000;
record->fwd_status++;
} // End of UpdateRecord
int main( int argc, char **argv ) {
int i, c;
master_record_t record;
nffile_t *nffile;
when = ISO2UNIX(strdup("200407111030"));
while ((c = getopt(argc, argv, "h")) != EOF) {
switch(c) {
case 'h':
break;
default:
fprintf(stderr, "ERROR: Unsupported option: '%c'\n", c);
exit(255);
}
}
extension_info.map = (extension_map_t *)malloc(sizeof(extension_map_t) + 32 * sizeof(uint16_t));
if ( !extension_info.map ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
exit(255);
}
extension_info.map->type = ExtensionMapType;
extension_info.map->map_id = 0;
i = 0;
extension_info.map->ex_id[i++] = EX_IO_SNMP_2;
extension_info.map->ex_id[i++] = EX_AS_2;
extension_info.map->ex_id[i++] = EX_MULIPLE;
extension_info.map->ex_id[i++] = EX_NEXT_HOP_v4;
extension_info.map->ex_id[i++] = EX_NEXT_HOP_BGP_v4;
extension_info.map->ex_id[i++] = EX_VLAN;
extension_info.map->ex_id[i++] = EX_OUT_PKG_4;
extension_info.map->ex_id[i++] = EX_OUT_BYTES_4;
extension_info.map->ex_id[i++] = EX_AGGR_FLOWS_4;
extension_info.map->ex_id[i++] = EX_MAC_1;
extension_info.map->ex_id[i++] = EX_MAC_2;
extension_info.map->ex_id[i++] = EX_MPLS;
extension_info.map->ex_id[i++] = EX_ROUTER_IP_v4;
extension_info.map->ex_id[i++] = EX_ROUTER_ID;
extension_info.map->ex_id[i++] = EX_BGPADJ;
extension_info.map->ex_id[i] = 0;
extension_info.map->size = sizeof(extension_map_t) + i * sizeof(uint16_t);
// align 32bits
if (( extension_info.map->size & 0x3 ) != 0 ) {
extension_info.map->size += 4 - ( extension_info.map->size & 0x3 );
}
extension_info.map->extension_size = 0;
i=0;
while (extension_info.map->ex_id[i]) {
int id = extension_info.map->ex_id[i];
extension_info.map->extension_size += extension_descriptor[id].size;
i++;
}
memset((void *)&record, 0, sizeof(record));
nffile = OpenNewFile("-", NULL, 0, 0, NULL);
if ( !nffile ) {
exit(255);
}
AppendToBuffer(nffile, (void *)extension_info.map, extension_info.map->size);
record.map_ref = extension_info.map;
record.type = CommonRecordType;
record.flags = 0;
record.exporter_sysid = 1;
record.tcp_flags = 1;
record.tos = 2;
record.fwd_status = 0;
record.srcport = 1024;
record.dstport = 25;
record.prot = IPPROTO_TCP;
record.input = 12;
record.output = 14;
record.srcas = 775;
record.dstas = 8404;
SetIPaddress(&record, PF_INET, "172.16.1.66", "192.168.170.100");
SetNextIPaddress(&record, PF_INET, "172.72.1.2");
SetBGPNextIPaddress(&record, PF_INET, "172.73.2.3");
SetRouterIPaddress(&record, PF_INET, "127.0.0.1");
record.engine_type = 5;
record.engine_id = 6;
record.dPkts = 202;
record.dOctets = 303;
record.dst_tos = 128;
record.dir = 1;
record.src_mask = 16;
record.dst_mask = 24;
record.src_vlan = 82;
record.dst_vlan = 93;
record.out_pkts = 212;
record.out_bytes = 3234;
record.aggr_flows = 3;
record.in_src_mac = 0x0234567890aaLL;
record.out_dst_mac = 0xffeeddccbbaaLL;
record.out_src_mac = 0xaa3456789002LL;
record.in_dst_mac = 0xaaeeddccbbffLL;
record.mpls_label[0] = 1010 << 4;
record.mpls_label[1] = 2020 << 4;
record.mpls_label[2] = 3030 << 4;
record.mpls_label[3] = 4040 << 4;
record.mpls_label[4] = 5050 << 4;
record.mpls_label[5] = 6060 << 4;
record.mpls_label[6] = 7070 << 4;
record.mpls_label[7] = 8080 << 4;
record.mpls_label[8] = 9090 << 4;
record.mpls_label[9] = (100100 << 4) + 1;
record.client_nw_delay_usec = 2;
record.server_nw_delay_usec = 22;
record.appl_latency_usec = 222;
record.bgpNextAdjacentAS = 45804;
record.bgpPrevAdjacentAS = 32775;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.2.66", "192.168.170.101");
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
record.dPkts = 101;
record.dOctets = 102;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.3.66", "192.168.170.102");
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.4.66", "192.168.170.103");
record.srcport = 2024;
record.prot = IPPROTO_UDP;
record.tcp_flags = 1;
record.tos = 1;
record.dPkts = 1001;
record.dOctets = 1002;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.5.66", "192.168.170.104");
record.srcport = 3024;
record.prot = 51;
record.tcp_flags = 2;
record.tos = 2;
record.dPkts = 10001;
record.dOctets = 10002;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.6.66", "192.168.170.105");
record.srcport = 4024;
record.prot = IPPROTO_TCP;
record.tcp_flags = 4;
record.tos = 3;
record.dPkts = 100001;
record.dOctets = 100002;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.7.66", "192.168.170.106");
record.srcport = 5024;
record.tcp_flags = 8;
record.tos = 4;
record.dPkts = 1000001;
record.dOctets = 1000002;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.8.66", "192.168.170.107");
record.tcp_flags = 1;
record.tos = 4;
record.dPkts = 10000001;
record.dOctets = 1001;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.9.66", "192.168.170.108");
record.srcport = 6024;
record.tcp_flags = 16;
record.tos = 5;
record.dPkts = 500;
record.dOctets = 10000001;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.10.66", "192.168.170.109");
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.11.66", "192.168.170.110");
record.srcport = 7024;
record.tcp_flags = 32;
record.tos = 255;
record.dPkts = 5000;
record.dOctets = 100000001;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.12.66", "192.168.170.111");
record.srcport = 8024;
record.tcp_flags = 63;
record.tos = 0;
record.dOctets = 1000000001;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.13.66", "192.168.170.112");
record.srcport = 0;
record.dstport = 8;
record.prot = 1;
record.tcp_flags = 0;
record.tos = 0;
record.dPkts = 50002;
record.dOctets = 50000;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.160.160.166", "172.160.160.180");
record.srcport = 10024;
record.dstport = 25000;
record.prot = IPPROTO_TCP;
record.dPkts = 500001;
record.dOctets = 500000;
fprintf(stderr, "IPv4 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET6, "fe80::2110:abcd:1234:0", "fe80::2110:abcd:1235:4321");
// SetNextIPaddress(&record, PF_INET6, "2003:234:aabb::211:24ff:fe80:d01e");
// SetBGPNextIPaddress(&record, PF_INET6, "2004:234:aabb::211:24ff:fe80:d01e");
record.srcport = 1024;
record.dstport = 25;
record.tcp_flags = 27;
record.dPkts = 10;
record.dOctets = 15100;
fprintf(stderr, "IPv6 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET6, "2001:234:aabb::211:24ff:fe80:d01e", "2001:620::8:203:baff:fe52:38e5");
record.srcport = 10240;
record.dstport = 52345;
record.dPkts = 10100;
record.dOctets = 15000000;
fprintf(stderr, "IPv6 32bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
record.dPkts = 10100000;
record.dOctets = 0x100000000LL;
fprintf(stderr, "IPv6 32bit packets 64bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
record.dPkts = 0x100000000LL;
record.dOctets = 15000000;
fprintf(stderr, "IPv6 64bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
record.dOctets = 0x200000000LL;
fprintf(stderr, "IPv6 64bit packets 64bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.14.18", "192.168.170.113");
// SetNextIPaddress(&record, PF_INET, "172.72.1.2");
// SetBGPNextIPaddress(&record, PF_INET, "172.73.2.3");
record.srcport = 10240;
record.dstport = 52345;
record.dPkts = 10100000;
record.dOctets = 0x100000000LL;
fprintf(stderr, "IPv4 32bit packets 64bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.15.18", "192.168.170.114");
record.dPkts = 0x100000000LL;
record.dOctets = 15000000;
fprintf(stderr, "IPv4 64bit packets 32bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
SetIPaddress(&record, PF_INET, "172.16.16.18", "192.168.170.115");
record.dOctets = 0x200000000LL;
fprintf(stderr, "IPv4 64bit packets 64bit bytes\n");
UpdateRecord(&record);
PackRecord(&record, nffile);
extension_info.map->ex_id[0] = EX_IO_SNMP_4;
extension_info.map->extension_size = 0;
i=0;
while (extension_info.map->ex_id[i]) {
int id = extension_info.map->ex_id[i];
extension_info.map->extension_size += extension_descriptor[id].size;
i++;
}
memcpy(nffile->buff_ptr, (void *)extension_info.map, extension_info.map->size);
nffile->buff_ptr += extension_info.map->size;
nffile->block_header->NumRecords++;
nffile->block_header->size += extension_info.map->size;
UpdateRecord(&record);
fprintf(stderr, "4 bytes interfaces, 2 bytes AS numbers %d %d\n", record.fwd_status, nffile->block_header->NumRecords);
PackRecord(&record, nffile);
extension_info.map->ex_id[0] = EX_IO_SNMP_2;
extension_info.map->ex_id[1] = EX_AS_4;
extension_info.map->extension_size = 0;
i=0;
while (extension_info.map->ex_id[i]) {
int id = extension_info.map->ex_id[i];
extension_info.map->extension_size += extension_descriptor[id].size;
i++;
}
memcpy(nffile->buff_ptr, (void *)extension_info.map, extension_info.map->size);
nffile->buff_ptr += extension_info.map->size;
nffile->block_header->NumRecords++;
nffile->block_header->size += extension_info.map->size;
UpdateRecord(&record);
fprintf(stderr, "2 bytes interfaces, 4 bytes AS numbers %d %d\n", record.fwd_status, nffile->block_header->NumRecords);
PackRecord(&record, nffile);
extension_info.map->ex_id[0] = EX_IO_SNMP_4;
extension_info.map->extension_size = 0;
i=0;
while (extension_info.map->ex_id[i]) {
int id = extension_info.map->ex_id[i];
extension_info.map->extension_size += extension_descriptor[id].size;
i++;
}
memcpy(nffile->buff_ptr, (void *)extension_info.map, extension_info.map->size);
nffile->buff_ptr += extension_info.map->size;
nffile->block_header->NumRecords++;
nffile->block_header->size += extension_info.map->size;
UpdateRecord(&record);
fprintf(stderr, "4 bytes interfaces, 4 bytes AS numbers %d %d\n", record.fwd_status, nffile->block_header->NumRecords);
PackRecord(&record, nffile);
if ( nffile->block_header->NumRecords ) {
if ( WriteBlock(nffile) <= 0 ) {
fprintf(stderr, "Failed to write output buffer to disk: '%s'" , strerror(errno));
}
}
return 0;
}

933
bin/nflowcache.c Executable file
View File

@ -0,0 +1,933 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nflowcache.c 40 2009-12-16 10:41:44Z haag $
*
* $LastChangedRevision: 40 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <time.h>
#include <string.h>
#include <ctype.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "nfx.h"
#include "nflowcache.h"
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#define ALIGN_BYTES (offsetof (struct { char x; uint64_t y; }, y) - 1)
extern int hash_hit;
extern int hash_miss;
extern int hash_skip;
/* function prototypes */
static void MemoryHandle_free(MemoryHandle_t *handle);
static int MemoryHandle_init(MemoryHandle_t *handle);
static inline void *MemoryHandle_get(MemoryHandle_t *handle, uint32_t size);
static inline FlowTableRecord_t *hash_insert_FlowTable(uint32_t index_cache, void *flowkey, common_record_t *flow_record);
static inline int TimeMsec_CMP(time_t t1, uint16_t offset1, time_t t2, uint16_t offset2 );
static inline uint32_t SuperFastHash (const char * data, int len);
static inline void New_Hash_Key(void *keymem, master_record_t *flow_record, int swap_flow);
/* locals */
static hash_FlowTable FlowTable;
static int initialised = 0;
uint32_t loopcnt = 0;
typedef struct aggregate_param_s {
uint32_t size; // size of parameter in bytes
uint32_t offset; // offset in master record
uint64_t mask; // mask for this value in master record
uint64_t shift; // bis shift for this value in master record
} aggregate_param_t;
static struct aggregate_info_s {
char *aggregate_token; // name of aggregation parameter
aggregate_param_t param; // the parameter array
int merge; // apply bis mask? => -1 no, otherwise index of mask[] array
int active; // is this parameter set?
char *fmt; // for automatic output format generation
} aggregate_info [] = {
{ "srcip4", { 8, OffsetSrcIPv6a, MaskIPv6, ShiftIPv6 }, 0, 0, "%sa" },
{ "srcip4", { 8, OffsetSrcIPv6b, MaskIPv6, ShiftIPv6 }, 1, 0, NULL },
{ "srcip6", { 8, OffsetSrcIPv6a, MaskIPv6, ShiftIPv6 }, 0, 0, "%sa" },
{ "srcip6", { 8, OffsetSrcIPv6b, MaskIPv6, ShiftIPv6 }, 1, 0, NULL },
{ "srcnet", { 8, OffsetSrcIPv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%sn" },
{ "srcnet", { 8, OffsetSrcIPv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "dstnet", { 8, OffsetDstIPv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%dn" },
{ "dstnet", { 8, OffsetDstIPv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "srcip", { 8, OffsetSrcIPv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%sa" },
{ "srcip", { 8, OffsetSrcIPv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "dstip", { 8, OffsetDstIPv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%da" },
{ "dstip", { 8, OffsetDstIPv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "dstip4", { 8, OffsetDstIPv6a, MaskIPv6, ShiftIPv6 }, 0, 0, "%da" },
{ "dstip4", { 8, OffsetDstIPv6b, MaskIPv6, ShiftIPv6 }, 1, 0, NULL },
{ "dstip6", { 8, OffsetDstIPv6a, MaskIPv6, ShiftIPv6 }, 0, 0, "%da" },
{ "dstip6", { 8, OffsetDstIPv6b, MaskIPv6, ShiftIPv6 }, 1, 0, NULL },
{ "next", { 8, OffsetNexthopv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%nh" },
{ "next", { 8, OffsetNexthopv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "bgpnext", { 8, OffsetBGPNexthopv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%nhb" },
{ "bgpnext", { 8, OffsetBGPNexthopv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "router", { 8, OffsetRouterv6a, MaskIPv6, ShiftIPv6 }, -1, 0, "%ra" },
{ "router", { 8, OffsetRouterv6b, MaskIPv6, ShiftIPv6 }, -1, 0, NULL },
{ "insrcmac", { 8, OffsetInSrcMAC, MaskMac, ShiftIPv6 }, -1, 0, "%ismc" },
{ "outdstmac", { 8, OffsetOutDstMAC, MaskMac, ShiftIPv6 }, -1, 0, "%odmc" },
{ "indstmac", { 8, OffsetInDstMAC, MaskMac, ShiftIPv6 }, -1, 0, "%idmc" },
{ "outsrcmac", { 8, OffsetOutSrcMAC, MaskMac, ShiftIPv6 }, -1, 0, "%osmc" },
{ "srcas", { 4, OffsetAS, MaskSrcAS, ShiftSrcAS }, -1, 0, "%sas" },
{ "dstas", { 4, OffsetAS, MaskDstAS, ShiftDstAS }, -1, 0, "%das" },
{ "nextas", { 4, OffsetBGPadj, MaskBGPadjNext, ShiftBGPadjNext }, -1, 0, "%nas" },
{ "prevas", { 4, OffsetBGPadj, MaskBGPadjPrev, ShiftBGPadjPrev }, -1, 0, "%pas" },
{ "inif", { 4, OffsetInOut, MaskInput, ShiftInput }, -1, 0, "%in" },
{ "outif", { 4, OffsetInOut, MaskOutput, ShiftOutput }, -1, 0, "%out" },
{ "mpls1", { 4, OffsetMPLS12, MaskMPLSlabelOdd, ShiftMPLSlabelOdd }, -1, 0, "%mpls1"},
{ "mpls2", { 4, OffsetMPLS12, MaskMPLSlabelEven, ShiftMPLSlabelEven }, -1, 0, "%mpls2"},
{ "mpls3", { 4, OffsetMPLS34, MaskMPLSlabelOdd, ShiftMPLSlabelOdd }, -1, 0, "%mpls3"},
{ "mpls4", { 4, OffsetMPLS34, MaskMPLSlabelEven, ShiftMPLSlabelEven }, -1, 0, "%mpls4"},
{ "mpls5", { 4, OffsetMPLS56, MaskMPLSlabelOdd, ShiftMPLSlabelOdd }, -1, 0, "%mpls5"},
{ "mpls6", { 4, OffsetMPLS56, MaskMPLSlabelEven, ShiftMPLSlabelEven }, -1, 0, "%mpls6"},
{ "mpls7", { 4, OffsetMPLS78, MaskMPLSlabelOdd, ShiftMPLSlabelOdd }, -1, 0, "%mpls7"},
{ "mpls8", { 4, OffsetMPLS78, MaskMPLSlabelEven, ShiftMPLSlabelEven }, -1, 0, "%mpls8"},
{ "mpls9", { 4, OffsetMPLS910, MaskMPLSlabelOdd, ShiftMPLSlabelOdd }, -1, 0, "%mpls9"},
{ "mpls10", { 4, OffsetMPLS910, MaskMPLSlabelEven, ShiftMPLSlabelEven }, -1, 0, "%mpls10"},
{ "srcport", { 2, OffsetPort, MaskSrcPort, ShiftSrcPort }, -1, 0, "%sp" },
{ "dstport", { 2, OffsetPort, MaskDstPort, ShiftDstPort }, -1, 0, "%dp" },
{ "srcvlan", { 2, OffsetVlan, MaskSrcVlan, ShiftSrcVlan }, -1, 0, "%svln" },
{ "dstvlan", { 2, OffsetVlan, MaskDstVlan, ShiftDstVlan }, -1, 0, "%dvln" },
{ "srcmask", { 1, OffsetMask, MaskSrcMask, ShiftSrcMask }, -1, 0, "%smk" },
{ "dstmask", { 1, OffsetMask, MaskDstMask, ShiftDstMask }, -1, 0, "%dmk" },
{ "proto", { 1, OffsetProto, MaskProto, ShiftProto }, -1, 0, "%pr" },
{ "tos", { 1, OffsetTos, MaskTos, ShiftTos }, -1, 0, "%tos" },
{ "srctos", { 1, OffsetTos, MaskTos, ShiftTos }, -1, 0, "%stos" },
{ "dsttos", { 1, OffsetDstTos, MaskDstTos, ShiftDstTos }, -1, 0, "%dtos" },
{ NULL, { 0, 0, 0, 0}, 0, 0, NULL}
};
typedef struct Default_key_s {
uint16_t srcport;
uint16_t dstport;
uint64_t srcaddr[2];
uint64_t dstaddr[2];
uint32_t proto;
} Default_key_t;
static aggregate_param_t *aggregate_stack = NULL;
static uint32_t aggregate_key_len = sizeof(Default_key_t);
static uint32_t bidir_flows = 0;
// counter indices
// The array size of FlowTableRecord_t array counter must match.
enum CNT_IND { FLOWS = 0, INPACKETS, INBYTES, OUTPACKETS, OUTBYTES };
#include "applybits_inline.c"
/* Functions */
static inline int TimeMsec_CMP(time_t t1, uint16_t offset1, time_t t2, uint16_t offset2 ) {
if ( t1 > t2 )
return 1;
if ( t2 > t1 )
return 2;
// else t1 == t2 - offset is now relevant
if ( offset1 > offset2 )
return 1;
if ( offset2 > offset1 )
return 2;
else
// both times are the same
return 0;
} // End of TimeMsec_CMP
static int MemoryHandle_init(MemoryHandle_t *handle) {
handle->memblock = (void **)calloc(MaxMemBlocks, sizeof(void *));
if ( !handle->memblock ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
handle->BlockSize = MemBlockSize;
handle->memblock[0] = malloc(MemBlockSize);
handle->MaxBlocks = MaxMemBlocks;
handle->NumBlocks = 1;
handle->CurrentBlock = 0;
handle->Allocted = 0;
return 1;
} // End of MemoryHandle_init
static void MemoryHandle_free(MemoryHandle_t *handle) {
int i;
dbg_printf("MEM: NumBlocks: %u\n", handle->NumBlocks);
for ( i=0; i < handle->NumBlocks; i++ ) {
free(handle->memblock[i]);
}
handle->NumBlocks = 0;
handle->CurrentBlock = 0;
handle->Allocted = 0;
free((void *)handle->memblock);
handle->memblock = NULL;
handle->MaxBlocks = 0;
} // End of MemoryHandle_free
static inline void *MemoryHandle_get(MemoryHandle_t *handle, uint32_t size) {
void *p;
uint32_t aligned_size;
// make sure size of memory is aligned
aligned_size = (((u_int)(size) + ALIGN_BYTES) &~ ALIGN_BYTES);
if ( (handle->Allocted+aligned_size) > MemBlockSize ) {
// not enough space - allocate a new memblock
handle->CurrentBlock++;
if ( handle->CurrentBlock >= handle->MaxBlocks ) {
// we run out in memblock array - re-allocate memblock array
handle->MaxBlocks += MaxMemBlocks;
handle->memblock = (void **)realloc(handle->memblock, handle->MaxBlocks * sizeof(void *));
if ( !handle->memblock ) {
fprintf(stderr, "realloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
exit(255);
}
}
// allocate new memblock
p = malloc(MemBlockSize);
if ( !p ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
exit(255);
}
handle->memblock[handle->CurrentBlock] = p;
// reset counter for new memblock
handle->Allocted = 0;
handle->NumBlocks++;
}
// enough space available in current memblock
p = handle->memblock[handle->CurrentBlock] + handle->Allocted;
handle->Allocted += aligned_size;
dbg_printf("Mem Handle: Requested: %u, aligned: %u, ptr: %lu\n", size, aligned_size, (long unsigned)p);
return p;
} // End of MemoryHandle_get
hash_FlowTable *GetFlowTable(void) {
return &FlowTable;
} // End of GetFlowTable
int Init_FlowTable(void) {
uint32_t maxindex;
maxindex = (1 << HashBits);
FlowTable.IndexMask = maxindex -1;
FlowTable.NumBits = HashBits;
FlowTable.NumRecords = 0;
FlowTable.bucket = (FlowTableRecord_t **)calloc(maxindex, sizeof(FlowTableRecord_t *));
FlowTable.bucketcache = (FlowTableRecord_t **)calloc(maxindex, sizeof(FlowTableRecord_t *));
if ( !FlowTable.bucket || !FlowTable.bucketcache ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
FlowTable.keysize = aggregate_key_len;
// keylen = number of uint64_t
FlowTable.keylen = aggregate_key_len >> 3; // aggregate_key_len / 8
if ( (aggregate_key_len & 0x7 ) != 0 )
FlowTable.keylen++;
dbg_printf("FlowTable.keysize %i bytes\n", FlowTable.keysize);
dbg_printf("FlowTable.keylen %i uint64_t\n", FlowTable.keylen);
if ( !MemoryHandle_init(&FlowTable.mem) )
return 0;
initialised = 1;
return 1;
} // End of Init_FlowTable
void Dispose_FlowTable(void) {
if ( !initialised )
return;
free((void *)FlowTable.bucket);
free((void *)FlowTable.bucketcache);
MemoryHandle_free(&FlowTable.mem);
FlowTable.NumRecords = 0;
FlowTable.bucket = NULL;
FlowTable.bucketcache = NULL;
} // End of Dispose_FlowTable
static inline FlowTableRecord_t *hash_lookup_FlowTable(uint32_t *index_cache, void *flowkey, master_record_t *flow_record) {
uint32_t index;
int bsize;
FlowTableRecord_t *record;
*index_cache = SuperFastHash((char *)flowkey, FlowTable.keysize);
index = *index_cache & FlowTable.IndexMask;
if ( FlowTable.bucket[index] == NULL ) {
hash_hit++;
return NULL;
}
record = FlowTable.bucket[index];
// skip records with different hash value ( full 32bit )
while ( record )
if ( record->hash != *index_cache ) {
hash_skip++;
record = record->next;
} else
break;
bsize = 0;
while ( record ) {
uint64_t *k1 = (uint64_t *)flowkey;
uint64_t *k2 = (uint64_t *)record->hash_key;
int i;
// compare key and break as soon as keys do not match
i = 0;
while ( i < FlowTable.keylen ) {
if ( k1[i] == k2[i] )
i++;
else
break;
}
loopcnt += i;
if ( i == FlowTable.keylen ) {
// hit - record found
// some stats for debugging
if ( bsize == 0 )
hash_hit++;
else
hash_miss++;
return record;
}
record = record->next;
bsize++;
}
return NULL;
} // End of hash_lookup_FlowTable
inline static FlowTableRecord_t *hash_insert_FlowTable(uint32_t index_cache, void *flowkey, common_record_t *raw_record) {
FlowTableRecord_t *record;
uint32_t index = index_cache & FlowTable.IndexMask;
// allocate enough memory for the new flow including all additional information in FlowTableRecord_t
// MemoryHandle_get always succeeds. If no memory, MemoryHandle_get already exists cleanly
record = MemoryHandle_get(&FlowTable.mem, sizeof(FlowTableRecord_t) - sizeof(common_record_t) + raw_record->size);
record->next = NULL;
record->hash = index_cache;
record->hash_key = flowkey;
memcpy((void *)&record->flowrecord, (void *)raw_record, raw_record->size);
if ( FlowTable.bucket[index] == NULL )
FlowTable.bucket[index] = record;
else
FlowTable.bucketcache[index]->next = record;
FlowTable.bucketcache[index] = record;
FlowTable.NumRecords++;
return record;
} // End of hash_insert_FlowTable
void InsertFlow(common_record_t *raw_record, master_record_t *flow_record, extension_info_t *extension_info) {
FlowTableRecord_t *record;
// allocate enough memory for the new flow including all additional information in FlowTableRecord_t
// MemoryHandle_get always succeeds. If no memory, MemoryHandle_get already exits cleanly
record = MemoryHandle_get(&FlowTable.mem, sizeof(FlowTableRecord_t) - sizeof(common_record_t) + raw_record->size);
record->next = NULL;
record->hash = 0;
record->hash_key = NULL;
memcpy((void *)&record->flowrecord, (void *)raw_record, raw_record->size);
if ( FlowTable.bucket[0] == NULL )
FlowTable.bucket[0] = record;
else
FlowTable.bucketcache[0]->next = record;
FlowTable.bucketcache[0] = record;
// safe the extension map and exporter reference
record->map_info_ref = extension_info;
record->exp_ref = flow_record->exp_ref;
record->counter[INBYTES] = flow_record->dOctets;
record->counter[INPACKETS] = flow_record->dPkts;
record->counter[OUTBYTES] = flow_record->out_bytes;
record->counter[OUTPACKETS] = flow_record->out_pkts;
record->counter[FLOWS] = flow_record->aggr_flows ? flow_record->aggr_flows : 1;
FlowTable.NumRecords++;
} // End of InsertFlow
void AddFlow(common_record_t *raw_record, master_record_t *flow_record, extension_info_t *extension_info ) {
static void *keymem = NULL, *bidirkeymem = NULL;
FlowTableRecord_t *FlowTableRecord;
uint32_t index_cache;
if ( keymem == NULL ) {
keymem = MemoryHandle_get(&FlowTable.mem ,FlowTable.keysize );
// the last aligned word may not be fully used. set it to 0 to guarantee
// a proper comarison
// for 64 bit arch int == 8 bytes otherwise 4
((int *)keymem)[FlowTable.keylen-1] = 0;
}
New_Hash_Key(keymem, flow_record, 0);
// Update netflow statistics
FlowTableRecord = hash_lookup_FlowTable(&index_cache, keymem, flow_record);
if ( FlowTableRecord ) {
// flow record found - best case! update all fields
FlowTableRecord->counter[INBYTES] += flow_record->dOctets;
FlowTableRecord->counter[INPACKETS] += flow_record->dPkts;
FlowTableRecord->counter[OUTBYTES] += flow_record->out_bytes;
FlowTableRecord->counter[OUTPACKETS] += flow_record->out_pkts;
if ( TimeMsec_CMP(flow_record->first, flow_record->msec_first,
FlowTableRecord->flowrecord.first, FlowTableRecord->flowrecord.msec_first) == 2) {
FlowTableRecord->flowrecord.first = flow_record->first;
FlowTableRecord->flowrecord.msec_first = flow_record->msec_first;
}
if ( TimeMsec_CMP(flow_record->last, flow_record->msec_last,
FlowTableRecord->flowrecord.last, FlowTableRecord->flowrecord.msec_last) == 1) {
FlowTableRecord->flowrecord.last = flow_record->last;
FlowTableRecord->flowrecord.msec_last = flow_record->msec_last;
}
FlowTableRecord->counter[FLOWS] += flow_record->aggr_flows ? flow_record->aggr_flows : 1;
FlowTableRecord->flowrecord.tcp_flags |= flow_record->tcp_flags;
} else if ( !bidir_flows || ( flow_record->prot != IPPROTO_TCP && flow_record->prot != IPPROTO_UDP) ) {
// no flow record found and no TCP/UDP bidir flows. Insert flow record into hash
FlowTableRecord = hash_insert_FlowTable(index_cache, keymem, raw_record);
FlowTableRecord->counter[INBYTES] = flow_record->dOctets;
FlowTableRecord->counter[INPACKETS] = flow_record->dPkts;
FlowTableRecord->counter[OUTBYTES] = flow_record->out_bytes;
FlowTableRecord->counter[OUTPACKETS] = flow_record->out_pkts;
FlowTableRecord->counter[FLOWS] = flow_record->aggr_flows ? flow_record->aggr_flows : 1;
FlowTableRecord->map_info_ref = extension_info;
FlowTableRecord->exp_ref = flow_record->exp_ref;
// keymen got part of the cache
keymem = NULL;
} else {
// for bidir flows do
uint32_t bidir_index_cache;
// use tmp memory for bidir hash key to search for bidir flow
// we need it only to lookup
if ( bidirkeymem == NULL ) {
bidirkeymem = MemoryHandle_get(&FlowTable.mem ,FlowTable.keysize );
// the last aligned word may not be fully used. set it to 0 to guarantee
// a proper comarison
// for 64 bit arch int == 8 bytes otherwise 4
((int *)bidirkeymem)[FlowTable.keylen-1] = 0;
}
// generate the hash key for reverse record (bidir)
New_Hash_Key(bidirkeymem, flow_record, 1);
FlowTableRecord = hash_lookup_FlowTable(&bidir_index_cache, bidirkeymem, flow_record);
if ( FlowTableRecord ) {
// we found a corresponding flow - so update all fields in reverse direction
FlowTableRecord->counter[OUTBYTES] += flow_record->dOctets;
FlowTableRecord->counter[OUTPACKETS] += flow_record->dPkts;
FlowTableRecord->counter[INBYTES] += flow_record->out_bytes;
FlowTableRecord->counter[INPACKETS] += flow_record->out_pkts;
if ( TimeMsec_CMP(flow_record->first, flow_record->msec_first,
FlowTableRecord->flowrecord.first, FlowTableRecord->flowrecord.msec_first) == 2) {
FlowTableRecord->flowrecord.first = flow_record->first;
FlowTableRecord->flowrecord.msec_first = flow_record->msec_first;
}
if ( TimeMsec_CMP(flow_record->last, flow_record->msec_last,
FlowTableRecord->flowrecord.last, FlowTableRecord->flowrecord.msec_last) == 1) {
FlowTableRecord->flowrecord.last = flow_record->last;
FlowTableRecord->flowrecord.msec_last = flow_record->msec_last;
}
FlowTableRecord->counter[FLOWS] += flow_record->aggr_flows ? flow_record->aggr_flows : 1;
FlowTableRecord->flowrecord.tcp_flags |= flow_record->tcp_flags;
} else {
// no bidir flow found
// insert original flow into the cache
FlowTableRecord = hash_insert_FlowTable(index_cache, keymem, raw_record);
FlowTableRecord->counter[INBYTES] = flow_record->dOctets;
FlowTableRecord->counter[INPACKETS] = flow_record->dPkts;
FlowTableRecord->counter[OUTBYTES] = flow_record->out_bytes;
FlowTableRecord->counter[OUTPACKETS] = flow_record->out_pkts;
FlowTableRecord->counter[FLOWS] = flow_record->aggr_flows ? flow_record->aggr_flows : 1;
FlowTableRecord->map_info_ref = extension_info;
FlowTableRecord->exp_ref = flow_record->exp_ref;
keymem = NULL;
}
}
} // End of AddFlow
#undef get16bits
#if (defined(__GNUC__) && defined(__i386__)) || defined(__WATCOMC__) \
|| defined(_MSC_VER) || defined (__BORLANDC__) || defined (__TURBOC__)
#define get16bits(d) (*((const uint16_t *) (d)))
#endif
#if !defined (get16bits)
#define get16bits(d) ((((uint32_t)(((const uint8_t *)(d))[1])) << 8)\
+(uint32_t)(((const uint8_t *)(d))[0]) )
#endif
static inline uint32_t SuperFastHash (const char * data, int len) {
uint32_t hash = len, tmp;
int rem;
if (len <= 0 || data == NULL) return 0;
rem = len & 3;
len >>= 2;
/* Main loop */
for (;len > 0; len--) {
hash += get16bits (data);
tmp = (get16bits (data+2) << 11) ^ hash;
hash = (hash << 16) ^ tmp;
data += 2*sizeof (uint16_t);
hash += hash >> 11;
}
/* Handle end cases */
switch (rem) {
case 3: hash += get16bits (data);
hash ^= hash << 16;
hash ^= data[sizeof (uint16_t)] << 18;
hash += hash >> 11;
break;
case 2: hash += get16bits (data);
hash ^= hash << 11;
hash += hash >> 17;
break;
case 1: hash += *data;
hash ^= hash << 10;
hash += hash >> 1;
}
/* Force "avalanching" of final 127 bits */
hash ^= hash << 3;
hash += hash >> 5;
hash ^= hash << 4;
hash += hash >> 17;
hash ^= hash << 25;
hash += hash >> 6;
return hash;
}
int SetBidirAggregation(void) {
if ( aggregate_stack ) {
fprintf(stderr, "Can not set bidir mode while custom aggregation is set.\n");
return 0;
}
bidir_flows = 1;
return 1;
} // End of SetBidirAggregation
int ParseAggregateMask( char *arg, char **aggr_fmt ) {
char *p, *q;
uint64_t mask[2];
uint32_t subnet, stack_count;
int i, fmt_len, has_mask;
struct aggregate_info_s *a;
if ( bidir_flows ) {
fprintf(stderr, "Can not set custom aggregation while bidir mode is set.\n");
return 0;
}
stack_count = 0;
subnet = 0;
has_mask = 0;
aggregate_key_len = 0;
fmt_len = 0;
i = 0;
while ( aggregate_info[i].aggregate_token != NULL ) {
if ( aggregate_info[i].active )
stack_count++;
if ( aggregate_info[i].fmt )
fmt_len += ( strlen(aggregate_info[i].fmt) + 1 );
i++;
}
fmt_len++; // trailing '\0'
if ( !*aggr_fmt ) {
*aggr_fmt = malloc(fmt_len);
(*aggr_fmt)[0] = '\0';
}
if ( !*aggr_fmt ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
FlowTable.apply_netbits = 0;
FlowTable.has_masks = 0;
FlowTable.IPmask[0] = 0xffffffffffffffffLL;;
FlowTable.IPmask[1] = 0xffffffffffffffffLL;;
FlowTable.IPmask[2] = 0xffffffffffffffffLL;;
FlowTable.IPmask[3] = 0xffffffffffffffffLL;;
// separate tokens
p = strtok(arg, ",");
while ( p ) {
// check for subnet bits
q = strchr(p, '/');
if ( q ) {
char *n;
has_mask = 1;
*q = 0;
subnet = atoi(q+1);
// get IP version
n = &(p[strlen(p)-1]);
if ( *n == '4' ) {
// IPv4
if ( subnet < 1 || subnet > 32 ) {
fprintf(stderr, "Subnet specifier '%s' out of range for IPv4\n", q+1);
return 0;
}
mask[0] = 0xffffffffffffffffLL;
mask[1] = 0xffffffffffffffffLL << ( 32 - subnet );
} else if ( *n == '6' ) {
// IPv6
if ( subnet < 1 || subnet > 128 ) {
fprintf(stderr, "Subnet specifier '%s' out of range for IPv6\n", q+1);
return 0;
}
if ( subnet > 64 ) {
mask[0] = 0xffffffffffffffffLL;
mask[1] = 0xffffffffffffffffLL << ( 64 - subnet );
} else {
mask[0] = 0xffffffffffffffffLL << ( 64 - subnet );
mask[1] = 0;
}
} else {
// rubbish
*q = '/';
fprintf(stderr, "Need src4/dst4 src6/dst6 for IPv4 or IPv6 to aggregate with explicit netmask: '%s'\n", p);
return 0;
}
} else {
has_mask = 0;
}
a = aggregate_info;
while ( a->aggregate_token && (strcasecmp(p, a->aggregate_token ) != 0) )
a++;
if ( a->active ) {
fprintf(stderr, "Skip already given aggregation mask: %s\n", p);
} else if ( a->aggregate_token != NULL ) {
if ( a->fmt != NULL ) {
strncat(*aggr_fmt, a->fmt, fmt_len);
fmt_len -= strlen(a->fmt);
strncat(*aggr_fmt, " ", fmt_len);
fmt_len -= 1;
}
if ( strcasecmp(p, "srcnet" ) == 0 ) {
FlowTable.apply_netbits |= 1;
}
if ( strcasecmp(p, "dstnet" ) == 0 ) {
FlowTable.apply_netbits |= 2;
}
do {
int i = a->merge;
if ( i != -1 ) {
if ( has_mask ) {
a->param.mask = mask[i];
} else {
fprintf(stderr, "'%s' needs subnet bits too aggregate\n", p);
return 0;
}
} else {
if ( has_mask ) {
fprintf(stderr, "'%s' No subnet bits allowed here!\n", p);
return 0;
}
}
a->active = 1;
aggregate_key_len += a->param.size;
stack_count++;
a++;
} while (a->aggregate_token && (strcasecmp(p, a->aggregate_token ) == 0));
if ( has_mask ) {
FlowTable.has_masks = 1;
switch (p[0]) {
case 's':
FlowTable.IPmask[0] = mask[0];
FlowTable.IPmask[1] = mask[1];
break;
case 'd':
FlowTable.IPmask[2] = mask[0];
FlowTable.IPmask[3] = mask[1];
break;
}
}
} else {
fprintf(stderr, "Unknown aggregation specifier '%s'\n", p);
return 0;
}
p = strtok(NULL, ",");
}
if ( stack_count == 0 ) {
fprintf(stderr, "No aggregation specified!\n");
return 0;
}
aggregate_stack = (aggregate_param_t *)malloc((stack_count+1) * sizeof(aggregate_param_t));
stack_count = 0;
a = aggregate_info;
while ( a->aggregate_token ) {
if ( a->active ) {
aggregate_stack[stack_count++] = a->param;
dbg_printf("Set aggregate param: %s\n", a->aggregate_token);
}
a++;
}
// final '0' record
aggregate_stack[stack_count] = a->param;
dbg_printf("Aggregate key len: %i bytes\n", aggregate_key_len);
dbg_printf("Aggregate format string: '%s'\n", *aggr_fmt);
#ifdef DEVEL
if ( aggregate_stack ) {
aggregate_param_t *aggr_param = aggregate_stack;
printf("Aggregate stack:\n");
while ( aggr_param->size ) {
printf("Offset: %u, Mask: %llx, Shift: %llu\n", aggr_param->offset,
(long long unsigned)aggr_param->mask, (long long unsigned)aggr_param->shift);
aggr_param++;
} // while
}
printf("Has IP mask: %i %i\n", has_mask, FlowTable.has_masks);
printf("Mask 0: 0x%llx\n", (unsigned long long)FlowTable.IPmask[0]);
printf("Mask 1: 0x%llx\n", (unsigned long long)FlowTable.IPmask[1]);
printf("Mask 2: 0x%llx\n", (unsigned long long)FlowTable.IPmask[2]);
printf("Mask 3: 0x%llx\n", (unsigned long long)FlowTable.IPmask[3]);
#endif
return 1;
} // End of ParseAggregateMask
master_record_t *GetMasterAggregateMask(void) {
master_record_t *aggr_record_mask;
if ( aggregate_stack ) {
uint64_t *r;
aggregate_param_t *aggr_param = aggregate_stack;
aggr_record_mask = (master_record_t *)malloc(sizeof(master_record_t));
if ( !aggr_record_mask ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror (errno));
return 0;
}
r = (uint64_t *)aggr_record_mask;
memset((void *)aggr_record_mask, 0, sizeof(master_record_t));
while ( aggr_param->size ) {
int offset = aggr_param->offset;
r[offset] |= aggr_param->mask;
aggr_param++;
}
// not really needed, but preset it anyway
r[0] = 0xffffffffffffffffLL;
r[1] = 0xffffffffffffffffLL;
aggr_record_mask->dPkts = 0xffffffffffffffffLL;
aggr_record_mask->dOctets = 0xffffffffffffffffLL;
aggr_record_mask->out_pkts = 0xffffffffffffffffLL;
aggr_record_mask->out_bytes = 0xffffffffffffffffLL;
aggr_record_mask->aggr_flows = 0xffffffffffffffffLL;
aggr_record_mask->last = 0xffffffff;
return aggr_record_mask;
} else {
return NULL;
}
} // End of GetMasterAggregateMask
static inline void New_Hash_Key(void *keymem, master_record_t *flow_record, int swap_flow) {
uint64_t *record = (uint64_t *)flow_record;
Default_key_t *keyptr;
// apply src/dst mask bits if requested
if ( FlowTable.apply_netbits ) {
ApplyNetMaskBits(flow_record, FlowTable.apply_netbits);
}
if ( aggregate_stack ) {
// custom user aggregation
aggregate_param_t *aggr_param = aggregate_stack;
while ( aggr_param->size ) {
uint64_t val = (record[aggr_param->offset] & aggr_param->mask) >> aggr_param->shift;
switch ( aggr_param->size ) {
case 8: {
uint64_t *_v = (uint64_t *)keymem;
*_v = val;
keymem += sizeof(uint64_t);
} break;
case 4: {
uint32_t *_v = (uint32_t *)keymem;
*_v = val;
keymem += sizeof(uint32_t);
} break;
case 2: {
uint16_t *_v = (uint16_t *)keymem;
*_v = val;
keymem += sizeof(uint16_t);
} break;
case 1: {
uint8_t *_v = (uint8_t *)keymem;
*_v = val;
keymem += sizeof(uint8_t);
} break;
default:
fprintf(stderr, "Panic: Software error in %s line %d\n", __FILE__, __LINE__);
exit(255);
} // switch
aggr_param++;
} // while
} else if ( swap_flow ) {
// default 5-tuple aggregation for bidirectional flows
keyptr = (Default_key_t *)keymem;
keyptr->srcaddr[0] = flow_record->v6.dstaddr[0];
keyptr->srcaddr[1] = flow_record->v6.dstaddr[1];
keyptr->dstaddr[0] = flow_record->v6.srcaddr[0];
keyptr->dstaddr[1] = flow_record->v6.srcaddr[1];
keyptr->srcport = flow_record->dstport;
keyptr->dstport = flow_record->srcport;
keyptr->proto = flow_record->prot;
} else {
// default 5-tuple aggregation
keyptr = (Default_key_t *)keymem;
keyptr->srcaddr[0] = flow_record->v6.srcaddr[0];
keyptr->srcaddr[1] = flow_record->v6.srcaddr[1];
keyptr->dstaddr[0] = flow_record->v6.dstaddr[0];
keyptr->dstaddr[1] = flow_record->v6.dstaddr[1];
keyptr->srcport = flow_record->srcport;
keyptr->dstport = flow_record->dstport;
keyptr->proto = flow_record->prot;
}
} // End of New_Hash_Key

149
bin/nflowcache.h Executable file
View File

@ -0,0 +1,149 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the auhor nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nflowcache.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFLOWCACHE_H
#define _NFLOWCACHE_H 1
/* Definitions */
/*
* Flow Table
* In order to aggregate flows or to generate any flow statistics, the flows passed the filter
* are stored into an internal hash table.
*/
/* Element of the Flow Table ( cache ) */
typedef struct FlowTableRecord {
// record chain - points to next record with same hash in case of a hash collision
struct FlowTableRecord *next;
// Hash papameters
uint32_t hash; // the full 32bit hash value
char *hash_key; // all keys in sequence to generate the hash
// flow counter parameters for FLOWS, INPACKETS, INBYTES, OUTPACKETS, OUTBYTES
uint64_t counter[5];
extension_info_t *map_info_ref;
exporter_info_record_t *exp_ref;
// flow record follows
// flow data size may vary depending on the number of extensions
// common_record_t already contains a pointer to more data ( extensions ) at the end
common_record_t flowrecord;
// no further vars beyond this point! The flow record above has additional data.
} FlowTableRecord_t;
typedef struct MemoryHandle_s {
/*
* to speedup aggregation/record statistics, the internal hash tables use their own memory management.
* memory is allocated with malloc in chunks of MemBlockSize. All memory blocks are kept in the
* memblock array. Memory blocks are allocated on request up to the number of MaxMemBlocks. If more
* blocks are requested, the memblock array is automatically extended.
* Memory allocated from a memblock is aligned accoording ALIGN
*/
uint32_t BlockSize; /* max size of each pre-allocated memblock */
/* memory blocks - containing the flow records and keys */
void **memblock; /* array holding all NumBlocks allocated memory blocks */
uint32_t MaxBlocks; /* Size of memblock array */
uint32_t NumBlocks; /* number of allocated flow blocks in memblock array */
uint32_t CurrentBlock; /* Index of current memblock to allocate memory from */
uint32_t Allocted; /* Number of bytes already allocated in memblock */
} MemoryHandle_t;
#ifdef __x86_64
# define ALIGN_MASK 0xFFFFFFF8
#else
# define ALIGN_MASK 0xFFFFFFFC
#endif
// number of bits for hash width for floe table
// Size: 0 < HashBits < 32
// typically 20 - tradeoff memory/speed
#define HashBits 20
// Each pre-allocated memory block is 10M
#define MemBlockSize 10*1024*1024
#define MaxMemBlocks 256
typedef struct hash_FlowTable {
/* hash table data */
uint16_t NumBits; /* width of the hash table */
uint32_t IndexMask; /* Mask which corresponds to NumBits */
uint32_t NumRecords; /* number of records in table */
FlowTableRecord_t **bucket; /* Hash entry point: points to elements in the flow block */
FlowTableRecord_t **bucketcache; /* in case of index collisions, this array points to the last element with that index */
uint32_t keylen; /* key length of hash key as number of 4byte ints */
uint32_t keysize; /* size of key in bytes */
/* use a MemoryHandle for the table */
MemoryHandle_t mem;
/* src/dst IP aggr masks - use to properly maks the IP before printing */
uint64_t IPmask[4]; // 0-1 srcIP, 2-3 dstIP
int has_masks;
int apply_netbits; // bit 0: src, bit 1: dst
} hash_FlowTable;
hash_FlowTable *GetFlowTable(void);
int Init_FlowTable(void);
void Dispose_FlowTable(void);
char *VerifyStat(uint16_t Aggregate_Bits);
int SetStat(char *str, int *element_stat, int *flow_stat);
void InsertFlow(common_record_t *raw_record, master_record_t *flow_record, extension_info_t *extension_info);
void AddFlow(common_record_t *raw_record, master_record_t *flow_record, extension_info_t *extension_info );
int SetBidirAggregation( void );
int ParseAggregateMask( char *arg, char **aggr_fmt );
master_record_t *GetMasterAggregateMask(void);
#endif //_NFLOWCACHE_H

555
bin/nfnet.c Normal file
View File

@ -0,0 +1,555 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfnet.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*
*/
#include "config.h"
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#include <time.h>
#include <netinet/in.h>
#include <string.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "util.h"
#include "nfnet.h"
/* at least this number of byytes required, if we change the socket buffer */
#define Min_SOCKBUFF_LEN 65536
const int LISTEN_QUEUE = 128;
/* local function prototypes */
static int isMulticast(struct sockaddr_storage *addr);
static int joinGroup(int sockfd, int loopBack, int mcastTTL, struct sockaddr_storage *addr);
/* function definitions */
int Unicast_receive_socket(const char *bindhost, const char *listenport, int family, int sockbuflen ) {
struct addrinfo hints, *res, *ressave;
socklen_t optlen;
int error, p, sockfd;
if ( !listenport ) {
fprintf(stderr, "listen port required!\n");
LogError("listen port required!");
return -1;
}
// if nothing specified on command line, prefer IPv4 over IPv6, for compatibility
if ( bindhost == NULL && family == AF_UNSPEC )
family = AF_INET;
memset(&hints, 0, sizeof(struct addrinfo));
/*
AI_PASSIVE flag: we use the resulting address to bind
to a socket for accepting incoming connections.
So, when the hostname==NULL, getaddrinfo function will
return one entry per allowed protocol family containing
the unspecified address for that family.
*/
hints.ai_flags = AI_PASSIVE;
hints.ai_family = family;
hints.ai_socktype = SOCK_DGRAM;
error = getaddrinfo(bindhost, listenport, &hints, &res);
if (error) {
fprintf(stderr, "getaddrinfo error: [%s]\n", gai_strerror(error));
LogError("getaddrinfo error: [%s]", gai_strerror(error));
return -1;
}
/*
Try open socket with each address getaddrinfo returned,
until we get a valid listening socket.
*/
ressave = res;
sockfd = -1;
while ( res ) {
// we listen only on IPv4 or IPv6
if ( res->ai_family != AF_INET && res->ai_family != AF_INET6 )
continue;
sockfd = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if ( !( sockfd < 0 ) ) {
// socket call was successfull
if (bind(sockfd, res->ai_addr, res->ai_addrlen) == 0) {
if ( res->ai_family == AF_INET )
LogInfo("Bound to IPv4 host/IP: %s, Port: %s",
bindhost == NULL ? "any" : bindhost, listenport);
if ( res->ai_family == AF_INET6 )
LogInfo("Bound to IPv6 host/IP: %s, Port: %s",
bindhost == NULL ? "any" : bindhost, listenport);
// we are done
break;
}
// bind was unsuccessful :(
close(sockfd);
sockfd = -1;
}
res = res->ai_next;
}
if (sockfd < 0) {
freeaddrinfo(ressave);
fprintf(stderr, "Could not open the requested socket: %s\n", strerror (errno));
LogError("Receive socket error: could not open the requested socket", strerror (errno));
return -1;
}
listen(sockfd, LISTEN_QUEUE);
freeaddrinfo(ressave);
if ( sockbuflen ) {
if ( sockbuflen < Min_SOCKBUFF_LEN ) {
sockbuflen = Min_SOCKBUFF_LEN;
LogInfo("I want at least %i bytes as socket buffer", sockbuflen);
}
optlen = sizeof(p);
getsockopt(sockfd,SOL_SOCKET,SO_RCVBUF,&p,&optlen);
LogInfo("Standard setsockopt, SO_RCVBUF is %i Requested length is %i bytes",p, sockbuflen);
if ((setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &sockbuflen, sizeof(sockbuflen)) != 0) ) {
fprintf (stderr, "setsockopt(SO_RCVBUF,%d): %s\n", sockbuflen, strerror (errno));
LogError("setsockopt(SO_RCVBUF,%d): %s", sockbuflen, strerror (errno));
close(sockfd);
return -1;
} else {
getsockopt(sockfd,SOL_SOCKET,SO_RCVBUF,&p,&optlen);
LogInfo("System set setsockopt, SO_RCVBUF to %d bytes", p);
}
}
return sockfd;
} /* End of Unicast_receive_socket */
int Unicast_send_socket (const char *hostname, const char *sendport, int family,
unsigned int wmem_size, struct sockaddr_storage *addr, int *addrlen) {
struct addrinfo hints, *res, *ressave;
int n, sockfd;
unsigned int wmem_actual;
socklen_t optlen;
if ( !hostname || !sendport ) {
fprintf(stderr, "hostname and listen port required!\n");
LogError("hostname and listen port required!");
return -1;
}
// create socket
memset(&hints, 0, sizeof(struct addrinfo));
hints.ai_family = family;
hints.ai_socktype = SOCK_DGRAM;
n = getaddrinfo(hostname, sendport, &hints, &res);
if ( n < 0 ) {
fprintf(stderr, "getaddrinfo error: [%s]\n", strerror(errno));
return -1;
}
ressave = res;
sockfd = -1;
while (res) {
sockfd = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if ( !(sockfd < 0) ) {
// socket call was successsful
if (connect(sockfd, res->ai_addr, res->ai_addrlen) == 0) {
// connect successful - we are done
close(sockfd);
// ok - we need now an unconnected socket
sockfd = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
break;
}
// unsuccessful connect :(
close(sockfd);
sockfd = -1;
}
res=res->ai_next;
}
if (sockfd < 0) {
freeaddrinfo(ressave);
fprintf(stderr, "Send socket error: could not open the requested socket: %s\n", strerror (errno));
LogError("Send socket error: could not open the requested socket: %s", strerror (errno));
return -1;
}
*addrlen = res->ai_addrlen;
memcpy(addr, res->ai_addr, res->ai_addrlen);
freeaddrinfo(ressave);
// Set socket write buffer. Need to be root!
if ( wmem_size > 0 ) {
if ( geteuid() == 0 ) {
setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, &wmem_size, sizeof(wmem_size));
// check what was set (e.g. linux 2.4.20 sets twice of what was requested)
getsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, &wmem_actual, &optlen);
if (wmem_size != wmem_actual) {
printf("Warning: Socket write buffer size requested: %u set: %u\n",
wmem_size, wmem_actual);
}
} else {
printf("Warning: Socket buffer size can only be changed by root!\n");
}
}
return sockfd;
} // End of Unicast_send_socket
int Multicast_receive_socket (const char *hostname, const char *listenport, int family, int sockbuflen ) {
struct addrinfo hints, *res, *ressave;
socklen_t optlen;
int p, error, sockfd;
if ( !listenport ) {
fprintf(stderr, "listen port required!\n");
LogError("listen port required!");
return -1;
}
memset(&hints, 0, sizeof(struct addrinfo));
hints.ai_family = family;
hints.ai_socktype = SOCK_DGRAM;
error = getaddrinfo(hostname, listenport, &hints, &res);
if ( error ) {
fprintf(stderr, "getaddrinfo error:: [%s]\n", gai_strerror(error));
LogError("getaddrinfo error:: [%s]", gai_strerror(error));
return -1;
}
/*
Try open socket with each address getaddrinfo returned,
until we get a valid listening socket.
*/
sockfd = -1;
ressave = res;
while (res) {
sockfd = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if ( sockfd < 0 ) {
res=res->ai_next;
continue;
}
// we found a valid socket and are done in this loop
break;
}
if ( sockfd < 0 ) {
// nothing found - bye bye
fprintf(stderr, "Could not create a socket for [%s:%s]\n", hostname, listenport);
LogError("Could not create a socket for [%s:%s]", hostname, listenport);
freeaddrinfo(ressave);
return -1;
}
if ( isMulticast((struct sockaddr_storage *)res->ai_addr) < 0 ) {
fprintf(stderr, "Not a multicast address [%s]\n", hostname);
LogError("Not a multicast address [%s]", hostname);
freeaddrinfo(ressave);
return -1;
}
close(sockfd);
sockfd = socket(res->ai_family, SOCK_DGRAM, 0);
if (bind(sockfd, res->ai_addr, res->ai_addrlen) < 0) {
fprintf(stderr, "bind: %s\n", strerror (errno));
LogError("bind: %s", strerror (errno));
close(sockfd);
freeaddrinfo(ressave);
return -1;
}
if (joinGroup(sockfd, 1 , 1, (struct sockaddr_storage *)res->ai_addr) <0) {
close(sockfd);
freeaddrinfo(ressave);
return -1;
}
if ( res->ai_family == AF_INET )
LogInfo("Joined IPv4 multicast group: %s Port: %s", hostname, listenport);
if ( res->ai_family == AF_INET6 )
LogInfo( "Joined IPv6 multicat group: %s Port: %s", hostname, listenport);
freeaddrinfo(ressave);
if ( sockbuflen ) {
if ( sockbuflen < Min_SOCKBUFF_LEN ) {
sockbuflen = Min_SOCKBUFF_LEN;
LogInfo("I want at least %i bytes as socket buffer", sockbuflen);
}
optlen = sizeof(p);
getsockopt(sockfd,SOL_SOCKET,SO_RCVBUF,&p,&optlen);
LogInfo("Standard setsockopt, SO_RCVBUF is %i Requested length is %i bytes",p, sockbuflen);
if ((setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &sockbuflen, sizeof(sockbuflen)) != 0) ) {
fprintf (stderr, "setsockopt(SO_RCVBUF,%d): %s\n", sockbuflen, strerror (errno));
LogError("setsockopt(SO_RCVBUF,%d): %s", sockbuflen, strerror (errno));
close(sockfd);
return -1;
} else {
getsockopt(sockfd,SOL_SOCKET,SO_RCVBUF,&p,&optlen);
LogInfo("System set setsockopt, SO_RCVBUF to %d bytes", p);
}
}
return sockfd;
} /* End of Multicast_receive_socket */
int Multicast_send_socket (const char *hostname, const char *listenport, int family,
unsigned int wmem_size, struct sockaddr_storage *addr, int *addrlen) {
struct addrinfo hints, *res, *ressave;
int error, sockfd;
if ( !listenport || !hostname ) {
fprintf(stderr, "hostname and listen port required!\n");
LogError("hostname and listen port required!");
return -1;
}
memset(&hints, 0, sizeof(struct addrinfo));
hints.ai_family = family;
hints.ai_socktype = SOCK_DGRAM;
error = getaddrinfo(hostname, listenport, &hints, &res);
if ( error ) {
fprintf(stderr, "getaddrinfo error:: [%s]\n", gai_strerror(error));
LogError("getaddrinfo error:: [%s]", gai_strerror(error));
return -1;
}
/*
Try open socket with each address getaddrinfo returned,
until we get a valid listening socket.
*/
sockfd=-1;
ressave = res;
while (res) {
sockfd = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if ( sockfd < 0 ) {
res=res->ai_next;
continue;
}
// we found a valid socket and are done in this loop
break;
}
if ( sockfd < 0 ) {
// nothing found - bye bye
fprintf(stderr, "Could not create a socket for [%s:%s]\n", hostname, listenport);
LogError("Could not create a socket for [%s:%s]", hostname, listenport);
freeaddrinfo(ressave);
return -1;
}
if ( isMulticast((struct sockaddr_storage *)res->ai_addr) < 0 ) {
fprintf(stderr, "Not a multicast address [%s]\n", hostname);
LogError("Not a multicast address [%s]", hostname);
freeaddrinfo(ressave);
return -1;
}
close(sockfd);
sockfd = socket(res->ai_family, SOCK_DGRAM, 0);
*addrlen = res->ai_addrlen;
memcpy(addr, res->ai_addr, res->ai_addrlen);
/*
((struct sockaddr_in *)addr)->sin_family=AF_INET;
addr.sin_addr.s_addr=inet_addr(HELLO_GROUP);
addr.sin_port=htons(HELLO_PORT);
*/
if (joinGroup(sockfd, 1 , 1, (struct sockaddr_storage *)res->ai_addr) <0) {
close(sockfd);
freeaddrinfo(ressave);
return -1;
}
freeaddrinfo(ressave);
/*
message = "hello world\n";
printf("Send %lu bytes, Message: %s\n", strlen(message)+1, message);
while (1) {
ret = sendto(sockfd,message,strlen(message)+1,0,(struct sockaddr *) addr, *addrlen);
if ( ret != strlen(message)+1 ) {
perror("sendto");
exit(1);
}
printf("sleep\n");
sleep(1);
}
exit(255);
*/
return sockfd;
} /* End of Multicast_send_socket */
static int joinGroup(int sockfd, int loopBack, int mcastTTL, struct sockaddr_storage *addr) {
int ret, err;
ret=-1;
switch (addr->ss_family) {
case AF_INET: {
struct ip_mreq mreq;
mreq.imr_multiaddr.s_addr = ((struct sockaddr_in *)addr)->sin_addr.s_addr;
mreq.imr_interface.s_addr = INADDR_ANY;
err = setsockopt(sockfd, IPPROTO_IP, IP_ADD_MEMBERSHIP, (const void *)&mreq, sizeof(mreq));
if ( err ) {
fprintf(stderr, "setsockopt IP_ADD_MEMBERSHIP: %s\n", strerror (errno));
LogError("setsockopt IP_ADD_MEMBERSHIP: %s", strerror (errno));
break;
}
ret = 0;
/*
// These two setsockopt may fail because not implemented on every OS, but harmless
err = setsockopt(sockfd, IPPROTO_IP, IP_MULTICAST_LOOP, &loopBack, sizeof(loopBack));
if ( err ) {
fprintf(stderr, "setsockopt IP_MULTICAST_LOOP: %s\n", strerror (errno));
LogError("setsockopt IP_MULTICAST_LOOP: %s", strerror (errno));
}
err = setsockopt(sockfd, IPPROTO_IP, IP_MULTICAST_TTL, &mcastTTL, sizeof(mcastTTL));
if ( err ) {
fprintf(stderr, "setsockopt IP_MULTICAST_LOOP: %s\n", strerror (errno));
LogError("setsockopt IP_MULTICAST_LOOP: %s", strerror (errno));
}
*/
} break;
case AF_INET6: {
struct ipv6_mreq mreq6;
memcpy(&mreq6.ipv6mr_multiaddr, &(((struct sockaddr_in6 *)addr)->sin6_addr), sizeof(struct in6_addr));
mreq6.ipv6mr_interface= 0;
err = setsockopt(sockfd, IPPROTO_IPV6, IPV6_JOIN_GROUP, &mreq6, sizeof(mreq6));
if ( err ) {
fprintf(stderr, "setsockopt IPV6_JOIN_GROUP: %s\n", strerror (errno));
LogError("setsockopt IPV6_JOIN_GROUP: %s", strerror (errno));
break;
}
ret = 0;
/*
// These two setsockopt may fail because not implemented on every OS, but harmless
err = setsockopt(sockfd, IPPROTO_IPV6, IPV6_MULTICAST_LOOP, &loopBack, sizeof(loopBack));
if ( err ) {
fprintf(stderr, "setsockopt IPV6_MULTICAST_LOOP: %s\n", strerror (errno));
LogError("setsockopt IPV6_MULTICAST_LOOP: %s", strerror (errno));
}
err = setsockopt(sockfd, IPPROTO_IPV6, IPV6_MULTICAST_HOPS, &mcastTTL, sizeof(mcastTTL));
if ( err ) {
fprintf(stderr, "setsockopt IPV6_MULTICAST_HOPS: %s\n", strerror (errno));
LogError("setsockopt IPV6_MULTICAST_HOPS: %s", strerror (errno));
}
*/
} break;
default:
;
}
return ret;
} /* joinGroup */
static int isMulticast(struct sockaddr_storage *addr) {
int ret;
ret = -1;
switch (addr->ss_family) {
case AF_INET: {
struct sockaddr_in *addr4=(struct sockaddr_in *)addr;
ret = IN_MULTICAST(ntohl(addr4->sin_addr.s_addr));
}
break;
case AF_INET6: {
struct sockaddr_in6 *addr6=(struct sockaddr_in6 *)addr;
ret = IN6_IS_ADDR_MULTICAST(&addr6->sin6_addr);
}
break;
default:
;
}
return ret;
} /* End of isMulticast */

72
bin/nfnet.h Normal file
View File

@ -0,0 +1,72 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfnet.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFNET_H
#define _NFNET_H 1
/* Definitions */
#define UDP_PACKET_SIZE 1472
typedef struct send_peer_s {
struct sockaddr_storage addr;
int addrlen;
int sockfd;
int family;
char *port;
char *hostname;
int mcast;
int flush;
void *send_buffer;
void *buff_ptr;
void *endp;
} send_peer_t;
/* Function prototypes */
int Unicast_receive_socket(const char *bindhost, const char *listenport, int family, int sockbuflen );
int Multicast_receive_socket (const char *hostname, const char *listenport, int family, int sockbuflen);
int Unicast_send_socket (const char *hostname, const char *listenport, int family,
unsigned int wmem_size, struct sockaddr_storage *addr, int *addrlen);
int Multicast_send_socket (const char *hostname, const char *listenport, int family,
unsigned int wmem_size, struct sockaddr_storage *addr, int *addrlen);
#endif //_NFNET_H

1566
bin/nfpcapd.c Normal file
View File

File diff suppressed because it is too large Load Diff

134
bin/nfprof.c Normal file
View File

@ -0,0 +1,134 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfprof.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <strings.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include <sys/time.h>
#include <sys/resource.h>
#include "nfprof.h"
/*
* Initialize profiling.
*
*/
int nfprof_start(nfprof_t *profile_data) {
bzero (profile_data, sizeof(nfprof_t));
return gettimeofday(&profile_data->tstart, (struct timezone*)NULL) == 0 ? 1 : 0;
} // End of nfprof_start
/*
* Finish profiling.
*
*/
int nfprof_end(nfprof_t *profile_data, uint64_t numflows) {
int ret;
if ((ret = gettimeofday(&profile_data->tend, (struct timezone*)NULL)) == -1)
return 1;
if ((ret = getrusage(RUSAGE_SELF, &profile_data->used)) == -1)
return 1;
profile_data->numflows = numflows;
return 0;
} // End of nfprof_end
/*
* Dump nfprof contents to std
*
*/
void nfprof_print(nfprof_t *profile_data, FILE *std) {
u_long usec, sec;
double fps, allsecs;
usec = profile_data->used.ru_utime.tv_usec + profile_data->used.ru_stime.tv_usec;
sec = profile_data->used.ru_utime.tv_sec + profile_data->used.ru_stime.tv_sec;
if (usec > 1000000)
usec -= 1000000, ++sec;
allsecs = (double)sec + ((double)usec/1000000);
if ( allsecs == 0.0 )
fps = 0;
else
fps = (double)profile_data->numflows / ((double)sec + ((double)usec/1000000));
fprintf(std, "Sys: %lu.%-3.3lus flows/second: %-10.1f ", sec, usec/1000, fps);
if (profile_data->tend.tv_usec < profile_data->tstart.tv_usec)
profile_data->tend.tv_usec += 1000000, --profile_data->tend.tv_sec;
usec = profile_data->tend.tv_usec - profile_data->tstart.tv_usec;
sec = profile_data->tend.tv_sec - profile_data->tstart.tv_sec;
if ( usec == 0 && sec == 0 )
// acctually should never happen, but catch it anyway
fps = 0;
else
fps = (double)profile_data->numflows / ((double)sec + ((double)usec/1000000));
fprintf(std, "Wall: %lu.%-3.3lus flows/second: %-10.1f\n", sec, usec/1000, fps);
/*
fprintf(std, "\n");
fprintf(std, "integral max resident set size: %u\n", profile_data->used.ru_maxrss);
fprintf(std, "integral shared text memory size: %u\n", profile_data->used.ru_ixrss);
fprintf(std, "integral unshared data size: %u\n", profile_data->used.ru_idrss);
fprintf(std, "integral unshared stack size: %u\n", profile_data->used.ru_isrss);
fprintf(std, "page reclaims: %u\n", profile_data->used.ru_minflt);
fprintf(std, "page faults: %u\n", profile_data->used.ru_majflt);
fprintf(std, "swaps: %u\n", profile_data->used.ru_nswap);
fprintf(std, "block input operations: %u\n", profile_data->used.ru_inblock);
fprintf(std, "block output operations: %u\n", profile_data->used.ru_oublock);
*/
} // End of nfprof_print

55
bin/nfprof.h Normal file
View File

@ -0,0 +1,55 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfprof.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFPROF_H
#define _NFPROF_H 1
typedef struct nfprof_s {
struct timeval tstart; /* start time */
struct timeval tend; /* end time */
struct rusage used; /* system resources used */
uint64_t numflows; /* total # of flows processed */
} nfprof_t;
int nfprof_start(nfprof_t *profile_data);
int nfprof_end(nfprof_t *profile_data, uint64_t numflows);
void nfprof_print(nfprof_t *profile_data, FILE *std);
#endif //_NFPROF_H

694
bin/nfprofile.c Normal file
View File

@ -0,0 +1,694 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfprofile.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdarg.h>
#include <errno.h>
#include <time.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/param.h>
#include <netinet/in.h>
#include <fcntl.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nf_common.h"
#include "rbtree.h"
#include "nftree.h"
#include "nfdump.h"
#include "nffile.h"
#include "nfx.h"
#include "nfstat.h"
#include "nfstatfile.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "ipconv.h"
#include "flist.h"
#include "util.h"
#include "profile.h"
/* externals */
extern generic_exporter_t **exporter_list;
#ifdef COMPAT15
extern extension_descriptor_t extension_descriptor[];
#endif
/* Local Variables */
static const char *nfdump_version = VERSION;
extension_map_list_t *extension_map_list;
uint32_t is_anonymized;
char Ident[IDENTLEN];
/* Function Prototypes */
static void usage(char *name);
static profile_param_info_t *ParseParams (char *profile_datadir);
static void process_data(profile_channel_info_t *channels, unsigned int num_channels, time_t tslot, int do_xstat);
/* Functions */
#include "nfdump_inline.c"
#include "nffile_inline.c"
static void usage(char *name) {
printf("usage %s [options] \n"
"-h\t\tthis text you see right here\n"
"-V\t\tPrint version and exit.\n"
"-D <dns>\tUse nameserver <dns> for host lookup.\n"
"-H Add xstat histogram data to flow file.(default 'no')\n"
"-M <expr>\tRead input from multiple directories.\n"
"-r\t\tread input from file\n"
"-f\t\tfilename with filter syntaxfile\n"
"-p\t\tprofile data dir.\n"
"-P\t\tprofile stat dir.\n"
"-s\t\tprofile subdir.\n"
"-Z\t\tCheck filter syntax and exit.\n"
"-S subdir\tSub directory format. see nfcapd(1) for format\n"
"-z\t\tCompress flows in output file.\n"
"-t <time>\ttime for RRD update\n", name);
} /* usage */
static void process_data(profile_channel_info_t *channels, unsigned int num_channels, time_t tslot, int do_xstat) {
common_record_t *flow_record;
nffile_t *nffile;
FilterEngine_data_t *engine;
int i, j, done, ret ;
#ifdef COMPAT15
int v1_map_done = 0;
#endif
nffile = GetNextFile(NULL, 0, 0);
if ( !nffile ) {
LogError("GetNextFile() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
if ( nffile == EMPTY_LIST ) {
LogError("Empty file list. No files to process\n");
return;
}
// store infos away for later use
// although multiple files may be processed, it is assumed that all
// have the same settings
is_anonymized = IP_ANONYMIZED(nffile);
strncpy(Ident, nffile->file_header->ident, IDENTLEN);
Ident[IDENTLEN-1] = '\0';
done = 0;
while ( !done ) {
// get next data block from file
ret = ReadBlock(nffile);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
LogError("Skip corrupt data file '%s'\n",GetCurrentFilename());
else
LogError("Read error in file '%s': %s\n",GetCurrentFilename(), strerror(errno) );
// fall through - get next file in chain
case NF_EOF: {
nffile_t *next = GetNextFile(nffile, 0, 0);
if ( next == EMPTY_LIST ) {
done = 1;
}
if ( next == NULL ) {
done = 1;
LogError("Unexpected end of file list\n");
}
continue;
} break; // not really needed
}
#ifdef COMPAT15
if ( nffile->block_header->id == DATA_BLOCK_TYPE_1 ) {
common_record_v1_t *v1_record = (common_record_v1_t *)nffile->buff_ptr;
// create an extension map for v1 blocks
if ( v1_map_done == 0 ) {
extension_map_t *map = malloc(sizeof(extension_map_t) + 2 * sizeof(uint16_t) );
if ( ! map ) {
LogError("malloc() allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
map->type = ExtensionMapType;
map->size = sizeof(extension_map_t) + 2 * sizeof(uint16_t);
if (( map->size & 0x3 ) != 0 ) {
map->size += 4 - ( map->size & 0x3 );
}
map->map_id = INIT_ID;
map->ex_id[0] = EX_IO_SNMP_2;
map->ex_id[1] = EX_AS_2;
map->ex_id[2] = 0;
map->extension_size = 0;
map->extension_size += extension_descriptor[EX_IO_SNMP_2].size;
map->extension_size += extension_descriptor[EX_AS_2].size;
if ( Insert_Extension_Map(extension_map_list, map) ) {
int j;
for ( j=0; j < num_channels; j++ ) {
if ( channels[j].nffile != NULL) {
// flush new map
AppendToBuffer(channels[j].nffile, (void *)map, map->size);
}
}
} // else map already known and flushed
v1_map_done = 1;
}
// convert the records to v2
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
common_record_t *v2_record = (common_record_t *)v1_record;
Convert_v1_to_v2((void *)v1_record);
// now we have a v2 record -> use size of v2_record->size
v1_record = (common_record_v1_t *)((pointer_addr_t)v1_record + v2_record->size);
}
nffile->block_header->id = DATA_BLOCK_TYPE_2;
}
#endif
if ( nffile->block_header->id == Large_BLOCK_Type ) {
// skip
continue;
}
if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) {
LogError("Can't process block type %u. Skip block.\n", nffile->block_header->id);
continue;
}
flow_record = nffile->buff_ptr;
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
switch ( flow_record->type ) {
case CommonRecordV0Type:
case CommonRecordType: {
generic_exporter_t *exp_info = exporter_list[flow_record->exporter_sysid];
uint32_t map_id = flow_record->ext_map;
master_record_t *master_record;
if ( extension_map_list->slot[map_id] == NULL ) {
LogError("Corrupt data file. Missing extension map %u. Skip record.\n", flow_record->ext_map);
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
continue;
}
master_record = &(extension_map_list->slot[map_id]->master_record);
ExpandRecord_v2( flow_record, extension_map_list->slot[flow_record->ext_map],
exp_info ? &(exp_info->info) : NULL, master_record);
for ( j=0; j < num_channels; j++ ) {
int match;
// apply profile filter
(channels[j].engine)->nfrecord = (uint64_t *)master_record;
engine = channels[j].engine;
match = (*engine->FilterEngine)(engine);
// if profile filter failed -> next profile
if ( !match )
continue;
// filter was successful -> continue record processing
// update statistics
UpdateStat(&channels[j].stat_record, master_record);
if ( channels[j].nffile )
UpdateStat(channels[j].nffile->stat_record, master_record);
if ( channels[j].xstat )
UpdateXStat(channels[j].xstat, master_record);
// do we need to write data to new file - shadow profiles do not have files.
// check if we need to flush the output buffer
if ( channels[j].nffile != NULL ) {
// write record to output buffer
AppendToBuffer(channels[j].nffile, (void *)flow_record, flow_record->size);
}
} // End of for all channels
} break;
case ExtensionMapType: {
extension_map_t *map = (extension_map_t *)flow_record;
if ( Insert_Extension_Map(extension_map_list, map) ) {
int j;
for ( j=0; j < num_channels; j++ ) {
if ( channels[j].nffile != NULL ) {
// flush new map
AppendToBuffer(channels[j].nffile, (void *)map, map->size);
}
}
} // else map already known and flushed
} break;
case ExporterInfoRecordType: {
int ret = AddExporterInfo((exporter_info_record_t *)flow_record);
if ( ret != 0 ) {
int j;
for ( j=0; j < num_channels; j++ ) {
if ( channels[j].nffile != NULL && ret == 1) {
// flush new exporter
AppendToBuffer(channels[j].nffile, (void *)flow_record, flow_record->size);
}
}
} else {
LogError("Failed to add Exporter Record\n");
}
} break;
case SamplerInfoRecordype: {
int ret = AddSamplerInfo((sampler_info_record_t *)flow_record);
if ( ret != 0 ) {
int j;
for ( j=0; j < num_channels; j++ ) {
if ( channels[j].nffile != NULL && ret == 1 ) {
// flush new map
AppendToBuffer(channels[j].nffile, (void *)flow_record, flow_record->size);
}
}
} else {
LogError("Failed to add Sampler Record\n");
}
} break;
case ExporterRecordType:
case SamplerRecordype:
case ExporterStatRecordType:
// Silently skip exporter records
break;
default: {
LogError("Skip unknown record type %i\n", flow_record->type);
}
}
// Advance pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
} // End of for all umRecords
} // End of while !done
// do we need to write data to new file - shadow profiles do not have files.
for ( j=0; j < num_channels; j++ ) {
if ( channels[j].nffile != NULL ) {
// flush output buffer
if ( channels[j].nffile->block_header->NumRecords ) {
if ( WriteBlock(channels[j].nffile) <= 0 ) {
LogError("Failed to write output buffer to disk: '%s'" , strerror(errno));
}
}
}
}
CloseFile(nffile);
DisposeFile(nffile);
} // End of process_data
static profile_param_info_t *ParseParams (char *profile_datadir) {
struct stat stat_buf;
char line[512], path[MAXPATHLEN], *p, *q, *s;
profile_param_info_t *profile_list;
profile_param_info_t **list = &profile_list;
profile_list = NULL;
while ( ( fgets(line, 512, stdin) != NULL )) {
LogInfo("Process line '%s'\n", line);
line[511] = '\0';
if ( *list == NULL )
*list = (profile_param_info_t *)malloc(sizeof(profile_param_info_t));
// else we come from a continue statement with illegal data - overwrite
if ( !*list) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
(*list)->next = NULL;
(*list)->profilegroup = NULL;
(*list)->profilename = NULL;
(*list)->channelname = NULL;
(*list)->channel_sourcelist = NULL;
(*list)->profiletype = 0;
// delete '\n' at the end of line
// format of stdin config line:
// <profilegroup>#<profilename>#<profiletype>#<channelname>#<channel_sourcelist>
p = strchr(line, '\n');
if ( p ) *p = '\0';
q = line;
p = strchr(q, '#');
if ( p )
*p = '\0';
s = line;
// savety check: if no separator found loop to next line
if ( !p ) {
LogError("Incomplete line - channel skipped.\n");
continue;
}
q = p;
q++;
p = strchr(q, '#');
if ( p )
*p = '\0';
snprintf(path, MAXPATHLEN-1, "%s/%s/%s", profile_datadir, s, q);
path[MAXPATHLEN-1] = '\0';
if ( stat(path, &stat_buf) || !S_ISDIR(stat_buf.st_mode) ) {
LogError("profile '%s' not found in group %s. Skipped.\n", q, s);
continue;
}
(*list)->profilegroup = strdup(s);
(*list)->profilename = strdup(q);
// savety check: if no separator found loop to next line
if ( !p ) {
LogError("Incomplete line - channel skipped.\n");
continue;
}
q = p;
q++;
p = strchr(q, '#');
if ( p )
*p = '\0';
s = q;
while ( *s ) {
if ( *s < '0' || *s > '9' ) {
LogError("Not a valid number: %s\n", q);
s = NULL;
break;
}
s++;
}
if ( s == NULL )
continue;
(*list)->profiletype = (int)strtol(q, (char **)NULL, 10);
// savety check: if no separator found loop to next line
if ( !p ) {
LogError("Incomplete line - channel skipped.\n");
continue;
}
q = p;
q++;
p = strchr(q, '#');
if ( p )
*p = '\0';
snprintf(path, MAXPATHLEN-1, "%s/%s/%s/%s", profile_datadir, (*list)->profilegroup, (*list)->profilename, q);
path[MAXPATHLEN-1] = '\0';
if ( stat(path, &stat_buf) || !S_ISDIR(stat_buf.st_mode) ) {
LogError("channel '%s' in profile '%s' not found. Skipped.\n", q, (*list)->profilename);
continue;
}
(*list)->channelname = strdup(q);
if ( !p ) {
LogError("Incomplete line - Skipped.\n");
continue;
}
q = p;
q++;
p = strchr(q, '#');
if ( p )
*p = '\0';
// Skip leading '| chars
while ( *q && *q == '|' ) {
q++;
}
s = q;
// if q is already empty ( '\0' ) loop is not processed
while ( *s ) {
// as s[0] is not '\0' s[1] may be '\0' but still valid and in range
if ( s[0] == '|' && s[1] == '|' ) {
char *t = s;
t++;
while ( *t ) { // delete this empty channel name
t[0] = t[1];
t++;
}
} else
s++;
}
// we have no doublicate '|' here any more
// check if last char is an extra '|'
if ( *q && (q[strlen(q)-1] == '|') )
q[strlen(q)-1] = '\0';
if ( *q && (strcmp(q, "*") != 0) )
(*list)->channel_sourcelist = strdup(q);
list = &((*list)->next);
}
if ( *list != NULL ) {
free(*list);
*list = NULL;
}
if ( ferror(stdin) ) {
LogError("fgets() error: %s", strerror(errno));
return NULL;
}
return profile_list;
} // End of ParseParams
int main( int argc, char **argv ) {
unsigned int num_channels, compress;
struct stat stat_buf;
profile_param_info_t *profile_list;
char *rfile, *ffile, *filename, *Mdirs, *tstring;
char *profile_datadir, *profile_statdir, *nameserver;
int c, syntax_only, subdir_index, stdin_profile_params, do_xstat;
time_t tslot;
tstring = NULL;
profile_datadir = NULL;
profile_statdir = NULL;
Mdirs = NULL;
tslot = 0;
syntax_only = 0;
do_xstat = 0;
compress = 0;
subdir_index = 0;
profile_list = NULL;
nameserver = NULL;
stdin_profile_params = 0;
is_anonymized = 0;
strncpy(Ident, "none", IDENTLEN);
Ident[IDENTLEN-1] = '\0';
// default file names
ffile = "filter.txt";
rfile = NULL;
while ((c = getopt(argc, argv, "D:HIL:p:P:hf:r:n:M:S:t:VzZ")) != EOF) {
switch (c) {
case 'h':
usage(argv[0]);
exit(0);
break;
case 'D':
nameserver = optarg;
if ( !set_nameserver(nameserver) ) {
exit(255);
}
break;
case 'I':
stdin_profile_params = 1;
break;
case 'H':
do_xstat = 1;
break;
case 'L':
if ( !InitLog("nfprofile", optarg) )
exit(255);
break;
case 'Z':
syntax_only = 1;
break;
case 'p':
profile_datadir = optarg;
break;
case 'P':
profile_statdir = optarg;
break;
case 'S':
subdir_index = atoi(optarg);
break;
case 'V':
printf("%s: Version: %s\n",argv[0], nfdump_version);
exit(0);
break;
case 'f':
ffile = optarg;
break;
case 't':
tslot = atoi(optarg);
break;
case 'M':
Mdirs = optarg;
break;
case 'r':
rfile = optarg;
break;
case 'z':
compress = 1;
break;
default:
usage(argv[0]);
exit(0);
}
}
if ( subdir_index && !InitHierPath(subdir_index) ) {
exit(255);
}
if ( !profile_datadir ) {
LogError("Profile data directory required!\n");
exit(255);
}
if ( !profile_statdir ) {
profile_statdir = profile_datadir;
}
if ( stat(profile_datadir, &stat_buf) || !S_ISDIR(stat_buf.st_mode) ) {
LogError("'%s' not a directory\n", profile_datadir);
exit(255);
}
if ( stdin_profile_params ) {
profile_list = ParseParams(profile_datadir);
if ( !profile_list ) {
exit(254);
}
}
if ( syntax_only ) {
filename = NULL;
rfile = NULL;
} else {
char *p;
if ( rfile == NULL ) {
LogError("-r filename required!\n");
exit(255);
}
p = strrchr(rfile, '/');
filename = p == NULL ? rfile : ++p;
if ( strlen(filename) == 0 ) {
LogError("Filename error: zero length filename\n");
exit(254);
}
}
if ( chdir(profile_datadir)) {
LogError("Error can't chdir to '%s': %s", profile_datadir, strerror(errno));
exit(255);
}
num_channels = InitChannels(profile_datadir, profile_statdir, profile_list, ffile, filename, subdir_index, syntax_only, compress, do_xstat);
// nothing to do
if ( num_channels == 0 ) {
LogInfo("No channels to process.\n");
return 0;
}
if ( syntax_only ) {
printf("Syntax check done.\n");
return 0;
}
if ( !rfile ) {
LogError("Input file (-r) required!\n");
exit(255);
}
extension_map_list = InitExtensionMaps(NEEDS_EXTENSION_LIST);
if ( !InitExporterList() ) {
exit(255);
}
SetupInputFileSequence(Mdirs,rfile, NULL);
process_data(GetChannelInfoList(), num_channels, tslot, do_xstat);
CloseChannels(tslot, compress);
FreeExtensionMaps(extension_map_list);
return 0;
}

387
bin/nfreader.c Executable file
View File

@ -0,0 +1,387 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfreader.c 48 2010-01-02 08:06:27Z haag $
*
* $LastChangedRevision: 48 $
*
*
*/
/*
* nfreader is sample code for reading nfdump binary files.
* It accepts the standard nfdump file select options -r, -M and -R
* Therefore it allows you to loop over multiple files and process the netflow record.
*
* Insert your code in the process_data function after the call to ExpandRecord
* To build the binary: first compile nfdump as usual.
* Then compile nfreader:
*
* make nfreader
*
* This compiles this code and links the required nfdump files
* If you do it by hand:
*
* gcc -c nfreader.c
* gcc -o nfreader nfreader.o nffile.o flist.o util.o
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdarg.h>
#include <errno.h>
#include <time.h>
#include <string.h>
#include <ctype.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <fcntl.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "nfx.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "util.h"
#include "flist.h"
#if ( SIZEOF_VOID_P == 8 )
typedef uint64_t pointer_addr_t;
#else
typedef uint32_t pointer_addr_t;
#endif
// module limited globals
extension_map_list_t *extension_map_list;
extern generic_exporter_t **exporter_list;
/* Function Prototypes */
static void usage(char *name);
static void print_record(void *record, char *s );
static void process_data(void);
/* Functions */
#include "nffile_inline.c"
static void usage(char *name) {
printf("usage %s [options] \n"
"-h\t\tthis text you see right here\n"
"-r\t\tread input from file\n"
"-M <expr>\tRead input from multiple directories.\n"
"-R <expr>\tRead input from sequence of files.\n"
, name);
} /* usage */
static void print_record(void *record, char *s ) {
char as[40], ds[40], datestr1[64], datestr2[64];
time_t when;
struct tm *ts;
master_record_t *r = (master_record_t *)record;
if ( (r->flags & FLAG_IPV6_ADDR ) != 0 ) { // IPv6
r->v6.srcaddr[0] = htonll(r->v6.srcaddr[0]);
r->v6.srcaddr[1] = htonll(r->v6.srcaddr[1]);
r->v6.dstaddr[0] = htonll(r->v6.dstaddr[0]);
r->v6.dstaddr[1] = htonll(r->v6.dstaddr[1]);
inet_ntop(AF_INET6, r->v6.srcaddr, as, sizeof(as));
inet_ntop(AF_INET6, r->v6.dstaddr, ds, sizeof(ds));
} else { // IPv4
r->v4.srcaddr = htonl(r->v4.srcaddr);
r->v4.dstaddr = htonl(r->v4.dstaddr);
inet_ntop(AF_INET, &r->v4.srcaddr, as, sizeof(as));
inet_ntop(AF_INET, &r->v4.dstaddr, ds, sizeof(ds));
}
as[40-1] = 0;
ds[40-1] = 0;
when = r->first;
ts = localtime(&when);
strftime(datestr1, 63, "%Y-%m-%d %H:%M:%S", ts);
when = r->last;
ts = localtime(&when);
strftime(datestr2, 63, "%Y-%m-%d %H:%M:%S", ts);
snprintf(s, 1023, "\n"
"Flow Record: \n"
" srcaddr = %16s\n"
" dstaddr = %16s\n"
" first = %10u [%s]\n"
" last = %10u [%s]\n"
" msec_first = %5u\n"
" msec_last = %5u\n"
" prot = %3u\n"
" srcport = %5u\n"
" dstport = %5u\n"
" dPkts = %10llu\n"
" dOctets = %10llu\n"
,
as, ds, r->first, datestr1, r->last, datestr2, r->msec_first, r->msec_last,
r->prot, r->srcport, r->dstport,
(unsigned long long)r->dPkts, (unsigned long long)r->dOctets);
s[1024-1] = 0;
} // End of print_record
static void process_data(void) {
master_record_t master_record;
common_record_t *flow_record;
nffile_t *nffile;
int i, done, ret;
#ifdef COMPAT15
int v1_map_done = 0;
#endif
// Get the first file handle
nffile = GetNextFile(NULL, 0, 0);
if ( !nffile ) {
LogError("GetNextFile() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
if ( nffile == EMPTY_LIST ) {
LogError("Empty file list. No files to process\n");
return;
}
done = 0;
while ( !done ) {
// get next data block from file
ret = ReadBlock(nffile);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
fprintf(stderr, "Skip corrupt data file '%s'\n",GetCurrentFilename());
else
fprintf(stderr, "Read error in file '%s': %s\n",GetCurrentFilename(), strerror(errno) );
// fall through - get next file in chain
case NF_EOF: {
nffile_t *next = GetNextFile(nffile, 0, 0);
if ( next == EMPTY_LIST ) {
done = 1;
}
if ( next == NULL ) {
done = 1;
LogError("Unexpected end of file list\n");
}
// else continue with next file
continue;
} break; // not really needed
}
#ifdef COMPAT15
if ( nffile->block_header->id == DATA_BLOCK_TYPE_1 ) {
common_record_v1_t *v1_record = (common_record_v1_t *)nffile->buff_ptr;
// create an extension map for v1 blocks
if ( v1_map_done == 0 ) {
extension_map_t *map = malloc(sizeof(extension_map_t) + 2 * sizeof(uint16_t) );
if ( ! map ) {
LogError("malloc() allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
map->type = ExtensionMapType;
map->size = sizeof(extension_map_t) + 2 * sizeof(uint16_t);
if (( map->size & 0x3 ) != 0 ) {
map->size += 4 - ( map->size & 0x3 );
}
map->map_id = INIT_ID;
map->ex_id[0] = EX_IO_SNMP_2;
map->ex_id[1] = EX_AS_2;
map->ex_id[2] = 0;
map->extension_size = 0;
Insert_Extension_Map(extension_map_list, map);
v1_map_done = 1;
}
// convert the records to v2
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
common_record_t *v2_record = (common_record_t *)v1_record;
Convert_v1_to_v2((void *)v1_record);
// now we have a v2 record -> use size of v2_record->size
v1_record = (common_record_v1_t *)((pointer_addr_t)v1_record + v2_record->size);
}
nffile->block_header->id = DATA_BLOCK_TYPE_2;
}
#endif
if ( nffile->block_header->id == Large_BLOCK_Type ) {
// skip
continue;
}
if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) {
fprintf(stderr, "Can't process block type %u. Skip block.\n", nffile->block_header->id);
continue;
}
flow_record = nffile->buff_ptr;
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
char string[1024];
switch ( flow_record->type ) {
case CommonRecordType: {
uint32_t map_id = flow_record->ext_map;
generic_exporter_t *exp_info = exporter_list[flow_record->exporter_sysid];
if ( extension_map_list->slot[map_id] == NULL ) {
snprintf(string, 1024, "Corrupt data file! No such extension map id: %u. Skip record", flow_record->ext_map );
string[1023] = '\0';
} else {
ExpandRecord_v2( flow_record, extension_map_list->slot[flow_record->ext_map],
exp_info ? &(exp_info->info) : NULL, &master_record);
// update number of flows matching a given map
extension_map_list->slot[map_id]->ref_count++;
/*
* insert hier your calls to your processing routine
* master_record now contains the next flow record as specified in nffile.c
* for example you can print each record:
*
*/
print_record(&master_record, string);
printf("%s\n", string);
}
} break;
case ExtensionMapType: {
extension_map_t *map = (extension_map_t *)flow_record;
if ( Insert_Extension_Map(extension_map_list, map) ) {
// flush new map
} // else map already known and flushed
} break;
case ExporterInfoRecordType:
case ExporterStatRecordType:
case SamplerInfoRecordype:
// Silently skip exporter records
break;
default: {
fprintf(stderr, "Skip unknown record type %i\n", flow_record->type);
}
}
// Advance pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
} // for all records
} // while
CloseFile(nffile);
DisposeFile(nffile);
PackExtensionMapList(extension_map_list);
} // End of process_data
int main( int argc, char **argv ) {
char *rfile, *Rfile, *Mdirs;
int c;
rfile = Rfile = Mdirs = NULL;
while ((c = getopt(argc, argv, "L:r:M:R:")) != EOF) {
switch (c) {
case 'h':
usage(argv[0]);
exit(0);
break;
break;
case 'L':
if ( !InitLog("argv[0]", optarg) )
exit(255);
break;
case 'r':
rfile = optarg;
if ( strcmp(rfile, "-") == 0 )
rfile = NULL;
break;
case 'M':
Mdirs = optarg;
break;
case 'R':
Rfile = optarg;
break;
default:
usage(argv[0]);
exit(0);
}
}
if ( rfile && Rfile ) {
fprintf(stderr, "-r and -R are mutually exclusive. Please specify either -r or -R\n");
exit(255);
}
if ( Mdirs && !(rfile || Rfile) ) {
fprintf(stderr, "-M needs either -r or -R to specify the file or file list. Add '-R .' for all files in the directories.\n");
exit(255);
}
extension_map_list = InitExtensionMaps(NEEDS_EXTENSION_LIST);
if ( !InitExporterList() ) {
exit(255);
}
SetupInputFileSequence(Mdirs, rfile, Rfile);
process_data();
FreeExtensionMaps(extension_map_list);
return 0;
}

625
bin/nfreplay.c Normal file
View File

@ -0,0 +1,625 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfreplay.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdarg.h>
#include <errno.h>
#include <time.h>
#include <string.h>
#include <ctype.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <netdb.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/resource.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nffile.h"
#include "nfx.h"
#include "nf_common.h"
#include "rbtree.h"
#include "nftree.h"
#include "nfdump.h"
#include "nfnet.h"
#include "bookkeeper.h"
#include "nfxstat.h"
#include "collector.h"
#include "exporter.h"
#include "netflow_v5_v7.h"
#include "netflow_v9.h"
#include "nfprof.h"
#include "flist.h"
#include "util.h"
#include "grammar.h"
#define DEFAULTCISCOPORT "9995"
#define DEFAULTHOSTNAME "127.0.0.1"
#undef FPURGE
#ifdef HAVE___FPURGE
#define FPURGE __fpurge
#endif
#ifdef HAVE_FPURGE
#define FPURGE fpurge
#endif
/* Externals */
extern int yydebug;
#ifdef COMPAT15
extern extension_descriptor_t extension_descriptor[];
#endif
/* Global Variables */
FilterEngine_data_t *Engine;
int verbose;
/* Local Variables */
static const char *nfdump_version = VERSION;
send_peer_t peer;
extension_map_list_t *extension_map_list;
generic_exporter_t **exporter_list;
/* Function Prototypes */
static void usage(char *name);
static void send_blast(unsigned int delay );
static void send_data(char *rfile, time_t twin_start, time_t twin_end, uint32_t count,
unsigned int delay, int confirm, int netflow_version);
static int FlushBuffer(int confirm);
/* Functions */
#include "nffile_inline.c"
#include "nfdump_inline.c"
static void usage(char *name) {
printf("usage %s [options] [\"filter\"]\n"
"-h\t\tthis text you see right here\n"
"-V\t\tPrint version and exit.\n"
"-H <Host/ip>\tTarget IP address default: 127.0.0.1\n"
"-j <mcast>\tSend packets to multicast group\n"
"-4\t\tForce IPv4 protocol.\n"
"-6\t\tForce IPv6 protocol.\n"
"-L <log>\tLog to syslog facility <log>\n"
"-p <port>\tTarget port default 9995\n"
"-d <usec>\tDelay in usec between packets. default 10\n"
"-c <cnt>\tPacket count. default send all packets\n"
"-b <bsize>\tSend buffer size.\n"
"-r <input>\tread from file. default: stdin\n"
"-f <filter>\tfilter syntaxfile\n"
"-t <time>\ttime window for sending packets\n"
"\t\tyyyy/MM/dd.hh:mm:ss[-yyyy/MM/dd.hh:mm:ss]\n"
, name);
} /* usage */
static int FlushBuffer(int confirm) {
size_t len = (pointer_addr_t)peer.buff_ptr - (pointer_addr_t)peer.send_buffer;
static unsigned long cnt = 1;
peer.flush = 0;
peer.buff_ptr = peer.send_buffer;
if ( confirm ) {
FPURGE(stdin);
printf("Press any key to send next UDP packet [%lu] ", cnt++);
fflush(stdout);
fgetc(stdin);
}
return sendto(peer.sockfd, peer.send_buffer, len, 0, (struct sockaddr *)&(peer.addr), peer.addrlen);
} // End of FlushBuffer
static void send_blast(unsigned int delay ) {
common_flow_header_t *header;
nfprof_t profile_data;
int i, ret;
u_long usec, sec;
double fps;
peer.send_buffer = malloc(1400);
if ( !peer.send_buffer ) {
perror("Memory allocation error");
close(peer.sockfd);
return;
}
header = (common_flow_header_t *)peer.send_buffer;
header->version = htons(255);
nfprof_start(&profile_data);
for ( i = 0; i < 65535; i++ ) {
header->count = htons(i);
ret = sendto(peer.sockfd, peer.send_buffer, 1400, 0, (struct sockaddr *)&peer.addr, peer.addrlen);
if ( ret < 0 || ret != 1400 ) {
perror("Error sending data");
}
if ( delay ) {
// sleep as specified
usleep(delay);
}
}
nfprof_end(&profile_data, 8*65535*1400);
usec = profile_data.used.ru_utime.tv_usec + profile_data.used.ru_stime.tv_usec;
sec = profile_data.used.ru_utime.tv_sec + profile_data.used.ru_stime.tv_sec;
if (usec > 1000000)
usec -= 1000000, ++sec;
if (profile_data.tend.tv_usec < profile_data.tstart.tv_usec)
profile_data.tend.tv_usec += 1000000, --profile_data.tend.tv_sec;
usec = profile_data.tend.tv_usec - profile_data.tstart.tv_usec;
sec = profile_data.tend.tv_sec - profile_data.tstart.tv_sec;
fps = (double)profile_data.numflows / ((double)sec + ((double)usec/1000000));
fprintf(stdout, "Wall: %lu.%-3.3lus bps: %-10.1f\n", sec, usec/1000, fps);
} // End of send_blast
static void send_data(char *rfile, time_t twin_start,
time_t twin_end, uint32_t count, unsigned int delay, int confirm, int netflow_version) {
master_record_t master_record;
common_record_t *flow_record;
nffile_t *nffile;
int i, done, ret, again;
uint32_t numflows, cnt;
#ifdef COMPAT15
int v1_map_done = 0;
#endif
// Get the first file handle
nffile = GetNextFile(NULL, twin_start, twin_end);
if ( !nffile ) {
LogError("GetNextFile() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return;
}
if ( nffile == EMPTY_LIST ) {
LogError("Empty file list. No files to process\n");
return;
}
peer.send_buffer = malloc(UDP_PACKET_SIZE);
peer.flush = 0;
if ( !peer.send_buffer ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
CloseFile(nffile);
DisposeFile(nffile);
return;
}
peer.buff_ptr = peer.send_buffer;
peer.endp = (void *)((pointer_addr_t)peer.send_buffer + UDP_PACKET_SIZE - 1);
if ( netflow_version == 5 )
Init_v5_v7_output(&peer);
else
Init_v9_output(&peer);
numflows = 0;
done = 0;
// setup Filter Engine to point to master_record, as any record read from file
// is expanded into this record
Engine->nfrecord = (uint64_t *)&master_record;
cnt = 0;
while ( !done ) {
// get next data block from file
ret = ReadBlock(nffile);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
LogError("Skip corrupt data file '%s'\n",GetCurrentFilename());
else
LogError("Read error in file '%s': %s\n",GetCurrentFilename(), strerror(errno) );
// fall through - get next file in chain
case NF_EOF: {
nffile_t *next = GetNextFile(nffile, twin_start, twin_end);
if ( next == EMPTY_LIST ) {
done = 1;
}
if ( next == NULL ) {
done = 1;
LogError("Unexpected end of file list\n");
}
// else continue with next file
continue;
} break; // not really needed
}
#ifdef COMPAT15
if ( nffile->block_header->id == DATA_BLOCK_TYPE_1 ) {
common_record_v1_t *v1_record = (common_record_v1_t *)nffile->buff_ptr;
// create an extension map for v1 blocks
if ( v1_map_done == 0 ) {
extension_map_t *map = malloc(sizeof(extension_map_t) + 2 * sizeof(uint16_t) );
if ( ! map ) {
perror("Memory allocation error");
exit(255);
}
map->type = ExtensionMapType;
map->size = sizeof(extension_map_t) + 2 * sizeof(uint16_t);
if (( map->size & 0x3 ) != 0 ) {
map->size += 4 - ( map->size & 0x3 );
}
map->map_id = INIT_ID;
map->ex_id[0] = EX_IO_SNMP_2;
map->ex_id[1] = EX_AS_2;
map->ex_id[2] = 0;
map->extension_size = 0;
map->extension_size += extension_descriptor[EX_IO_SNMP_2].size;
map->extension_size += extension_descriptor[EX_AS_2].size;
Insert_Extension_Map(extension_map_list, map);
v1_map_done = 1;
}
// convert the records to v2
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
common_record_t *v2_record = (common_record_t *)v1_record;
Convert_v1_to_v2((void *)v1_record);
// now we have a v2 record -> use size of v2_record->size
v1_record = (common_record_v1_t *)((pointer_addr_t)v1_record + v2_record->size);
}
nffile->block_header->id = DATA_BLOCK_TYPE_2;
}
#endif
if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) {
LogError("Can't process block type %u. Skip block.\n", nffile->block_header->id);
continue;
}
// cnt is the number of blocks, which survived the filter
// and added to the output buffer
flow_record = nffile->buff_ptr;
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
int match;
switch ( flow_record->type ) {
case CommonRecordV0Type:
case CommonRecordType: {
if ( extension_map_list->slot[flow_record->ext_map] == NULL ) {
LogError("Corrupt data file. Missing extension map %u. Skip record.\n", flow_record->ext_map);
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
continue;
}
// if no filter is given, the result is always true
ExpandRecord_v2( flow_record, extension_map_list->slot[flow_record->ext_map], NULL, &master_record);
match = twin_start && (master_record.first < twin_start || master_record.last > twin_end) ? 0 : 1;
// filter netflow record with user supplied filter
if ( match )
match = (*Engine->FilterEngine)(Engine);
if ( match == 0 ) { // record failed to pass all filters
// increment pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
// go to next record
continue;
}
// Records passed filter -> continue record processing
if ( netflow_version == 5 )
again = Add_v5_output_record(&master_record, &peer);
else
again = Add_v9_output_record(&master_record, &peer);
cnt++;
numflows++;
if ( peer.flush ) {
ret = FlushBuffer(confirm);
if ( ret < 0 ) {
perror("Error sending data");
CloseFile(nffile);
DisposeFile(nffile);
return;
}
if ( delay ) {
// sleep as specified
usleep(delay);
}
cnt = 0;
}
if ( again ) {
if ( netflow_version == 5 )
Add_v5_output_record(&master_record, &peer);
else
Add_v9_output_record(&master_record, &peer);
cnt++;
}
} break;
case ExtensionMapType: {
extension_map_t *map = (extension_map_t *)flow_record;
if ( Insert_Extension_Map(extension_map_list, map) ) {
// flush new map
} // else map already known and flushed
} break;
case ExporterRecordType:
case SamplerRecordype:
case ExporterInfoRecordType:
case ExporterStatRecordType:
case SamplerInfoRecordype:
// Silently skip exporter/sampler records
break;
default: {
LogError("Skip unknown record type %i\n", flow_record->type);
}
}
// Advance pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
}
} // while
// flush still remaining records
if ( cnt ) {
ret = FlushBuffer(confirm);
if ( ret < 0 ) {
perror("Error sending data");
}
} // if cnt
if (nffile) {
CloseFile(nffile);
DisposeFile(nffile);
}
close(peer.sockfd);
return;
} // End of send_data
int main( int argc, char **argv ) {
struct stat stat_buff;
char *rfile, *ffile, *filter, *tstring;
int c, confirm, ffd, ret, blast, netflow_version;
unsigned int delay, count, sockbuff_size;
time_t t_start, t_end;
rfile = ffile = filter = tstring = NULL;
t_start = t_end = 0;
peer.hostname = NULL;
peer.port = DEFAULTCISCOPORT;
peer.mcast = 0;
peer.family = AF_UNSPEC;
peer.sockfd = 0;
delay = 1;
count = 0xFFFFFFFF;
sockbuff_size = 0;
netflow_version = 5;
blast = 0;
verbose = 0;
confirm = 0;
while ((c = getopt(argc, argv, "46BhH:i:K:L:p:d:c:b:j:r:f:t:v:VY")) != EOF) {
switch (c) {
case 'h':
usage(argv[0]);
exit(0);
break;
case 'B':
blast = 1;
break;
case 'V':
printf("%s: Version: %s\n",argv[0], nfdump_version);
exit(0);
break;
case 'Y':
confirm = 1;
break;
case 'H':
case 'i': // compatibility with old version
peer.hostname = strdup(optarg);
peer.mcast = 0;
break;
case 'j':
if ( peer.hostname == NULL ) {
peer.hostname = strdup(optarg);
peer.mcast = 1;
} else {
LogError("ERROR, -H(-i) and -j are mutually exclusive!!\n");
exit(255);
}
break;
case 'K':
LogError("*** Anonymization moved! Use nfanon to anonymize flows first!\n");
exit(255);
break;
case 'L':
if ( !InitLog(argv[0], optarg) )
exit(255);
break;
case 'p':
peer.port = strdup(optarg);
break;
case 'd':
delay = atoi(optarg);
break;
case 'v':
netflow_version = atoi(optarg);
if ( netflow_version != 5 && netflow_version != 9 ) {
LogError("Invalid netflow version: %s. Accept only 5 or 9!\n", optarg);
exit(255);
}
break;
case 'c':
count = atoi(optarg);
break;
case 'b':
sockbuff_size = atoi(optarg);
break;
case 'f':
ffile = optarg;
break;
case 't':
tstring = optarg;
break;
case 'r':
rfile = optarg;
break;
case '4':
if ( peer.family == AF_UNSPEC )
peer.family = AF_INET;
else {
LogError("ERROR, Accepts only one protocol IPv4 or IPv6!\n");
exit(255);
}
break;
case '6':
if ( peer.family == AF_UNSPEC )
peer.family = AF_INET6;
else {
LogError("ERROR, Accepts only one protocol IPv4 or IPv6!\n");
exit(255);
}
break;
default:
usage(argv[0]);
exit(0);
}
}
if (argc - optind > 1) {
usage(argv[0]);
exit(255);
} else {
/* user specified a pcap filter */
filter = argv[optind];
}
if ( peer.hostname == NULL )
peer.hostname = DEFAULTHOSTNAME;
if ( !filter && ffile ) {
if ( stat(ffile, &stat_buff) ) {
perror("Can't stat file");
exit(255);
}
filter = (char *)malloc(stat_buff.st_size);
if ( !filter ) {
perror("Memory error");
exit(255);
}
ffd = open(ffile, O_RDONLY);
if ( ffd < 0 ) {
perror("Can't open file");
exit(255);
}
ret = read(ffd, (void *)filter, stat_buff.st_size);
if ( ret < 0 ) {
perror("Error reading file");
close(ffd);
exit(255);
}
close(ffd);
}
if ( !filter )
filter = "any";
Engine = CompileFilter(filter);
if ( !Engine )
exit(254);
if ( peer.mcast )
peer.sockfd = Multicast_send_socket (peer.hostname, peer.port, peer.family, sockbuff_size,
&peer.addr, &peer.addrlen );
else
peer.sockfd = Unicast_send_socket (peer.hostname, peer.port, peer.family, sockbuff_size,
&peer.addr, &peer.addrlen );
if ( peer.sockfd <= 0 ) {
exit(255);
}
if ( blast ) {
send_blast(delay );
exit(0);
}
extension_map_list = InitExtensionMaps(NEEDS_EXTENSION_LIST);
SetupInputFileSequence(NULL,rfile, NULL);
if ( tstring ) {
if ( !ScanTimeFrame(tstring, &t_start, &t_end) )
exit(255);
}
send_data(rfile, t_start, t_end, count, delay, confirm, netflow_version);
FreeExtensionMaps(extension_map_list);
return 0;
}

1913
bin/nfstat.c Normal file
View File

File diff suppressed because it is too large Load Diff

125
bin/nfstat.h Normal file
View File

@ -0,0 +1,125 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfstat.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFSTAT_H
#define _NFSTAT_H 1
/* Definitions */
/*
* Stat Table
* In order to generate any flow element statistics, the flows passed the filter
* are stored into an internal hash table.
*/
typedef struct SumRecord_s {
uint64_t flows;
uint64_t ipkg;
uint64_t opkg;
uint64_t ibyte;
uint64_t obyte;
} SumRecord_t;
typedef struct StatRecord {
// record chain
struct StatRecord *next;
// flow parameters
uint64_t counter[5]; // flows ipkg ibyte opkg obyte
uint32_t first;
uint32_t last;
uint16_t msec_first;
uint16_t msec_last;
uint8_t record_flags;
uint8_t tcp_flags;
uint8_t tos;
// key
uint8_t prot;
uint64_t stat_key[2];
} StatRecord_t;
typedef struct hash_StatTable {
/* hash table data */
uint16_t NumBits; /* width of the hash table */
uint32_t IndexMask; /* Mask which corresponds to NumBits */
StatRecord_t **bucket; /* Hash entry point: points to elements in the stat block */
StatRecord_t **bucketcache; /* in case of index collisions, this array points to the last element with that index */
/* memory management */
/* memory blocks - containing the stat records */
StatRecord_t **memblock; /* array holding all NumBlocks allocated stat blocks */
uint32_t MaxBlocks; /* Size of memblock array */
/* stat blocks - containing the stat records */
uint32_t NumBlocks; /* number of allocated stat blocks in memblock array */
uint32_t Prealloc; /* Number of stat records in each stat block */
uint32_t NextBlock; /* This stat block contains the next free slot for a stat recorrd */
uint32_t NextElem; /* This element in the current stat block is the next free slot */
} hash_StatTable;
typedef struct SortElement {
void *record;
uint64_t count;
} SortElement_t;
#define MULTIPLE_LIST_ORDERS 1
#define SINGLE_LIST_ORDER 0
/* Function prototypes */
void SetLimits(int stat, char *packet_limit_string, char *byte_limit_string );
int Init_StatTable(uint16_t NumBits, uint32_t Prealloc);
void Dispose_StatTable(void);
int SetStat(char *str, int *element_stat, int *flow_stat);
int Parse_PrintOrder(char *order);
void AddStat(common_record_t *raw_record, master_record_t *flow_record );
void PrintFlowTable(printer_t print_record, uint32_t limitflows, int tag, int GuessDir, extension_map_list_t *extension_map_list);
void PrintFlowStat(char *record_header, printer_t print_record, int topN, int tag, int quiet, int cvs_output, extension_map_list_t *extension_map_list);
void PrintElementStat(stat_record_t *sum_stat, uint32_t limitflows, char *record_header, printer_t print_record, int topN, int tag, int quiet, int pipe_output, int cvs_output);
int ParseListOrder(char *s, int multiple_orders );
void PrintSortedFlows(printer_t print_record, uint32_t limitflows, int tag);
#endif //_NFSTAT_H

579
bin/nfstatfile.c Normal file
View File

@ -0,0 +1,579 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfstatfile.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/file.h>
#include <sys/param.h>
#include <sys/uio.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "nfstatfile.h"
#define stat_filename ".nfstat"
typedef struct config_def_s {
char *name;
// int type;
uint64_t *value;
} config_def_t;
static dirstat_t dirstat_tmpl;
static config_def_t config_def[] = {
{ "first", &dirstat_tmpl.first},
{ "last", &dirstat_tmpl.last },
{ "size", &dirstat_tmpl.filesize },
{ "maxsize", &dirstat_tmpl.max_size },
{ "numfiles", &dirstat_tmpl.numfiles },
{ "lifetime", &dirstat_tmpl.max_lifetime },
{ "watermark", &dirstat_tmpl.low_water },
{ "status", &dirstat_tmpl.status },
{ NULL, NULL },
};
#define STACK_BLOCK_SIZE 32
static int stack_max_entries = 0;
static dirstat_env_t *dirstat_stack = NULL;
static const double _1K = 1024.0;
static const double _1M = 1024.0 * 1024.0;
static const double _1G = 1024.0 * 1024.0 * 1024.0;
static const double _1T = 1024.0 * 1024.0 * 1024.0 * 1024.0;
static const double _1min = 60.0;
static const double _1hour = 3600.0;
static const double _1day = 86400.0;
static const double _1week = 604800.0;
/*
* expire.c is needed for daemon code as well as normal stdio code
* therefore a generic LogError is defined, which maps to the
* approriate logging channel - either stderr or syslog
*/
void LogError(char *format, ...);
static inline uint64_t string2uint64(char *s);
static int ParseString(char *str, char **key, char **value);
static void VerifyStatInfo(dirstat_t *statinfo);
char *ScaleValue(uint64_t v) {
double f = v;
static char s[64];
if ( f < _1K ) { // 1 K 1024
snprintf(s, 63, "%llu B", (unsigned long long)v);
} else if ( f < _1M ) {
snprintf(s, 63, "%llu = %.1f KB", (unsigned long long)v, f / _1K );
} else if ( f < _1G ) {
snprintf(s, 63, "%llu = %.1f MB", (unsigned long long)v, f / _1M );
} else if ( f < _1T ) {
snprintf(s, 63, "%llu = %.1f GB", (unsigned long long)v, f / _1G );
} else { // everything else in T
snprintf(s, 63, "%llu = %.1f TB", (unsigned long long)v, f / _1T );
}
s[63] = '\0';
return s;
} // End of ScaleValue
char *ScaleTime(uint64_t v) {
double f = v;
static char s[64];
if ( f < _1min ) {
snprintf(s, 63, "%llu sec", (unsigned long long)v);
} else if ( f < _1hour ) {
snprintf(s, 63, "%llu = %.1f min", (unsigned long long)v, f / _1min );
} else if ( f < _1day ) {
snprintf(s, 63, "%llu = %.1f hours", (unsigned long long)v, f / _1hour );
} else if ( f < _1week ) {
snprintf(s, 63, "%llu = %.1f days", (unsigned long long)v, f / _1day );
} else { // everything else in weeks
snprintf(s, 63, "%llu = %.1f weeks", (unsigned long long)v, f / _1week );
}
s[63] = '\0';
return s;
} // End of ScaleValue
static inline uint64_t string2uint64(char *s) {
uint64_t u=0;
char *p = s;
while( *p ) {
if ( *p < '0' || *p > '9' )
*p = '0';
u = 10LL*u + (*p++ - 48);
}
return u;
} // End of string2uint64
static int SetFileLock(int fd) {
struct flock fl;
fl.l_type = F_WRLCK; /* F_RDLCK, F_WRLCK, F_UNLCK */
fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
fl.l_start = 0; /* Offset from l_whence */
fl.l_len = 0; /* length, 0 = to EOF */
fl.l_pid = getpid(); /* our PID */
return fcntl(fd, F_SETLKW, &fl); /* F_GETLK, F_SETLK, F_SETLKW */
} // End of SetFileLock
static int ReleaseFileLock(int fd) {
struct flock fl;
fl.l_type = F_UNLCK; /* F_RDLCK, F_WRLCK, F_UNLCK */
fl.l_whence = SEEK_SET; /* SEEK_SET, SEEK_CUR, SEEK_END */
fl.l_start = 0; /* Offset from l_whence */
fl.l_len = 0; /* length, 0 = to EOF */
fl.l_pid = getpid(); /* our PID */
return fcntl(fd, F_SETLK, &fl); /* set the region to unlocked */
} // End of SetFileLock
static int ParseString(char *str, char **key, char **value) {
char *k, *v, *w;
k = str;
v = strpbrk(str, "=");
if ( !v ) {
printf("Invalid config line: '%s'\n", str);
*key = NULL;
*value = NULL;
return 0;
}
*v++ = '\0';
// strip white spaces from end of key
w = strpbrk(k, " ");
if ( w )
*w = '\0';
// strip white spaces from start of value
while ( *v == ' ' ) {
v++;
}
*key = k;
*value = v;
return 1;
} // End of ParseString
static void VerifyStatInfo(dirstat_t *statinfo) {
if ( ( statinfo->first == 0 ) || ( statinfo->first > statinfo->last ||
( statinfo->status > FORCE_REBUILD ) || ( statinfo->low_water > 100 ) ) )
statinfo->status = FORCE_REBUILD; // -> fishy
} // End of VerifyStatInfo
/*
* Reads the stat record from .nfstat file
* dirname: directory to read the .nfstat file
* dirstat_p: Assign a point of the result to this pointer
* lock: READ_ONLY file is locked while reading, and unlocked and closed thereafter
* CREATE_AND_LOCK if file does not exists, create it - continue as LOCK_IF_EXISTS
* LOCK_IF_EXISTS: lock the file if it exists - file remains open
* If file does not exists, an empty record is returned.
*/
int ReadStatInfo(char *dirname, dirstat_t **dirstat_p, int lock ) {
struct stat filestat;
char *in_buff, *s, *p, *k, *v;
char filename[MAXPATHLEN];
int fd, err, r_size, next_free;
*dirstat_p = NULL;
// if the dirstack does not exist, creat it
if ( !dirstat_stack ) {
int i;
dirstat_stack = (dirstat_env_t *)malloc(STACK_BLOCK_SIZE * sizeof(dirstat_env_t));
if ( !dirstat_stack ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return ERR_FAIL;
}
for ( i=0; i<STACK_BLOCK_SIZE; i++ ) {
dirstat_stack[i].dirstat = NULL;
}
stack_max_entries = STACK_BLOCK_SIZE;
}
// search for next free slot
next_free = 0;
while ( next_free < stack_max_entries && (dirstat_stack[next_free].dirstat != NULL) )
next_free++;
// if too many entries exist, expand the stack
if ( next_free >= stack_max_entries ) {
dirstat_env_t *tmp;
int i;
tmp = realloc((void *)dirstat_stack, (stack_max_entries+STACK_BLOCK_SIZE) * sizeof(dirstat_env_t));
if ( !tmp ) {
LogError("ralloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return ERR_FAIL;
}
dirstat_stack = tmp;
for ( i=stack_max_entries; i<stack_max_entries+STACK_BLOCK_SIZE; i++ ) {
dirstat_stack[i].dirstat = NULL;
}
next_free = stack_max_entries;
stack_max_entries += STACK_BLOCK_SIZE;
}
dirstat_stack[next_free].dirstat = (dirstat_t *)malloc(sizeof(dirstat_t));
if ( !dirstat_stack[next_free].dirstat ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return ERR_FAIL;
}
// Initialize config
snprintf(filename, MAXPATHLEN-1, "%s/%s", dirname, stat_filename);
filename[MAXPATHLEN-1] = '\0';
memset((void *)dirstat_stack[next_free].dirstat, 0, sizeof(dirstat_t));
memset((void *)&dirstat_tmpl, 0, sizeof(dirstat_t));
dirstat_tmpl.low_water = 95; // defaults to 95%
dirstat_tmpl.status = FORCE_REBUILD; // in case status is not set -> fishy
*dirstat_p = dirstat_stack[next_free].dirstat;
dirstat_stack[next_free].fd = 0;
dirstat_stack[next_free].filename = strdup(filename);
fd = open(filename, O_RDWR, 0);
if ( fd < 0 ) {
if ( errno == ENOENT ) {
if ( lock == READ_ONLY || lock == LOCK_IF_EXISTS) { // no lock need
return ERR_NOSTATFILE;
} else { // create the file, to and lock the file
fd = open(filename, O_RDWR|O_TRUNC|O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
if ( fd < 0 ) {
LogError("open() error on '%s' in %s line %d: %s\n", filename, __FILE__, __LINE__, strerror(errno) );
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
err = SetFileLock(fd);
if ( err != 0 ) {
LogError("ioctl(F_WRLCK) error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
close(fd);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
dirstat_stack[next_free].fd = fd;
return ERR_NOSTATFILE;
}
} else {
LogError("open() error on '%s' in %s line %d: %s\n", filename, __FILE__, __LINE__, strerror(errno) );
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
}
err = SetFileLock(fd);
if ( err != 0 ) {
LogError("ioctl(F_WRLCK) error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
close(fd);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
fstat(fd, &filestat);
// the file is not assumed to be larger than 1MB, otherwise it is likely corrupt
if ( filestat.st_size > 1024*1024 ) {
LogError("File size error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
ReleaseFileLock(fd);
close(fd);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
in_buff = (char *)malloc(filestat.st_size+1); // +1 for trailing '\0'
if ( !in_buff ) {
LogError("mallow() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
ReleaseFileLock(fd);
close(fd);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
r_size = read(fd, (void *)in_buff, filestat.st_size);
if ( r_size < 0 ) {
LogError("read() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
ReleaseFileLock(fd);
close(fd);
free(in_buff);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
in_buff[filestat.st_size] = '\0';
if ( r_size != filestat.st_size ) {
LogError("read() requested size error in %s line %d\n", __FILE__, __LINE__);
ReleaseFileLock(fd);
close(fd);
free(in_buff);
free(dirstat_stack[next_free].dirstat);
dirstat_stack[next_free].dirstat = NULL;
return ERR_FAIL;
}
if ( lock == READ_ONLY ) {
ReleaseFileLock(fd);
close(fd);
} else {
dirstat_stack[next_free].fd = fd;
}
p = in_buff;
while ( p && *p ) {
if ( *p == '#' ) { // skip comments
s = strpbrk(p, "\n");
if ( s ) { // "\n" found - advance p
*s = '\0';
printf("comment: '%s'\n",p);
p = s+1;
continue; // next line
}
}
// get gext key=value pair
s = strpbrk(p, "\n");
if ( s )
*s = '\0';
if ( ParseString(p, &k, &v) ) {
uint32_t i;
i = 0;
while ( config_def[i].name ) {
if ( strcmp(config_def[i].name, k) == 0 ) {
*(config_def[i].value) = string2uint64(v);
// printf("key: '%s', value '%s' int: %llu\n", k,v, *(config_def[i].value));
break;
}
i++;
}
if ( config_def[i].name == NULL ) {
printf("Invalid config key: '%s'\n", k);
}
}
p = s;
if ( p )
p++;
}
VerifyStatInfo(&dirstat_tmpl);
*dirstat_stack[next_free].dirstat = dirstat_tmpl;
free(in_buff);
return dirstat_tmpl.status;
} // End of ReadStatInfo
int WriteStatInfo(dirstat_t *dirstat) {
int i, index, fd, err;
char *filename, line[256];
// search for entry in dirstat stack
for (i=0; dirstat_stack[i].dirstat != dirstat && i < stack_max_entries; i++ ) {}
if ( i >= stack_max_entries ) {
LogError( "WriteStatInfo(): dirstat entry not found in %s line %d\n", __FILE__, __LINE__ );
return ERR_FAIL;
}
index = i;
fd = dirstat_stack[index].fd;
filename = dirstat_stack[index].filename;
if ( fd == 0 ) {
fd = open(filename, O_RDWR|O_TRUNC|O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
if ( fd < 0 ) {
LogError( "open() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return ERR_FAIL;
}
err = SetFileLock(fd);
if ( err != 0 ) {
LogError( "ioctl(F_WRLCK) error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
close(fd);
return ERR_FAIL;
}
} else {
err = lseek(fd, SEEK_SET, 0);
if ( err == -1 ) {
LogError( "lseek() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
ReleaseFileLock(fd);
close(fd);
return ERR_FAIL;
}
if ( ftruncate(fd, 0) < 0 ) {
LogError( "ftruncate() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
}
dirstat_tmpl = *dirstat_stack[index].dirstat;
i = 0;
while ( config_def[i].name ) {
size_t len;
snprintf(line, 255, "%s=%llu\n", config_def[i].name, (unsigned long long)*(config_def[i].value));
line[255] = '\0';
len = strlen(line);
if ( write(fd, line, len) < 0 ) {
LogError( "write() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
}
i++;
}
ReleaseFileLock(fd);
err = close(fd);
dirstat_stack[index].fd = 0;
if ( err == -1 ) {
LogError( "close() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return ERR_FAIL;
}
return STATFILE_OK;
} // End of WriteStatInfo
int ReleaseStatInfo(dirstat_t *dirstat) {
int i, index;
// search for entry in dirstat stack
for (i=0; dirstat_stack[i].dirstat != dirstat && i < stack_max_entries; i++ ) {}
if ( i >= stack_max_entries ) {
LogError( "ReleaseStatInfo() error in %s line %d: %s\n", __FILE__, __LINE__, "dirstat entry not found" );
return ERR_FAIL;
}
index = i;
if ( dirstat_stack[index].filename == NULL ) {
LogError( "ReleaseStatInfo() error in %s line %d: %s\n", __FILE__, __LINE__, "Attempted to free NULL pointer" );
return ERR_FAIL;
}
free(dirstat_stack[index].filename);
free(dirstat_stack[index].dirstat);
dirstat_stack[index].dirstat = NULL;
return 0;
} // End of ReleaseStatInfo
void PrintDirStat(dirstat_t *dirstat) {
struct tm *ts;
time_t t;
char string[32];
t = dirstat->first;
ts = localtime(&t);
strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts);
string[31] = '\0';
printf("First: %s\n", string);
t = dirstat->last;
ts = localtime(&t);
strftime(string, 31, "%Y-%m-%d %H:%M:%S", ts);
string[31] = '\0';
printf("Last: %s\n", string);
printf("Lifetime: %s\n", ScaleTime(dirstat->last - dirstat->first));
printf("Numfiles: %llu\n", (unsigned long long)dirstat->numfiles);
printf("Filesize: %s\n", ScaleValue(dirstat->filesize));
if ( dirstat->max_size )
printf("Max Size: %s\n", ScaleValue(dirstat->max_size));
else
printf("Max Size: <none>\n");
if ( dirstat->max_lifetime )
printf("Max Life: %s\n", ScaleTime(dirstat->max_lifetime));
else
printf("Max Life: <none>\n");
printf("Watermark: %llu%%\n", (unsigned long long)dirstat->low_water);
switch(dirstat->status) {
case STATFILE_OK:
printf("Status: OK\n");
break;
case FORCE_REBUILD:
printf("Status: Force rebuild\n");
break;
default:
printf("Status: Unexpected: %llu\n", (unsigned long long)dirstat->status);
break;
}
} // End of PrintDirStat

80
bin/nfstatfile.h Normal file
View File

@ -0,0 +1,80 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfstatfile.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFSTATFILE_H
#define _NFSTATFILE_H 1
typedef struct dirstat_s {
uint64_t first; // for more easy parsing and assigning, take a uint64_t also for the time_t type
uint64_t last;
uint64_t numfiles;
uint64_t filesize;
uint64_t max_size;
uint64_t max_lifetime;
uint64_t low_water;
uint64_t status;
} dirstat_t;
typedef struct dirstat_env_s {
dirstat_t *dirstat;
int fd;
char *filename;
int index;
} dirstat_env_t;
enum { STATFILE_OK = 0, ERR_FAIL, ERR_NOSTATFILE, FORCE_REBUILD };
#define READ_ONLY 0
#define CREATE_AND_LOCK 1
#define LOCK_IF_EXISTS 2
#define stat_filename ".nfstat"
char *ScaleValue(uint64_t v);
char *ScaleTime(uint64_t v);
void PrintDirStat(dirstat_t *dirstat);
int ReadStatInfo(char *dirname, dirstat_t **dirstat_p, int lock );
int WriteStatInfo(dirstat_t *dirstat);
int ReleaseStatInfo(dirstat_t *dirstat);
#endif //_NFSTATFILE_H

1244
bin/nftest.c Normal file
View File

File diff suppressed because it is too large Load Diff

631
bin/nftree.c Normal file
View File

@ -0,0 +1,631 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nftree.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <string.h>
#include <errno.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "rbtree.h"
#include "nfdump.h"
#include "nffile.h"
#include "nf_common.h"
#include "ipconv.h"
#include "nftree.h"
#include "grammar.h"
/*
* netflow filter engine
*
*/
extern char *CurrentIdent;
#define MAXBLOCKS 1024
static FilterBlock_t *FilterTree;
static uint32_t memblocks;
static uint32_t NumBlocks = 1; /* index 0 reserved */
#define IdentNumBlockSize 32
static uint16_t MaxIdents;
static uint16_t NumIdents;
static char **IdentList;
static void UpdateList(uint32_t a, uint32_t b);
/* flow processing functions */
static inline void pps_function(uint64_t *record_data, uint64_t *comp_values);
static inline void bps_function(uint64_t *record_data, uint64_t *comp_values);
static inline void bpp_function(uint64_t *record_data, uint64_t *comp_values);
static inline void duration_function(uint64_t *record_data, uint64_t *comp_values);
static inline void mpls_eos_function(uint64_t *record_data, uint64_t *comp_values);
static inline void mpls_any_function(uint64_t *record_data, uint64_t *comp_values);
static inline void pblock_function(uint64_t *record_data, uint64_t *comp_values);
/*
* flow processing function table:
* order of entries must correspond with filter functions enum in nftree.h
*/
static struct flow_procs_map_s {
char *name;
flow_proc_t function;
} flow_procs_map[] = {
{"none", NULL},
{"pps", pps_function},
{"bps", bps_function},
{"bpp", bpp_function},
{"duration", duration_function},
{"mpls eos", mpls_eos_function},
{"mpls any", mpls_any_function},
{"pblock", pblock_function},
{NULL, NULL}
};
uint64_t *IPstack = NULL;
uint32_t StartNode;
uint16_t Extended;
// 128bit compare for IPv6
static int IPNodeCMP(struct IPListNode *e1, struct IPListNode *e2) {
uint64_t ip_e1[2], ip_e2[2];
ip_e1[0] = e1->ip[0] & e2->mask[0];
ip_e1[1] = e1->ip[1] & e2->mask[1];
ip_e2[0] = e2->ip[0] & e1->mask[0];
ip_e2[1] = e2->ip[1] & e1->mask[1];
if ( ip_e1[0] == ip_e2[0] ) {
if ( ip_e1[1] == ip_e2[1] )
return 0;
else
return (ip_e1[1] < ip_e2[1] ? -1 : 1);
} else {
return (ip_e1[0] < ip_e2[0] ? -1 : 1);
}
} // End of IPNodeCMP
// 64bit uint64 compare
static int ULNodeCMP(struct ULongListNode *e1, struct ULongListNode *e2) {
if ( e1->value == e2->value )
return 0;
else
return (e1->value < e2->value ? -1 : 1);
} // End of ULNodeCMP
// Insert the IP RB tree code here
RB_GENERATE(IPtree, IPListNode, entry, IPNodeCMP);
// Insert the Ulong RB tree code here
RB_GENERATE(ULongtree, ULongListNode, entry, ULNodeCMP);
void InitTree(void) {
memblocks = 1;
FilterTree = (FilterBlock_t *)malloc(MAXBLOCKS * sizeof(FilterBlock_t));
if ( !FilterTree ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
ClearFilter();
} // End of InitTree
/*
* Clear Filter
*/
void ClearFilter(void) {
NumBlocks = 1;
Extended = 0;
MaxIdents = 0;
NumIdents = 0;
IdentList = NULL;
memset((void *)FilterTree, 0, MAXBLOCKS * sizeof(FilterBlock_t));
} /* End of ClearFilter */
FilterEngine_data_t *CompileFilter(char *FilterSyntax) {
FilterEngine_data_t *engine;
int ret;
if ( !FilterSyntax )
return NULL;
IPstack = (uint64_t *)malloc(16 * MAXHOSTS);
if ( !IPstack ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
if ( !InitSymbols() )
exit(255);
InitTree();
lex_init(FilterSyntax);
ret = yyparse();
if ( ret != 0 ) {
return NULL;
}
lex_cleanup();
free(IPstack);
engine = malloc(sizeof(FilterEngine_data_t));
if ( !engine ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
engine->nfrecord = NULL;
engine->StartNode = StartNode;
engine->Extended = Extended;
engine->IdentList = IdentList;
engine->filter = FilterTree;
if ( Extended )
engine->FilterEngine = RunExtendedFilter;
else
engine->FilterEngine = RunFilter;
return engine;
} // End of GetTree
/*
* For testing purpose only
*/
int nblocks(void) {
return NumBlocks - 1;
} /* End of nblocks */
/*
* Returns next free slot in blocklist
*/
uint32_t NewBlock(uint32_t offset, uint64_t mask, uint64_t value, uint16_t comp, uint32_t function, void *data) {
uint32_t n = NumBlocks;
if ( n >= ( memblocks * MAXBLOCKS ) ) {
memblocks++;
FilterTree = realloc(FilterTree, memblocks * MAXBLOCKS * sizeof(FilterBlock_t));
if ( !FilterTree ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
}
FilterTree[n].offset = offset;
FilterTree[n].mask = mask;
FilterTree[n].value = value;
FilterTree[n].invert = 0;
FilterTree[n].OnTrue = 0;
FilterTree[n].OnFalse = 0;
FilterTree[n].comp = comp;
FilterTree[n].function = flow_procs_map[function].function;
FilterTree[n].fname = flow_procs_map[function].name;
FilterTree[n].data = data;
if ( comp > 0 || function > 0 )
Extended = 1;
FilterTree[n].numblocks = 1;
FilterTree[n].blocklist = (uint32_t *)malloc(sizeof(uint32_t));
FilterTree[n].superblock = n;
FilterTree[n].blocklist[0] = n;
NumBlocks++;
return n;
} /* End of NewBlock */
/*
* Connects the two blocks b1 and b2 ( AND ) and returns index of superblock
*/
uint32_t Connect_AND(uint32_t b1, uint32_t b2) {
uint32_t a, b, i, j;
if ( FilterTree[b1].numblocks <= FilterTree[b2].numblocks ) {
a = b1;
b = b2;
} else {
a = b2;
b = b1;
}
/* a points to block with less children and becomes the superblock
* connect b to a
*/
for ( i=0; i < FilterTree[a].numblocks; i++ ) {
j = FilterTree[a].blocklist[i];
if ( FilterTree[j].invert ) {
if ( FilterTree[j].OnFalse == 0 ) {
FilterTree[j].OnFalse = b;
}
} else {
if ( FilterTree[j].OnTrue == 0 ) {
FilterTree[j].OnTrue = b;
}
}
}
UpdateList(a,b);
return a;
} /* End of Connect_AND */
/*
* Connects the two blocks b1 and b2 ( OR ) and returns index of superblock
*/
uint32_t Connect_OR(uint32_t b1, uint32_t b2) {
uint32_t a, b, i, j;
if ( FilterTree[b1].numblocks <= FilterTree[b2].numblocks ) {
a = b1;
b = b2;
} else {
a = b2;
b = b1;
}
/* a points to block with less children and becomes the superblock
* connect b to a
*/
for ( i=0; i < FilterTree[a].numblocks; i++ ) {
j = FilterTree[a].blocklist[i];
if ( FilterTree[j].invert ) {
if ( FilterTree[j].OnTrue == 0 ) {
FilterTree[j].OnTrue = b;
}
} else {
if ( FilterTree[j].OnFalse == 0 ) {
FilterTree[j].OnFalse = b;
}
}
}
UpdateList(a,b);
return a;
} /* End of Connect_OR */
/*
* Inverts OnTrue and OnFalse
*/
uint32_t Invert(uint32_t a) {
uint32_t i, j;
for ( i=0; i< FilterTree[a].numblocks; i++ ) {
j = FilterTree[a].blocklist[i];
FilterTree[j].invert = FilterTree[j].invert ? 0 : 1 ;
}
return a;
} /* End of Invert */
/*
* Update supernode infos:
* node 'b' was connected to 'a'. update node 'a' supernode data
*/
static void UpdateList(uint32_t a, uint32_t b) {
size_t s;
uint32_t i,j;
/* numblocks contains the number of blocks in the superblock */
s = FilterTree[a].numblocks + FilterTree[b].numblocks;
FilterTree[a].blocklist = (uint32_t *)realloc(FilterTree[a].blocklist, s * sizeof(uint32_t));
if ( !FilterTree[a].blocklist ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(250);
}
/* connect list of node 'b' after list of node 'a' */
j = FilterTree[a].numblocks;
for ( i=0; i< FilterTree[b].numblocks; i++ ) {
FilterTree[a].blocklist[j+i] = FilterTree[b].blocklist[i];
}
FilterTree[a].numblocks = s;
/* set superblock info of all children to new superblock */
for ( i=0; i< FilterTree[a].numblocks; i++ ) {
j = FilterTree[a].blocklist[i];
FilterTree[j].superblock = a;
}
/* cleanup old node 'b' */
FilterTree[b].numblocks = 0;
if ( FilterTree[b].blocklist )
free(FilterTree[b].blocklist);
} /* End of UpdateList */
/*
* Dump Filterlist
*/
void DumpList(FilterEngine_data_t *args) {
uint32_t i, j;
for (i=1; i<NumBlocks; i++ ) {
if ( args->filter[i].invert )
printf("Index: %u, Offset: %u, Mask: %.16llx, Value: %.16llx, Superblock: %u, Numblocks: %u, !OnTrue: %u, !OnFalse: %u Comp: %u Function: %s\n",
i, args->filter[i].offset, (unsigned long long)args->filter[i].mask,
(unsigned long long)args->filter[i].value, args->filter[i].superblock,
args->filter[i].numblocks, args->filter[i].OnTrue, args->filter[i].OnFalse, args->filter[i].comp, args->filter[i].fname);
else
printf("Index: %u, Offset: %u, Mask: %.16llx, Value: %.16llx, Superblock: %u, Numblocks: %u, OnTrue: %u, OnFalse: %u Comp: %u Function: %s\n",
i, args->filter[i].offset, (unsigned long long)args->filter[i].mask,
(unsigned long long)args->filter[i].value, args->filter[i].superblock,
args->filter[i].numblocks, args->filter[i].OnTrue, args->filter[i].OnFalse, args->filter[i].comp, args->filter[i].fname);
if ( args->filter[i].OnTrue > (memblocks * MAXBLOCKS) || args->filter[i].OnFalse > (memblocks * MAXBLOCKS) ) {
fprintf(stderr, "Tree pointer out of range for index %u. *** ABORT ***\n", i);
exit(255);
}
if ( args->filter[i].data ) {
if ( args->filter[i].comp == CMP_IPLIST ) {
struct IPListNode *node;
RB_FOREACH(node, IPtree, args->filter[i].data) {
printf("value: %.16llx %.16llx mask: %.16llx %.16llx\n",
(unsigned long long)node->ip[0], (unsigned long long)node->ip[1],
(unsigned long long)node->mask[0], (unsigned long long)node->mask[1]);
}
} else if ( args->filter[i].comp == CMP_ULLIST ) {
struct ULongListNode *node;
RB_FOREACH(node, ULongtree, args->filter[i].data) {
printf("%.16llx \n", (unsigned long long)node->value);
}
} else
printf("Error comp: %i\n", args->filter[i].comp);
}
printf("\tBlocks: ");
for ( j=0; j<args->filter[i].numblocks; j++ )
printf("%i ", args->filter[i].blocklist[j]);
printf("\n");
}
printf("NumBlocks: %i\n", NumBlocks - 1);
for ( i=0; i<NumIdents; i++ ) {
printf("Ident %i: %s\n", i, IdentList[i]);
}
} /* End of DumpList */
/* fast filter engine */
int RunFilter(FilterEngine_data_t *args) {
uint32_t index, offset;
int evaluate, invert;
index = args->StartNode;
evaluate = 0;
invert = 0;
while ( index ) {
offset = args->filter[index].offset;
invert = args->filter[index].invert;
evaluate = ( args->nfrecord[offset] & args->filter[index].mask ) == args->filter[index].value;
index = evaluate ? args->filter[index].OnTrue : args->filter[index].OnFalse;
}
return invert ? !evaluate : evaluate;
} /* End of RunFilter */
/* extended filter engine */
int RunExtendedFilter(FilterEngine_data_t *args) {
uint32_t index, offset;
uint64_t comp_value[2];
int evaluate, invert;
index = args->StartNode;
evaluate = 0;
invert = 0;
while ( index ) {
offset = args->filter[index].offset;
invert = args->filter[index].invert;
comp_value[0] = args->nfrecord[offset] & args->filter[index].mask;
comp_value[1] = args->filter[index].value;
if (args->filter[index].function != NULL)
args->filter[index].function(args->nfrecord, comp_value);
switch (args->filter[index].comp) {
case CMP_EQ:
evaluate = comp_value[0] == comp_value[1];
break;
case CMP_GT:
evaluate = comp_value[0] > comp_value[1];
break;
case CMP_LT:
evaluate = comp_value[0] < comp_value[1];
break;
case CMP_IDENT:
evaluate = strncmp(CurrentIdent, args->IdentList[comp_value[1]], IDENTLEN) == 0 ;
break;
case CMP_FLAGS:
if ( invert )
evaluate = comp_value[0] > 0;
else
evaluate = comp_value[0] == comp_value[1];
break;
case CMP_IPLIST: {
struct IPListNode find;
find.ip[0] = args->nfrecord[offset];
find.ip[1] = args->nfrecord[offset+1];
find.mask[0] = 0xffffffffffffffffLL;
find.mask[1] = 0xffffffffffffffffLL;
evaluate = RB_FIND(IPtree, args->filter[index].data, &find) != NULL; }
break;
case CMP_ULLIST: {
struct ULongListNode find;
find.value = comp_value[0];
evaluate = RB_FIND(ULongtree, args->filter[index].data, &find ) != NULL; }
break;
}
index = evaluate ? args->filter[index].OnTrue : args->filter[index].OnFalse;
}
return invert ? !evaluate : evaluate;
} /* End of RunExtendedFilter */
uint32_t AddIdent(char *Ident) {
uint32_t num;
if ( MaxIdents == 0 ) {
// allocate first array block
MaxIdents = IdentNumBlockSize;
IdentList = (char **)malloc( MaxIdents * sizeof(char *));
if ( !IdentList ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(254);
}
memset((void *)IdentList, 0, MaxIdents * sizeof(char *));
NumIdents = 0;
} else if ( NumIdents == MaxIdents ) {
// extend array block
MaxIdents += IdentNumBlockSize;
IdentList = realloc((void *)IdentList, MaxIdents * sizeof(char *));
if ( !IdentList ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(254);
}
}
num = NumIdents++;
IdentList[num] = strdup(Ident);
if ( !IdentList[num] ) {
fprintf(stderr, "Memory allocation error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(254);
}
return num;
} // End of AddIdent
/* record processing functions */
static inline void duration_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
/* duration in msec */
comp_values[0] = 1000*(record->last - record->first) + record->msec_last - record->msec_first;
} // End of duration_function
static inline void pps_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
uint64_t duration;
/* duration in msec */
duration = 1000*(record->last - record->first) + record->msec_last - record->msec_first;
if ( duration == 0 )
comp_values[0] = 0;
else
comp_values[0] = ( 1000LL * record->dPkts ) / duration;
} // End of pps_function
static inline void bps_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
uint64_t duration;
/* duration in msec */
duration = 1000*(record->last - record->first) + record->msec_last - record->msec_first;
if ( duration == 0 )
comp_values[0] = 0;
else
comp_values[0] = ( 8000LL * record->dOctets ) / duration; /* 8 bits per Octet - x 1000 for msec */
} // End of bps_function
static inline void bpp_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
comp_values[0] = record->dPkts ? record->dOctets / record->dPkts : 0;
} // End of bpp_function
static inline void mpls_eos_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
int i;
// search for end of MPLS stack label
for (i=0; i<10; i++ ) {
if ( record->mpls_label[i] & 1 ) {
// End of stack found -> mask exp and eos bits
comp_values[0] = record->mpls_label[i] & 0x00FFFFF0;
return;
}
}
// trick filter to fail with an invalid mpls label value
comp_values[0] = 0xFF000000;
} // End of mpls_eos_function
static inline void mpls_any_function(uint64_t *record_data, uint64_t *comp_values) {
master_record_t *record = (master_record_t *)record_data;
int i;
// search for end of MPLS stack label
for (i=0; i<10; i++ ) {
if ( (record->mpls_label[i] & 1) == 1 ) {
// End of stack found -> mask exp and eos bits
comp_values[0] = record->mpls_label[i] & 0x00FFFFF0;
return;
}
}
// trick filter to fail with an invalid mpls label value
comp_values[0] = 0xFF000000;
} // End of mpls_eos_function
static inline void pblock_function(uint64_t *record_data, uint64_t *comp_values) {
#ifdef NSEL
master_record_t *record = (master_record_t *)record_data;
comp_values[0] = comp_values[0] >> comp_values[1];
if ( (comp_values[0] >= record->block_start) && (comp_values[0] <= record->block_end) ) {
comp_values[1] = comp_values[0];
} else {
// force "not equal"
comp_values[1] = comp_values[0] + 1;
}
#else
comp_values[1] = 0;
#endif
} // End of pblock_function

178
bin/nftree.h Normal file
View File

@ -0,0 +1,178 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nftree.h 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
#ifndef _NFTREE_H
#define _NFTREE_H 1
/*
* type definitions for nf tree
*/
typedef void (*flow_proc_t)(uint64_t *, uint64_t *);
typedef struct FilterBlock {
/* Filter specific data */
uint32_t offset;
uint64_t mask;
uint64_t value;
/* Internal block info for tree setup */
uint32_t superblock; /* Index of superblock */
uint32_t *blocklist; /* index array of blocks, belonging to
this superblock */
uint32_t numblocks; /* number of blocks in blocklist */
uint32_t OnTrue, OnFalse; /* Jump Index for tree */
int16_t invert; /* Invert result of test */
uint16_t comp; /* comperator */
flow_proc_t function; /* function for flow processing */
char *fname; /* ascii function name */
void *data; /* any additional data for this block */
} FilterBlock_t;
typedef struct FilterEngine_data_s {
FilterBlock_t *filter;
uint32_t StartNode;
uint32_t Extended;
char **IdentList;
uint64_t *nfrecord;
int (*FilterEngine)(struct FilterEngine_data_s *);
} FilterEngine_data_t;
/*
* Definitions
*/
enum { CMP_EQ = 0, CMP_GT, CMP_LT, CMP_IDENT, CMP_FLAGS, CMP_IPLIST, CMP_ULLIST };
/*
* filter functions:
* For some filter functions, netflow records need to be processed first in order to filter them
* This involves all data not directly available in the netflow record, such as packets per second etc.
* Filter speed is a bit slower due to extra netflow processsing
* The sequence of the enum values must correspond with the entries in the flow_procs array
*/
enum { FUNC_NONE = 0, /* no function - just plain filtering - just to be complete here */
FUNC_PPS, /* function code for pps ( packet per second ) filter function */
FUNC_BPS, /* function code for bps ( bits per second ) filter function */
FUNC_BPP, /* function code for bpp ( bytes per packet ) filter function */
FUNC_DURATION, /* function code for duration ( in miliseconds ) filter function */
FUNC_MPLS_EOS, /* function code for matching End of MPLS Stack label */
FUNC_MPLS_ANY, /* function code for matching any MPLS label */
FUNC_PBLOCK /* function code for matching ports against pblock start */
};
/*
* Tree type defs
*/
/* Definition of the IP list node */
struct IPListNode {
RB_ENTRY(IPListNode) entry;
uint64_t ip[2];
uint64_t mask[2];
};
/* Definition of the port/AS list node */
struct ULongListNode {
RB_ENTRY(ULongListNode) entry;
uint64_t value;
};
/*
* Filter Engine Functions
*/
int RunFilter(FilterEngine_data_t *args);
int RunExtendedFilter(FilterEngine_data_t *args);
/*
* For testing purpose only
*/
int nblocks(void);
/*
* Initialize globals
*/
void InitTree(void);
/*
* Returns the current Filter Tree
*/
FilterEngine_data_t *CompileFilter(char *FilterSyntax);
/*
* Clear Filter
*/
void ClearFilter(void);
/*
* Returns next free slot in blocklist
*/
uint32_t NewBlock(uint32_t offset, uint64_t mask, uint64_t value, uint16_t comp, uint32_t function, void *data);
/*
* Connects the to blocks b1 and b2 ( AND ) and returns index of superblock
*/
uint32_t Connect_AND(uint32_t b1, uint32_t b2);
/*
* Connects the to blocks b1 and b2 ( OR ) and returns index of superblock
*/
uint32_t Connect_OR(uint32_t b1, uint32_t b2);
/*
* Inverts OnTrue and OnFalse
*/
uint32_t Invert(uint32_t a );
/*
* Add Ident to Identlist
*/
uint32_t AddIdent(char *Ident);
/*
* Dump Filterlist
*/
void DumpList(FilterEngine_data_t *args);
/*
* Prints info while filer is running
*/
int RunDebugFilter(uint32_t *block);
#endif //_NFTREE_H

635
bin/nfx.c Executable file
View File

@ -0,0 +1,635 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfx.c 58 2010-02-26 12:26:07Z haag $
*
* $LastChangedRevision: 58 $
*
*/
#include "config.h"
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>
#include <ctype.h>
#include <string.h>
#include <stdarg.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#include "nf_common.h"
#include "nffile.h"
#include "util.h"
#include "nfx.h"
/* global vars */
/*
* see nffile.h for detailed extension description
*/
extension_descriptor_t extension_descriptor[] = {
// fill indices 0 - 3
{ COMMON_BLOCK_ID, 0, 0, 1, "Required extension: Common record"},
{ EX_IPv4v6, 0, 0, 1, "Required extension: IPv4/IPv6 src/dst address"},
{ EX_PACKET_4_8, 0, 0, 1, "Required extension: 4/8 byte input packets"},
{ EX_BYTE_4_8, 0, 0, 1, "Required extension: 4/8 byte input bytes"},
// the optional extension
{ EX_IO_SNMP_2, 4, 1, 1, "2 byte input/output interface index"},
{ EX_IO_SNMP_4, 8, 1, 1, "4 byte input/output interface index"},
{ EX_AS_2, 4, 2, 1, "2 byte src/dst AS number"},
{ EX_AS_4, 8, 2, 1, "4 byte src/dst AS number"},
{ EX_MULIPLE, 4, 3, 0, "dst tos, direction, src/dst mask"},
{ EX_NEXT_HOP_v4, 4, 4, 0, "IPv4 next hop"},
{ EX_NEXT_HOP_v6, 16, 4, 0, "IPv6 next hop"},
{ EX_NEXT_HOP_BGP_v4, 4, 5, 0, "IPv4 BGP next IP"},
{ EX_NEXT_HOP_BGP_v6, 16, 5, 0, "IPv6 BGP next IP"},
{ EX_VLAN, 4, 6, 0, "src/dst vlan id"},
{ EX_OUT_PKG_4, 4, 7, 0, "4 byte output packets"},
{ EX_OUT_PKG_8, 8, 7, 0, "8 byte output packets"},
{ EX_OUT_BYTES_4, 4, 8, 0, "4 byte output bytes"},
{ EX_OUT_BYTES_8, 8, 8, 0, "8 byte output bytes"},
{ EX_AGGR_FLOWS_4, 4, 9, 0, "4 byte aggregated flows"},
{ EX_AGGR_FLOWS_8, 8, 9, 0, "8 byte aggregated flows"},
{ EX_MAC_1, 16, 10, 0, "in src/out dst mac address"},
{ EX_MAC_2, 16, 11, 0, "in dst/out src mac address"},
{ EX_MPLS, 40, 12, 0, "MPLS Labels"},
{ EX_ROUTER_IP_v4, 4, 13, 0, "IPv4 router IP addr"},
{ EX_ROUTER_IP_v6, 16, 13, 0, "IPv6 router IP addr"},
{ EX_ROUTER_ID, 4, 14, 0, "router ID"},
{ EX_BGPADJ, 8, 15, 0, "BGP adjacent prev/next AS"},
{ EX_RECEIVED, 8, 16, 0, "time packet received"},
// reserved for more v9/IPFIX
{ EX_RESERVED_1, 0, 0, 0, NULL},
{ EX_RESERVED_2, 0, 0, 0, NULL},
{ EX_RESERVED_3, 0, 0, 0, NULL},
{ EX_RESERVED_4, 0, 0, 0, NULL},
{ EX_RESERVED_5, 0, 0, 0, NULL},
{ EX_RESERVED_6, 0, 0, 0, NULL},
{ EX_RESERVED_7, 0, 0, 0, NULL},
{ EX_RESERVED_8, 0, 0, 0, NULL},
{ EX_RESERVED_9, 0, 0, 0, NULL},
// ASA - Network Security Event Logging NSEL extensions
{ EX_NSEL_COMMON, 20, 26, 0, "NSEL Common block"},
{ EX_NSEL_XLATE_PORTS, 4, 27, 0, "NSEL xlate ports"},
{ EX_NSEL_XLATE_IP_v4, 8, 28, 0, "NSEL xlate IPv4 addr"},
{ EX_NSEL_XLATE_IP_v6, 32, 28, 0, "NSEL xlate IPv6 addr"},
{ EX_NSEL_ACL, 24, 29, 0, "NSEL ACL ingress/egress acl ID"},
{ EX_NSEL_USER, 24, 30, 0, "NSEL username"},
{ EX_NSEL_USER_MAX, 72, 30, 0, "NSEL max username"},
{ EX_NSEL_RESERVED, 0, 0, 0, NULL},
// nprobe extensions
{ EX_LATENCY, 24, 64, 0, "nprobe latency"},
// NAT - Network Event Logging
{ EX_NEL_COMMON, 12, 31, 0, "NEL Common block"},
{ EX_NEL_GLOBAL_IP_v4, 0, 0, 0, "Compat NEL IPv4"},
{ EX_PORT_BLOCK_ALLOC, 8, 32, 0, "NAT Port Block Allocation"},
{ EX_NEL_RESERVED_1, 0, 0, 0, NULL},
// last entry
{ 0, 0, 0, 0, NULL }
};
uint32_t Max_num_extensions;
void FixExtensionMap(extension_map_t *map);
extension_map_list_t *InitExtensionMaps(int AllocateList) {
extension_map_list_t *list = NULL;
int i;
if ( AllocateList ) {
list = (extension_map_list_t *)calloc(1, sizeof(extension_map_list_t));
if ( !list ) {
LogError("calloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
list->last_map = &list->map_list;
}
Max_num_extensions = 0;
i = 1;
while ( extension_descriptor[i++].id ) {
Max_num_extensions++;
}
#ifdef DEVEL
i = 1;
while ( extension_descriptor[i].id ) {
if ( extension_descriptor[i].id != i ) {
printf("*** ERROR *** Init extension_descriptors at index %i: ID: %i, %s\n",
i, extension_descriptor[i].id, extension_descriptor[i].description);
}
i++;
}
#endif
return list;
} // End of InitExtensionMaps
void FreeExtensionMaps(extension_map_list_t *extension_map_list) {
extension_info_t *l;
if ( extension_map_list == NULL )
return;
// free all extension infos
l = extension_map_list->map_list;
while ( l ) {
extension_info_t *tmp = l;
l = l->next;
free(tmp->map);
free(tmp);
}
free(extension_map_list);
} // End of FreeExtensionMaps
int Insert_Extension_Map(extension_map_list_t *extension_map_list, extension_map_t *map) {
extension_info_t *l;
uint16_t map_id;
map_id = map->map_id == INIT_ID ? 0 : map->map_id & EXTENSION_MAP_MASK;
map->map_id = map_id;
dbg_printf("Insert Extension Map:\n");
#ifdef DEVEL
PrintExtensionMap(map);
#endif
// is this slot free
if ( extension_map_list->slot[map_id] ) {
int i;
dbg_printf("Extension info in slot %d already exists: 0x%llu\n", map_id, (long long unsigned)extension_map_list->slot[map_id]);
// no - check if same map already in slot
if ( extension_map_list->slot[map_id]->map->size == map->size ) {
// existing map and new map have the same size
dbg_printf("New map has same size, as existing:\n");
// we must compare the maps
i = 0;
while ( extension_map_list->slot[map_id]->map->ex_id[i] &&
(extension_map_list->slot[map_id]->map->ex_id[i] == map->ex_id[i]) )
i++;
// if last entry == 0 => last map entry => maps are the same
if ( extension_map_list->slot[map_id]->map->ex_id[i] == 0 ) {
dbg_printf("Same map => nothing to do\n");
// same map
return 0;
}
}
dbg_printf("Different map => continue\n");
}
#ifdef DEVEL
else
printf("Extension info in slot %d free\n", map_id);
#endif
dbg_printf("Search if extension info exists in extension page_list\n");
// new map is different but has same id - search for map in page list
for ( l = extension_map_list->map_list ; l != NULL ; l = l->next) {
int i = 0;
dbg_printf("Check map: %u\n", l->map->map_id);
if ( l->map->size == map->size && ( l->map->extension_size == map->extension_size ) ) {
while ( (l->map->ex_id[i] || map->ex_id[i]) && (l->map->ex_id[i] == map->ex_id[i]) )
i++;
if ( l->map->ex_id[i] == 0 ) {
dbg_printf("Map found: 0x%llu ID: %u\n", (long long unsigned)l, l->map->map_id);
break;
}
}
}
// if found l is our extension
if ( !l ) {
// no extension found in page_list
dbg_printf("Map not found in extension page list\n");
l = (extension_info_t *)malloc(sizeof(extension_info_t));
if ( !l ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
l->ref_count = 0;
l->next = NULL;
memset((void *)&l->master_record, 0, sizeof(master_record_t));
l->map = (extension_map_t *)malloc((ssize_t)map->size);
if ( !l->map ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
memcpy((void *)l->map, (void *)map, map->size);
// append new extension to list
*(extension_map_list->last_map) = l;
extension_map_list->last_map = &l->next;
// Sanity check
FixExtensionMap(map);
}
// l is now our valid extension
dbg_printf("Insert extension into slot %i: 0x%llu\n\n", map_id, (long long unsigned)l);
// remove map from lookup list, if it exists
if ( extension_map_list->slot[map_id] )
extension_map_list->slot[map_id]->map->map_id = 0;
// place existing extension_info into lookup list
extension_map_list->slot[map_id] = l;
l->map->map_id = map_id;
if ( map_id > extension_map_list->max_used ) {
extension_map_list->max_used = map_id;
}
// extension changed
return 1;
} // End of Insert_Extension_Map
void PackExtensionMapList(extension_map_list_t *extension_map_list) {
extension_info_t *l;
int i, free_slot;
dbg_printf("Pack extensions maps\n");
// compact extension map list - close gaps
// clear current list
for ( i=0; i <= extension_map_list->max_used; i++ ) {
extension_map_list->slot[i] = NULL;
}
// hangle though list
free_slot = 0;
l = extension_map_list->map_list;
while ( l ) {
dbg_printf("Check extension ref count: %u -> ", l->ref_count);
if ( l->ref_count ) {
// extension is referenced - insert into slot
dbg_printf("slot %u\n", free_slot);
extension_map_list->slot[free_slot] = l;
l->map->map_id = free_slot++;
l = l->next;
} else {
// extension can be removed - not referenced
l = l->next;
dbg_printf("Skipped\n");
}
if ( free_slot == MAX_EXTENSION_MAPS ) {
fprintf(stderr, "Critical error in %s line %d: %s\n", __FILE__, __LINE__, "Out of extension slots!" );
exit(255);
}
}
// this points to the next free slot
extension_map_list->max_used = free_slot > 0 ? free_slot - 1 : 0;
dbg_printf("Packed maps: %i\n", free_slot);
#ifdef DEVEL
// Check maps
i = 0;
while ( extension_map_list->slot[i] != NULL && i < MAX_EXTENSION_MAPS ) {
if ( extension_map_list->slot[i]->map->map_id != i )
printf("*** Map ID missmatch in slot: %i, id: %u\n", i, extension_map_list->slot[i]->map->map_id);
i++;
}
#endif
} // End of PackExtensionMapList
void SetupExtensionDescriptors(char *options) {
int i, *mask;
char *p, *q, *s;
mask = (int *)calloc(65536, sizeof(int));
if ( !mask ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
s = (char *)malloc(strlen(options)+1);
if ( !s ) {
fprintf(stderr, "malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
exit(255);
}
q = s;
*q = '\0';
p = options;
while ( *p ) {
if ( !isspace(*p) )
*q++ = *p;
p++;
}
*q = '\0';
p = s;
while ( p && *p ) {
int sign;
q = strchr(p, ',');
if ( q )
*q++ = '\0';
// get possible sign
sign = 1;
if ( *p == '-' ) {
sign = -1;
p++;
}
if ( *p == '+' ) {
sign = 1;
p++;
}
if ( strcmp(p, "all") == 0 ) {
for (i=4; extension_descriptor[i].id; i++ )
if ( extension_descriptor[i].description )
extension_descriptor[i].enabled = sign == 1 ? 1 : 0;
} else if ( strcmp(p, "nsel") == 0 ) {
extension_descriptor[EX_IO_SNMP_2].enabled = 0;
extension_descriptor[EX_IO_SNMP_4].enabled = 0;
extension_descriptor[EX_OUT_BYTES_4].enabled = 1;
extension_descriptor[EX_OUT_BYTES_8].enabled = 1;
extension_descriptor[EX_NSEL_COMMON].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_PORTS].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_IP_v4].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_IP_v6].enabled = 1;
extension_descriptor[EX_NSEL_ACL].enabled = 1;
extension_descriptor[EX_NSEL_USER].enabled = 1;
extension_descriptor[EX_NSEL_USER_MAX].enabled = 1;
} else if ( strcmp(p, "nel") == 0 ) {
extension_descriptor[EX_NEL_COMMON].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_PORTS].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_IP_v4].enabled = 1;
extension_descriptor[EX_NSEL_XLATE_IP_v6].enabled = 1;
} else {
switch ( *p ) {
case '\0':
fprintf(stderr, "Extension format error: Unexpected end of format.\n");
exit(255);
break;
case '*':
for (i=4; extension_descriptor[i].id; i++ )
if ( extension_descriptor[i].description )
extension_descriptor[i].enabled = sign == 1 ? 1 : 0;
break;
default: {
int i = strtol(p, NULL, 10);
if ( i == 0 ) {
fprintf(stderr, "Extension format error: Unexpected string: %s.\n", p);
exit(255);
}
if ( i > 65535 ) {
fprintf(stderr, "Extension format error: Invalid extension: %i\n", i);
exit(255);
}
mask[i] = sign;
}
}
}
p = q;
}
for (i=4; extension_descriptor[i].id; i++ ) {
int ui = extension_descriptor[i].user_index;
// Skip reserved extensions
if ( !extension_descriptor[i].description )
continue;
// mask[ui] == 0 means no input from user -> default behaviour or already overwritten by '*'
if ( mask[ui] < 0 ) {
extension_descriptor[i].enabled = 0;
}
if ( mask[ui] > 0 ) {
extension_descriptor[i].enabled = 1;
}
if ( extension_descriptor[i].enabled ) {
dbg_printf("Add extension: %s\n", extension_descriptor[i].description);
LogInfo("Add extension: %s", extension_descriptor[i].description);
}
}
free(mask);
} // End of SetupExtensionDescriptors
void PrintExtensionMap(extension_map_t *map) {
int i;
printf("Extension Map:\n");
printf(" Map ID = %u\n", map->map_id);
printf(" Map Size = %u\n", map->size);
printf(" Ext Size = %u\n", map->extension_size);
i=0;
while (map->ex_id[i]) {
int id = map->ex_id[i++];
printf(" ID %3i, ext %3u = %s\n", extension_descriptor[id].user_index, id, extension_descriptor[id].description );
}
printf("\n");
} // End of PrintExtensionMap
int VerifyExtensionMap(extension_map_t *map) {
int i, failed, extension_size, max_elements;
failed = 0;
if (( map->size & 0x3 ) != 0 ) {
printf("Verify map id %i: WARNING: map size %i not aligned!\n", map->map_id, map->size);
failed = 1;
}
if ( ((int)map->size - (int)sizeof(extension_map_t)) <= 0 ) {
printf("Verify map id %i: ERROR: map size %i too small!\n", map->map_id, map->size);
failed = 1;
return 0;
}
max_elements = (map->size - sizeof(extension_map_t)) / sizeof(uint16_t);
extension_size = 0;
i=0;
while (map->ex_id[i] && i <= max_elements) {
int id = map->ex_id[i];
if ( id > Max_num_extensions ) {
printf("Verify map id %i: ERROR: element id %i out of range [%i]!\n", map->map_id, id, Max_num_extensions);
failed = 1;
}
extension_size += extension_descriptor[id].size;
i++;
}
if ( (extension_size != map->extension_size ) ) {
printf("Verify map id %i: ERROR extension size: Expected %i, Map reports: %i!\n", map->map_id,
extension_size, map->extension_size);
failed = 1;
}
if ( (i != max_elements ) && ((max_elements-i) != 1) ) {
// off by 1 is the opt alignment
printf("Verify map id %i: ERROR: Expected %i elements in map, but found %i!\n", map->map_id, max_elements, i);
failed = 1;
}
return failed;
} // End of VerifyExtensionMap
/*
* Sanity check of map
*/
void FixExtensionMap(extension_map_t *map) {
int i, extension_size, max_elements;
if (( map->size & 0x3 ) != 0 ) {
printf("PANIC! - Verify map id %i: WARNING: map size %i not aligned!\n", map->map_id, map->size);
exit(255);
}
if ( ((int)map->size - (int)sizeof(extension_map_t)) <= 0 ) {
printf("PANIC! - Verify map id %i: ERROR: map size %i too small!\n", map->map_id, map->size);
exit(255);
}
max_elements = (map->size - sizeof(extension_map_t)) / sizeof(uint16_t);
extension_size = 0;
i=0;
while (map->ex_id[i] && i <= max_elements) {
int id = map->ex_id[i];
if ( id > Max_num_extensions ) {
printf("PANIC! - Verify map id %i: ERROR: element id %i out of range [%i]!\n", map->map_id, id, Max_num_extensions);
}
extension_size += extension_descriptor[id].size;
i++;
}
// silently fix extension size bug of nfdump <= 1.6.2 ..
if ( (extension_size != map->extension_size ) ) {
#ifdef DEVEL
printf("FixExtension map extension size from %i to %i\n", map->extension_size, extension_size);
#endif
map->extension_size = extension_size;
}
if ( (i != max_elements ) && ((max_elements-i) != 1) ) {
// off by 1 is the opt alignment
printf("Verify map id %i: ERROR: Expected %i elements in map, but found %i!\n", map->map_id, max_elements, i);
}
} // End of FixExtensionMap
void DumpExMaps(char *filename) {
int done;
nffile_t *nffile;
common_record_t *flow_record;
uint32_t skipped_blocks;
uint64_t total_bytes;
printf("\nDump all extension maps:\n");
printf("========================\n");
nffile = OpenFile(filename, NULL);
if ( !nffile ) {
return;
}
total_bytes = 0;
skipped_blocks = 0;
done = 0;
while ( !done ) {
int i, ret;
// get next data block from file
ret = ReadBlock(nffile);
switch (ret) {
case NF_CORRUPT:
case NF_ERROR:
if ( ret == NF_CORRUPT )
LogError("Corrupt data file '%s': '%s'\n",filename);
else
LogError("Read error in file '%s': %s\n",filename, strerror(errno) );
done = 1;
continue;
break;
// fall through - get next file in chain
case NF_EOF:
done = 1;
continue;
break;
default:
// successfully read block
total_bytes += ret;
}
if ( nffile->block_header->id != DATA_BLOCK_TYPE_2 ) {
skipped_blocks++;
continue;
}
// block type = 2
flow_record = (common_record_t *)nffile->buff_ptr;
for ( i=0; i < nffile->block_header->NumRecords; i++ ) {
if ( flow_record->type == ExtensionMapType ) {
extension_map_t *map = (extension_map_t *)flow_record;
VerifyExtensionMap(map);
PrintExtensionMap(map);
}
// Advance pointer by number of bytes for netflow record
flow_record = (common_record_t *)((pointer_addr_t)flow_record + flow_record->size);
}
}
CloseFile(nffile);
DisposeFile(nffile);
} // End of DumpExMaps

87
bin/nfx.h Executable file
View File

@ -0,0 +1,87 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: nfx.h 48 2010-01-02 08:06:27Z haag $
*
* $LastChangedRevision: 48 $
*
*/
#ifndef _NFX_H
#define _NFX_H 1
// MAX_EXTENSION_MAPS must be a power of 2
#define MAX_EXTENSION_MAPS 65536
#define EXTENSION_MAP_MASK (MAX_EXTENSION_MAPS-1)
typedef struct extension_descriptor_s {
uint16_t id; // id number
uint16_t size; // number of bytes
uint32_t user_index; // index specified by the user to enable this extension
uint32_t enabled; // extension is enabled or not
char *description;
} extension_descriptor_t;
typedef struct extension_info_s {
struct extension_info_s *next;
extension_map_t *map;
uint32_t ref_count;
uint32_t *offset_cache;
master_record_t master_record;
} extension_info_t;
typedef struct extension_map_list_s {
extension_info_t *slot[MAX_EXTENSION_MAPS];
extension_info_t *map_list;
extension_info_t **last_map;
uint32_t max_used;
} extension_map_list_t;
#define NEEDS_EXTENSION_LIST 1
#define NO_EXTENSION_LIST 0
extension_map_list_t *InitExtensionMaps(int AllocateList);
void FreeExtensionMaps(extension_map_list_t *extension_map_list);
void PackExtensionMapList(extension_map_list_t *extension_map_list);
int Insert_Extension_Map(extension_map_list_t *extension_map_list, extension_map_t *map);
void SetupExtensionDescriptors(char *options);
void PrintExtensionMap(extension_map_t *map);
int VerifyExtensionMap(extension_map_t *map);
void DumpExMaps(char *filename);
#endif //_NFX_H

112
bin/nfxstat.c Normal file
View File

@ -0,0 +1,112 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2011, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: nfxstat.c 25 2011-02-26 13:22:31Z peter $
*
* $LastChangedRevision: 25 $
*
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <unistd.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifndef DEVEL
# define dbg_printf(...) /* printf(__VA_ARGS__) */
#else
# define dbg_printf(...) printf(__VA_ARGS__)
#endif
#include "util.h"
#include "nf_common.h"
#include "nffile.h"
#include "nfxstat.h"
xstat_t *InitXStat(nffile_t *nffile) {
xstat_t *xs;
size_t block_size;
// extended by the record link 2*data[4], but does no harm
block_size = sizeof(xstat_t) + sizeof(data_block_header_t) + sizeof(flow_port_histogram_t) + sizeof(flow_bpp_histogram_t);
xs = (xstat_t *)malloc(block_size);
if ( !xs ) {
LogError("malloc() error in %s line %d: %s\n", __FILE__, __LINE__, strerror(errno) );
return NULL;
}
xs->block_header = (data_block_header_t *)((pointer_addr_t)xs + sizeof(xstat_t));
// preset the fields
xs->block_header->NumRecords = 2; // 2 histogram records are included
xs->block_header->size = block_size;
xs->block_header->id = Large_BLOCK_Type;
xs->block_header->flags = 0;
xs->port_histogram = (flow_port_histogram_t *)((pointer_addr_t)xs + sizeof(xstat_t) + sizeof(data_block_header_t));
xs->bpp_histogram = (flow_bpp_histogram_t *)((pointer_addr_t)xs + sizeof(xstat_t) + sizeof(data_block_header_t) + sizeof(flow_port_histogram_t) - 4); // without link pointer data[4]
// XXX add catalog entry
// SetFlag(nffile->file_header->flags, FLAG_EXTENDED_STATS);
ResetPortHistogram(xs->port_histogram);
ResetBppHistogram(xs->bpp_histogram);
return xs;
} // End of InitXStat
void ResetPortHistogram(flow_port_histogram_t *port_histogram) {
memset((void *)port_histogram, 0, sizeof(flow_port_histogram_t) - 4); // without link pointer data[4]
port_histogram->record_header.type = PortHistogramType;
port_histogram->record_header.size = sizeof(flow_port_histogram_t) - 4;
} // End of ResetPortHistogram
void ResetBppHistogram(flow_bpp_histogram_t *bpp_histogram) {
memset((void *)bpp_histogram, 0, sizeof(bpp_histogram_t) - 4); // without link pointer data[4]
bpp_histogram->record_header.type = BppHistogramType;
bpp_histogram->record_header.size = sizeof(flow_port_histogram_t) - 4;
} // End of ResetBppHistogram

82
bin/nfxstat.h Normal file
View File

@ -0,0 +1,82 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2011, Peter Haag
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: peter $
*
* $Id: nfxstat.h 25 2011-02-26 13:22:31Z peter $
*
* $LastChangedRevision: 25 $
*
*/
typedef struct port_histogram_s {
uint32_t count;
uint32_t port[65536];
} port_histogram_t;
typedef struct bpp_histogram_s {
uint32_t count;
#define MAX_BPP 9000
uint32_t bpp[MAX_BPP];
} bpp_histogram_t;
typedef struct flow_port_histogram_s {
// type = PortHistogramType
L_record_header_t record_header;
port_histogram_t src_tcp;
port_histogram_t dst_tcp;
port_histogram_t src_udp;
port_histogram_t dst_udp;
uint8_t data[4]; // .. more data below
} flow_port_histogram_t;
typedef struct flow_bpp_histogram_s {
// type = BppHistogramType
L_record_header_t record_header;
bpp_histogram_t tcp;
bpp_histogram_t udp;
uint8_t data[4]; // .. more data below
} flow_bpp_histogram_t;
// Extended stat record
typedef struct xstat_s {
data_block_header_t *block_header;
flow_port_histogram_t *port_histogram;
flow_bpp_histogram_t *bpp_histogram;
} xstat_t;
xstat_t *InitXStat(nffile_t *nffile);
void ResetPortHistogram(flow_port_histogram_t *port_histogram);
void ResetBppHistogram(flow_bpp_histogram_t *bpp_histogram);

229
bin/panonymizer.c Normal file
View File

@ -0,0 +1,229 @@
/*
* Copyright (c) 2014, Peter Haag
* Copyright (c) 2009, Peter Haag
* Copyright (c) 2004-2008, SWITCH - Teleinformatikdienste fuer Lehre und Forschung
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* * Neither the name of the author nor the names of its contributors may be
* used to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* $Author: haag $
*
* $Id: panonymizer.c 39 2009-11-25 08:11:15Z haag $
*
* $LastChangedRevision: 39 $
*
*/
/* Original disclaimer
* Atlanta, Georgia 30332.
* All Rights Reserved
*
* The following Software is posted on the Internet by the Georgia
* Tech Research Corporation (GTRC). It was developed by employees
* of the Georgia Institute of Technology in the College of Computing.
* GTRC hereby grants to the user a non-exclusive, royalty-free
* license to utilize such Software for the User's own purposes
* pursuant to the following conditions.
*
*
* THE SOFTWARE IS LICENSED ON AN "AS IS" BASIS. GTRC MAKES NO WARRANTY
* THAT ALL ERRORS CAN BE OR HAVE BEEN ELIMINATED FROM THE SOFTWARE.
* GTRC SHALL NOT BE RESPONSIBLE FOR LOSSES OF ANY KIND RESULTING FROM
* THE USE OF THE SOFTWARE AND ITS ACCOMPANYING DOCUMENTATION, AND CAN
* IN NO WAY PROVIDE COMPENSATION FOR ANY LOSSES SUSTAINED, INCLUDING
* BUT NOT LIMITED TO ANY OBLIGATION, LIABILITY, RIGHT, CLAIM OR REMEDY
* FOR TORT, OF FOR ANY ACTUAL OR ALLEGED INFRINGEMENT OF PATENTS, COPYRIGHTS,
* TRADE SECRETS, OR SIMILAR RIGHTS OF THIRD PARTIES, NOR ANY BUSINESS
* EXPENSE, MACHINE DOWNTIME, OR DAMAGES CAUSED LICENSEE BY ANY DEFICIENCY,
* DEFECT OR ERROR IN THE SOFTWARE OR MALFUNCTION THEREOF, NOR ANY
* INCIDENTAL OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED. GTRC DISCLAIMS
* ALL WARRANTIES, BOTH EXPRESS AND IMPLIED RESPECTING THE USE AND
* OPERATION OF THE SOFTWARE AND ANY ACCOMPANYING DOCUMENTATION,
* INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* PARTICULAR PURPOSE AND ANY IMPLIED WARRANTY ARISING FROM COURSE
* OF PERFORMANCE, COURSE OF DEALING OR USAGE OF TRADE. GTRC MAKES NO
* WARRANTY THAT THE SOFTWARE IS ADEQUATELY OR COMPLETELY DESCRIBED
* IN, OR BEHAVES IN ACCORDANCE WITH ANY OF THE ACCOMPANYING
* DOCUMENTATION. THE USER OF THE SOFTWARE IS EXPECTED TO MAKE THE FINAL
* EVALUATION OF THE SOFTWARE'S USEFULNESS IN USER'S OWN ENVIRONMENT.
*
*
* Package: Crypto-PAn 1.0
* File: panonymizer.cpp
* Last Update: April 17, 2002
* Author: Jinliang Fan
*
*/
#include "config.h"
#include <stdio.h>
#include <sys/types.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#include "panonymizer.h"
static uint8_t m_key[16]; //128 bit secret key
static uint8_t m_pad[16]; //128 bit secret pad
// Init
void PAnonymizer_Init(uint8_t * key) {
//initialize the 128-bit secret key.
memcpy(m_key, key, 16);
//initialize the Rijndael cipher.
Rijndael_init(ECB, Encrypt, key, Key16Bytes, NULL);
//initialize the 128-bit secret pad. The pad is encrypted before being used for padding.
Rijndael_blockEncrypt(key + 16, 128, m_pad);
}
int ParseCryptoPAnKey ( char *s, char *key ) {
int i, j;
char numstr[3];
uint32_t len = strlen(s);
if ( len < 32 || len > 66 ) {
fprintf(stderr, "*** CryptoPAnKey error: size: %u\n", len);
fprintf(stderr, "*** Need either a plain 32 char string, or a 32 byte hex key starting with 0x..\n");
return 0;
}
if ( strlen(s) == 32 ) {
// Key is a string
strncpy(key, s, 32);
return 1;
}
s[1] = tolower(s[1]);
numstr[2] = 0;
if ( strlen(s) == 66 && s[0] == '0' && s[1] == 'x' ) {
j = 2;
for ( i=0; i<32; i++ ) {
if ( !isxdigit((int)s[j]) || !isxdigit((int)s[j+1]) )
return 0;
numstr[0] = s[j++];
numstr[1] = s[j++];
key[i] = strtol(numstr, NULL, 16);
}
return 1;
}
// It's an invalid key
fprintf(stderr, "*** CryptoPAnKey error: size: %u\n", len);
fprintf(stderr, "*** Need either a plain 32 char string, or a 32 byte hex key starting with 0x..\n");
return 0;
} // End of ParseCryptoPAnKey
//Anonymization funtion
uint32_t anonymize(const uint32_t orig_addr) {
uint8_t rin_output[16];
uint8_t rin_input[16];
uint32_t result = 0;
uint32_t first4bytes_pad, first4bytes_input;
int pos;
memcpy(rin_input, m_pad, 16);
first4bytes_pad = (((uint32_t) m_pad[0]) << 24) + (((uint32_t) m_pad[1]) << 16) +
(((uint32_t) m_pad[2]) << 8) + (uint32_t) m_pad[3];
// For each prefixes with length from 0 to 31, generate a bit using the Rijndael cipher,
// which is used as a pseudorandom function here. The bits generated in every rounds
// are combineed into a pseudorandom one-time-pad.
for (pos = 0; pos <= 31 ; pos++) {
//Padding: The most significant pos bits are taken from orig_addr. The other 128-pos
//bits are taken from m_pad. The variables first4bytes_pad and first4bytes_input are used
//to handle the annoying byte order problem.
if (pos==0) {
first4bytes_input = first4bytes_pad;
}
else {
first4bytes_input = ((orig_addr >> (32-pos)) << (32-pos)) | ((first4bytes_pad<<pos) >> pos);
}
rin_input[0] = (uint8_t) (first4bytes_input >> 24);
rin_input[1] = (uint8_t) ((first4bytes_input << 8) >> 24);
rin_input[2] = (uint8_t) ((first4bytes_input << 16) >> 24);
rin_input[3] = (uint8_t) ((first4bytes_input << 24) >> 24);
//Encryption: The Rijndael cipher is used as pseudorandom function. During each
//round, only the first bit of rin_output is used.
Rijndael_blockEncrypt(rin_input, 128, rin_output);
//Combination: the bits are combined into a pseudorandom one-time-pad
result |= (rin_output[0] >> 7) << (31-pos);
}
//XOR the orginal address with the pseudorandom one-time-pad
return result ^ orig_addr;
}
/* little endian CPU's are boring! - but give it a try
* orig_addr is a ptr to memory, return by inet_pton for IPv6
* anon_addr return the result in the same order
*/
void anonymize_v6(const uint64_t orig_addr[2], uint64_t *anon_addr) {
uint8_t rin_output[16], *orig_bytes, *result;
uint8_t rin_input[16];
int pos, i, bit_num, left_byte;
anon_addr[0] = anon_addr[1] = 0;
result = (uint8_t *)anon_addr;
orig_bytes = (uint8_t *)orig_addr;
// For each prefixes with length from 0 to 127, generate a bit using the Rijndael cipher,
// which is used as a pseudorandom function here. The bits generated in every rounds
// are combineed into a pseudorandom one-time-pad.
for (pos = 0; pos <= 127 ; pos++) {
bit_num = pos & 0x7;
left_byte = (pos >> 3);
for ( i=0; i<left_byte; i++ ) {
rin_input[i] = orig_bytes[i];
}
rin_input[left_byte] = orig_bytes[left_byte] >> (7-bit_num) << (7-bit_num) | (m_pad[left_byte]<<bit_num) >> bit_num;
for ( i=left_byte+1; i<16; i++ ) {
rin_input[i] = m_pad[i];
}
//Encryption: The Rijndael cipher is used as pseudorandom function. During each
//round, only the first bit of rin_output is used.
Rijndael_blockEncrypt(rin_input, 128, rin_output);
//Combination: the bits are combined into a pseudorandom one-time-pad
result[left_byte] |= (rin_output[0] >> 7) << bit_num;
}
//XOR the orginal address with the pseudorandom one-time-pad
anon_addr[0] ^= orig_addr[0];
anon_addr[1] ^= orig_addr[1];
}

Some files were not shown because too many files have changed in this diff Show More