n2disk™

100 Gbit network traffic recorder with indexing capabilities

n2disk™ is a network traffic recorder application. With n2disk™ you can capture full-sized network packets at multi-Gigabit rate (above 10 Gigabit/s on adequate hardware) from a live network interface, and write them into files without any packet loss. n2disk™ has been designed to write files into disks for very long periods, you have to specify a maximum number of distinct file that may be written during the execution, and if n2disk™ reaches the maximum number of files, it will start recycling the files from the oldest one. This way you can have a complete view of the traffic for a fixed temporal window, knowing in advance the amount of disk space needed.
n2disk™ uses the industry standard PCAP file format to dump packets into files so the resulting output can be easily integrated with existing third party or even open/source analysis tools (e.g. Wireshark).
n2disk™ has been designed and developed mainly because most network security systems rely on capturing full-size packets, since any packets may have been responsible for the attack or could contain the problems that we are trying to find. Netflow information is more manageable and requires less disk space to be stored, but in some cases, like deep-packet-inspection analysis or controlled traffic regeneration, it is not useful.
n2disk™ can be effectively used to perform numerous activities, among these:

  • Off-line network packets analysis by feeding a specialized tools like Snort.
  • Reconstruct particular communication flows or network activities.
  • Reproduce the previous captured traffic to a different network interface.

Main n2disk™ Features

The current n2disk™ version is much more than a simple packet-to-disk application. Some of the n2disk™ features include:

    • Fully user configurable.
    • Use of the standard PCAP file format (regular and with nanoseconds).
    • Line rate 64 bytes packet to disk recording.
    • Support of Intel 1/10/40Gbit commodity adapters (Intel and Myricom) and FPGA-accelerated NICs (Accolade Technology, Napatech, and Silicom/Fiberblaze).
    • 40 Gbit continuous packet-to-disk with FPGA-accelerated NICs (and adequate storage subsystem).
    • BPF filters supports (using the same format as in the popular tcpdump tool) to filter out the unwanted network packets from the recording process.
    • Optimized BPF-like filters support, a faster replacement for BPF filters (a subset of the BPF syntax is supported), that can be used both in packet capture and post-capture filtering.
    • Multi-core support. n2disk™ has been designed with multicore architectures in mind. It uses at least 2 threads (one for the packet capture and one for the disk writing) and it is possible to further parallelize packet capture using multiple threads. The communication between threads has been carefully optimized.
    • PF_RING acceleration. n2disk™ exploit the packet capture acceleration offered both by standard PF_RING and PF_RING ZC.
    • Direct-IO disk access. n2disk™ uses the Direct IO access to the disks in order to obtain maximum disk-write throughput.
  • Real-Time indexing. n2disk™ is able to produce an index on-the-fly during packet capture. The index can be queried using a BPF-like syntax to quickly retrieve interesting packets in a specified time interval. Besides the per-dump-file index, n2disk™ can also produce a timeline, a way of keeping the whole captured traffic in chronological order. Using the utilities provided with n2disk™, it is possible to query the timeline for specific packets belonging to the whole dump set in a given time interval.
  • PCAP and index compression. n2disk™ can optionally compress on-the-fly both PCAP files and index, optimising I/O throughput and disk space.

Performance

n2disk™ has been designed to keep up with multi-Gigabit speeds on commodity hardware.

Packet Size (Bytes) n2disk™ Sustained Throughtput with no packet loss at 10 Gbit
fixed 64 Wire rate
fixed 128
fixed 512
random 64-1500

The table above shows the result of a worst-case performance test using the following system configuration:

  • OS: Ubuntu 16.04
  • CPU: Intel(R) Xeon(R) E5-1660 v3 @ 3.0GHz
  • Motherboard: Supermicro
  • Memory: 32 GB
  • Card: Intel PCIe X520 10 Gigabit
  • Disks: 8x 1TB 10K RPM SATA
  • Commands used:
    • n2disk -i zc:eth1 -o /storage/ -p 1024 -b 4096 -q 1 -C 4096 -S 0 -c 1 -w 2
Traffic Type Compression Rate Throughtput
Synthetic (64 bytes) 95% Wire rate
High-Frequency Trading 82%
Internet/GTP 6-10%

The table above shows the result of performance tests with indexing and PCAP compression enabled using the following system configuration:

  • OS: Ubuntu 16.04
  • CPU: Intel(R) Xeon(R) E5-1660 v3 @ 3.0GHz
  • Motherboard: Supermicro
  • Memory: 32 GB
  • Card: Intel PCIe X520 10 Gigabit
  • Disks: 8x 1TB 10K RPM SATA
  • Command used:
    • n2disk -i zc:eth1 -o /storage/ -b 4096 -C 4096 -p 1024 -g -s 1518 -M -I -A /storage/timeline/ -Z -S 0 -c 1 -z 2,3 -w 4 -m 100 -n 50 -H

User’s Guide

For all the n2disk™ configuration options and performance optimisation techniques, please refer to the  n2disk™ User’s Guide.

License

n2disk is distributed under the EULA and requires a license per system.

Operating Systems

Linux FreeBSD

Only n2disk 1/5 Gbit is supported on FreeBSD.

Get It

n2disk™ is available in three flavours. You can test it as binary package or get a permanent license. All Linux versions support Intel, Silicom, and Napatech NICs.
n2disk 10/40/100 Gbit also unlocks PF_RING FT for L7 filtering (no additional license required).

Version Max Dump Speed L7 Filtering Linux Unix / OSX
n2disk1g 1 Gigabit Supported (requires FT) Native PF_RING support. Basic libpcap-based packet capture.
Available on Request
n2disk5g 5 Gigabit Supported (requires FT) Enhanced PF_RING support
(i.e. full packet capture acceleration).		 Basic libpcap-based packet capture.
Available on Request
n2disk 10/40/100 Gigabit Supported (FT included) Multithreaded zero-copy packet capture. Not available.

NOTE

  • Test reports have been measured on Linux in the worst-case conditions (64 byte packets)
  • Dump speed depends on your disk setup and server being used.
  • You can use n2disk™ as software application or embedded on the nBox recorder.
  • Research and no-profict can have n2disk™ at no cost. Please contact us for details.