Flows Export

Using n2disk with other tools like ntopng is often convenient for for extending continuous traffic recording with traffic analysis/visualization. For this reason n2disk has been extended to be able to export statistics and flows metadata to other analysis tools through ZMQ.

For getting started with the integration and for learning more about the configuration please read the ntopng User’s Guide

The --zmq option should be used to instruct n2disk to export flows and statistics in TLV or JSON format over ZMQ.

Command line example:

n2disk -i eno1 -o /storage --zmq tcp://127.0.0.1:5556 -I -Z -z 0 --max-file-duration 10

In order to collect flows generated by n2disk using ntopng, please follow the same instructions required for using ntopng with nProbe running ntopng in collector mode (n2disk exports flows in probe bode, connecting to ntopng).

Example:

ntopng -i "tcp://*:5556c"

n2disk implements a basic template (what you usually define in nprobe with the -T option) that contains the following fields:

%SRC_VLAN %IN_SRC_MAC %IN_DST_MAC %IP_PROTOCOL_VERSION %DIRECTION
%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_ADDR
%INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES
%FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT
%TCP_FLAGS %PROTOCOL %SRC_TOS %L7_PROTO
%INPUT_SNMP %OUTPUT_SNMP %OBSERVATION_POINT_ID

It is possible to control flow expiration and export using the --lifetime-timeout and the --idle-timeout parameters.

Please note that some fields are populated depending on the configuration:

• %INPUT_SNMP %OUTPUT_SNMP %OBSERVATION_POINT_ID are available when device and port information are added to the packet as metadata by external packet brokers. For example Arista Metawatch devices export Metamako trailers encoding the device ID and port ID (add –hw-timestamp metawatch to the n2disk configuration to enable it).
• %L7_PROTO is available by default (no need to use -E 2 as in previous versions)

Please keep in mind that flows are computed as soon as a PCAP file has been filled up and ready to be written on disk, this means that if you have low traffic throughput and bug files, flows are exported with some delay (up to 10 minutes if you have timeline enabled and little traffic). In order to reduce this delay you can play with the --max-file-duration option to produce PCAP files more frequently.

Periodic stats are also exported, in JSON format, in addition to flow metadata. Example:

{
       "iface": {
               "name": "eth1",
               "speed": 1000,
               "ip": "10.0.0.2"
       },
       "probe": {
               "ip": "10.0.0.1"
       },
       "time" : 1568866123.456,
       "bytes": 3582434,
       "packets": 32141,
       "avg": {
               "bps": 1096415640,
               "pps": 207410
       },
         "drops" : {
               "export_queue_too_long": 0,
               "too_many_flows": 0
       },
         "timeout": {
               "lifetime": 120,
               "idle": 10
       },
         "zmq": {
               "num_flow_exports": 2431,
               "num_exporters": 1
       }
}

Where:

• iface reports information about the capture interface
• probe reports information about the machine (e.g. management IP)
• time is the current epoch timestamp
• bytes is the number of captured bytes
• packets is the number of captured packets
• avg is the average capture rate in bps and pps
• drops reports information about flow drops, based on the reason
• timeout reports information about flow export settings
• zmq reports statistics about flows exported on the ZMQ channel