PF_RING FT (Flow Table)

PF_RING FT implements an optimized flow table that can be used to keep track of flows and extract flow metadata including the L7 protocol thanks to the native integration with nDPI. Using PF_RING FT it is possible to write an event-driven flow processing application focusing on the actual flow processing rather then the flow table management and performance, in fact it provides many hooks that can be used to customize and extend the core table for building any type of flow analysis application, including probes, IDSs, IPSs.

_images/pfring_ft.png

API Overview

Designing and implementing a flow processing application on top of PF_RING FT is quite straightforward as it provides a clean API that can be used to do complex things in a few lines of code. The following code snippet shows how it is easy to capture traffic and export flow informations with PF_RING FT.

ft = pfring_ft_create_table(flags, 0, 0, 0);

pfring_ft_set_flow_export_callback(ft, processFlow, NULL);

while (1) {
  if (pfring_recv(pd, &buffer_p, 0, &hdr, 0) > 0)
    action = pfring_ft_process(ft, p, (pfring_ft_pcap_pkthdr *) h);
}

void processFlow(pfring_ft_flow *flow, void *user){
  pfring_ft_flow_key *k = pfring_ft_flow_get_key(flow);
  pfring_ft_flow_value *v = pfring_ft_flow_get_value(flow);
  /* flow export here with metadata in k and v */
}

The full example is available on github under examples_ft.

Please note that the FT API is capture-agnostic, this means that it is possible to use any capture framework for capturing raw traffic, in the example above we used PF_RING. When using PF_RING, there is no need to link additional libraries as PF_RING FT is already part of PF_RING, instead when using other framework, the libpfring_ft available under the libs folder need to be linked.

In order to write an application based on PF_RING FT, the first step is to create a pfring_ft_table object, which represents a flow table instance, this can be done calling pfring_ft_create_table:

pfring_ft_table *ft = pfring_ft_create_table(flags, 0, 0, 0);

The pfring_ft_create_table parameters include:

  • flags that can be used to enable optional features like L7 protocol detection based on nDPI (PFRING_FT_TABLE_FLAGS_DPI)
  • the maximum number of active flows the flow table should be able to handle (default: 500K)
  • the timeout for idle flows (default: 30 sec)
  • the maximum flow duration/lifetime (default: 120 sec)

As introduced before, the flow of an application designed on top of PF_RING FT is event-driven, before feeding the flow table with raw packets, the application can register for getting notified when there is a new event through hooks. Those events include:

  • New flow (pfring_ft_set_new_flow_callback)
  • Packet classified (pfring_ft_set_flow_packet_callback)
  • Flow expiration (pfring_ft_set_flow_export_callback or pfring_ft_set_flow_list_export_callback)

Registering for events notificaiton means setting a callback that will be called, for instance, when a flow is expired and should be processed (e.g. exported in case of a Netflow application):

pfring_ft_set_flow_export_callback(ft, processFlow, NULL);

In this example, processFlow is the callback that will be called when the event is triggered. This callback can access flow information including the flow key (5-tuple) and built-in flow data collected by FT (please note that the application can extend this data accessing custom metadata with the pfring_ft_flow_value.user pointer), in order to compute actions based on the flow status.

void processFlow(pfring_ft_flow *flow, void *user){
  pfring_ft_flow_value *v = pfring_ft_flow_get_value(flow);

  /* Example of printing the L7 protocol for this flow: */
  printf("l7: %s\n", pfring_ft_l7_protocol_name(ft, &v->l7_protocol, buf, sizeof(buf)));
}

Finally, we need to feed the flow table with raw packets, this is achieved calling pfring_ft_process in the main capture loop for every packet.

while (1) {
  if (pfring_recv(pd, &buffer_p, 0, &hdr, 0) > 0)
    action = pfring_ft_process(ft, p, (pfring_ft_pcap_pkthdr *) h);
}

The pfring_ft_process function returns an action (default/forward/discard) in case the flow has been marked by the application (according to custom policies) or by the built-in filtering engine as described in the following sections. This is where packet filtering should happen.

Before destroying the flow table with pfring_ft_destroy_table, it is recommended to call pfring_ft_flush to process all flows that are still active and have not been exported yet.

For detailed information please refer to the API documentation.

nDPI Integration

PF_RING FT is natively integrated with nDPI for providing L7 protocol information out of the box. The application itself does not need to deal with the nDPI library directly as everything happens behind the scenes. In order to get L7 information in the flow metadata in pfring_ft_flow_value.l7_protocol, all you need to do is to

  1. enable L7 detection through the PFRING_FT_TABLE_FLAGS_DPI flag:
ft = pfring_ft_create_table(PFRING_FT_TABLE_FLAGS_DPI);
  1. read the L7 protocol from pfring_ft_flow_value.l7_protocol

Note: since PF_RING 7.3 ndpi is installed as a dependency of pfring when installing from packages. If you are compiling from source code, or using an older version of PF_RING, you need to manually install the nDPI library from https://github.com/ntop/nDPI following the steps below:

git clone https://github.com/ntop/nDPI.git
cd nDPI
./autogen.sh
make && sudo make install

Note

If you are installing a stable version of PF_RING, you should also clone latest stable version of nDPI, to make sure that the API is in sync.

L7 Filtering and Shunting

PF_RING FT features a L7 filtering engine that can be used for filtering flows based on the application protocol, or can be extended with custom policies. This is usually the case for instance of IPSs, L7 firewalls, other inline applications. Using the built-in L7 filtering engine is possible by setting filtering rules through the API or through a configuration file. The API provides functions to filter or shunt traffic (specifying the number of packets for each flow that are allowed to pass before discarding them) based on the application protocol. Example:

pfring_ft_set_filter_protocol_by_name(ft, "UPnP", PFRING_FT_ACTION_DISCARD);
pfring_ft_set_shunt_protocol_by_name(ft, "SSH", 5);

It is also possible to specify filtering and shunting rules using a configuration file:

[shunt]
default = 10
tcp = 15
udp = 2
HTTP = 10

[filter]
YouTube = discard
Netflix = discard

Where:

  • supported entries for the shunt section include all the nDPI application protocols, tcp/udp for L4 protocols when there is no application protocol match, default for all traffic matching none of the application or L4 rules (default for default is 0, which means that shunting is disabled).
  • supported entries for the filter section include all the nDPI application protocols
  • supported actions for the filter section are forward and discard.
  • please check ndpiReader -h (included in ndpi) for the full list of supported nDPI protocols

It is also possible to set specify the default action for detected protocols adding the global default:

[global]
default = forward

This file can be provided to the filtering engine using the pfring_ft_load_configuration API:

pfring_ft_load_configuration(ft, "/etc/ft_rules.conf");

The pfring_ft_process API returns PFRING_FT_ACTION_DISCARD as action for packets that should be discarded according to the filtering or shunting policies.

In addition to the built-in filtering engine, the application can mark flows for filtering or shunting them based on custom policies using the pfring_ft_flow_set_action API.

The configuration file can also be used to configure advanced settings for nDPI detection and flow table, including the flow idleness and lifetime timeouts. Example:

[global]
default = forward
dpi_min_num_tcp_packets = 12
dpi_min_num_udp_packets = 5
flow_idle_timeout = 30
flow_lifetime_timeout = 120

Please keep in mind tha the Deep Packet Inspection engine implemented with nDPI requires the inspection of a few packets at the beginning of each communication before being able to identify with a reasonable degree of confidence the L7 protocol. For this reason, the initial packets of any communication are usually not marked with a protocol and not filtered when this is used for filtering traffic, until the L7 protocol is detected.

IDS Acceleration

The PF_RING FT L7 filtering engine can also be used for accelerating CPU-bound applications, such as IDS/IPSs, shunting flows based on the application protocol. Discarding elephant flows is becoming a common yet effective practice for reducing the amount of traffic an IDS/IPS need to inspect (typically multimedia traffic), dramatically reducing packet loss and improving the system performance.

Leveraging on PF_RING FT, any PF_RING-based or Libpcap-based application can take advantage of L7 shunting without changing a single line of code, all you need to do is to set the PF_RING_FT_CONF environment variable with the path of the configuration file.

For example, if we want to filter out Youtube and Netflix traffic, we need to create a configuration file like the one below:

# cat /etc/pf_ring/ft-rules.conf
[filter]
YouTube = discard
Netflix = discard

And run Suricata on top of PF_RING, setting the PF_RING_FT_CONF environment variable according to the configuration file path:

# PF_RING_FT_CONF=/etc/pf_ring/ft-rules.conf suricata --pfring-int=zc:eth1 -c /etc/suricata/suricata.yaml

Please note that it is also possible to define custom L7 protocols (e.g. based on ip or hostname) using the standard nDPI file format for custom protocols (see an example here) and provide the file path by setting the PF_RING_FT_PROTOCOLS environment variable.

Please refer to the Using Suricata with PF_RING and Using Bro with PF_RING guides for enabling PF_RING FT support in Suricata and Bro. Note that the same acceleration can be used with other IDSs like Snort.