10 Gbit Hardware Packet Filtering Using Commodity Network Adapters

Prerequisites

All the concepts and code described in this page requires modern networking hardware, an Intel 82599-based (e.g. Intel X520) or Silicom Director 10 Gbit NIC and a recent Linux kernel (>= 2.6.31).

Developing efficient networking applications in Linux is not a straight forward task and requires a fair amount of understanding of the hardware and the network stack. We strongly advise you to read this white paper in order to see some pointers that are important in developing networking applications.

Introduction: About Hardware Packet Filtering

Using filters in hardware gives two great performance advantages:

1. Allows the user to use software to set filters that direct specific flows into specific CPU cores allowing better cache utilization. This can also be achieved with software, but HW filtering in PF_RING™ allows for better performance than software filtering.
2. Allows the dropping of packets with zero CPU impact as these packets are now dropped on the wire and never hit any software stack, once the filter is in place.

PF_RING™ supports on the fly hardware filtering for Intel© 82599-based (in detail, it supports Flow Director and Five Tuple Queue Filtering (FTQF) as described in the 82599 datasheet) and Silicom Director network controllers. The figure below shows the queue assignment based in the hardware filtering of 82599 and PF_RING™.

How to Enable Hw Packet Filtering on Intel 82599-based NICs

Assuming the hardware is installed properly, the PF_RING™ enabled Intel© 10GbE driver (ixgbe) needs to be insmoded with flow director enabled in “perfect filter mode” for the desired while root (please see README of the driver for more details):

or you can use the load_driver.sh script file that you need to modify changing the interface name according to your system

Now make sure that the flow director in perfect filtering is indeed enabled.

The MSI-X interrupts in Linux as of the time this document is written are assigned randomly which means that each queue will deliver packets to a random core. Through SMP affinity the user can define re-assign the interrupts and choose themselves which queue to bind to each core. For more details how to set the SMP affinity please see this white paper mentioned before. Alternatively, one can use a script provided in the ixgbe driver that will map queue 0 to core 0, queue 1 to core 1 etc.

The output, depending on your configuration, will look somehow like this:

Perfect filters comprise of a tuple of [VLAN, protocol, source IP Address/mask, source port, destination IP Address/mask, destination port]. In order to drop a flow rather than redirecting to a particular queue we are using the convention of queue_id=-1.
In the perfect filtering there is a one global mask for all perfect filters. That means that if a /32 mask is used for example for the source IP address and later on another filter gets added with a /16 mask, the global mask will change from /32 to /16 for ALL already added filters, which may not be what the user intended, so we are always using a /32 mask.
Five Tuple Queue Filters instead comprise of a tuple of [protocol, source IP Address, source port, destination IP Address, destination port].
In order to add/remove hardware filters the PF_RING™ API provides two explicit functions:

or it is possible to use the same functions used for software filters by setting a filtering mode other than software_only via pfring_set_filtering_mode().

Final Notes

Affinitizing the queues of a multiqueue interface in 10 GbE is crucial in the performance of any networking application. With the above user guide the PF_RING™ can be used to take advantage of in-hardware on the fly queue to core assignment allowing the software to have full control over to which flow goes to which CPU core. The result will be a much more efficient application that has more control over how it handles the network packets.

FAQ

• Q. How many filters a 82599-based card typically supports?
  A. You can have up to 8K-2 Perfect filters and 128 Five-Tuple filters.
• Q. Do I need to reconfigure the NIC whenever I add/remove a filter?
  A. No. Filter manipulation is immediate and it does not interfere with other ongoing activities.
• Q. How long does it take to add/remove a filter?
  A. Contrary to many FPGA-based NICs, this operation is instantenous as filters are configured by means of registers.
• Q. How much a 82599-based card typically costs?
  A. As of today the cost per-port is well below 1000$ per 10 Gbit port.
• Q. Beside FlowDirector and FTQF, can 82599 provide other type of filters?
  A. Yes 82599 allows you for instance to filter VLAN and ethernet packets, not to mention payload content. See the 82599 datasheet for more information.
• Q. What Linux kernel version do I need in order to exploit this code?
  A. You need Linux 2.6.31 or above.
• Q. Do I get the driver source code?
  A. Absolutely. The code is released under GPL and you get the full source code.
• Q. What hardware PC do I need for wire-speed 1G packet capture at any packet size?
  A. A single quad-core Xeon is more than enough.
• Q. My computer has DCA disabled and the BIOS do not allow me to enable it. What shall I do?
  A. Have a look at this link.
• Q. Is this code useful for general-purpose networking?
  A. Yes, hardware filtering can be applied to all networking tasks.

References

• 82599 datasheet
• Wire-Speed Hardware-Assisted Traffic Filtering with Mainstream Network Adapters
• Design considerations for efficient network applications with Intel® multi-core processor-based systems on Linux
• 10 Gbit PF_RING™-based Hardware Packet Filtering and Balancing Previewed at the Intel Europe Conference