Anyone using Intel X710 NICs?

We recently purchased some Intel XXV710 NICs for our Zeek systems. However, symmetric hashing does not seem to work on them, at least not completely.

There was some discussion here regarding adding some functionality to the driver to make it work, however this never landed: https://sourceforge.net/p/e1000/mailman/message/35199068/

This post discusses how the X710 controller must be configured differently from the 82599 10G controller (used by the X520 cards): https://haryachyy.wordpress.com/2019/01/18/learning-dpdk-symmetric-rss/

The odd part is that following the SEPTun-MarkII guide[1] makes it mostly work, but we’re consistently finding that ~1-2% of the traffic is not being symmetrically hashed. We’re testing with can-i-use-afpacket-fanout[2] and Zeek 3.2.

The most damning evidence is an Intel rep telling[3] a customer:

Unfortunately, we have been informed that the only support to setup symmetric RSS is via DPDK.

Searching the mailing list archives, I found a couple of posts where people were encouraged to use X710-based cards, so I’m left wondering: Are there people using these? Are they also seeing this 1-2% asymmetry? Or am I missing a configuration tweak?

Thanks,

–Vlad

[1] - <https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst>
[2] - <https://github.com/JustinAzoff/can-i-use-afpacket-fanout>
[3] - <https://community.intel.com/t5/Ethernet-Products/X-L-710-supports-symmetric-RSS-hashing-See-quot-7-1-9-3/m-p/685998/highlight/true#M17746>

Alright, I think we figured this out. We plan on releasing an Ansible role that will handle all this configuration, but the short version follows.

NICs with the X710 controller have some additional features that must be disabled. Something like this should work for 8 workers:

INF=ens2

ethtool -L $INF combined 8
ethtool -K $INF rxhash on
ethtool -K $INF ntuple on
set_irq_affinity 8-15 $INF # script ships with the i40e driver
ethtool -X $INF hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 8
ethtool -A $INF rx off tx off
ethtool -C $INF adaptive-rx off adaptive-tx off rx-usecs 125
ethtool -G $INF rx 1024
for i in rx tx tso ufo gso gro lro tx nocache copy sg txvlan rxvlan; do ethtool -K $INF $i off; done
for proto in tcp4 udp4 tcp6 udp6; do ethtool -N $INF rx-flow-hash $proto sdfn; done

Note that many of these settings will not persist after a reboot.

On top of that, zeek-af_packet-plugin had a bug where you could not change the fanout mode, so you need to run version 2.1.2 or newer. These NICs require either FANOUT_CPU or FANOUT_QM (FANOUT_HASH could be symmetric, but the option to make it so is missing from ethtool).

With these changes, we’re seeing perfectly symmetric hashing, none of the bugs that plague Myricom SNFv5 cards, and increased performance compared to Myricom SNFv3 and SNFv5.

Hope that helps someone else.

–Vlad

Hey Vlad,

it’s been a while since I added those settings to the SepTUN Mark II and the research made recently - and some things are out of date.

Intel claims it’s necessary to use the hardware symmetric load-balancing. It could be used but there’s no advantage over just setting the symmetric key, like you did.

For all traffic to be symmetrically hashed in a consistent way you need to do what you did and I’d advise using 3-tuple - and skipping ports.

Fragmented traffic has no port numbers, so fragments further in the chain might get incorrectly hashed. If you know you have no fragmented packets in your monitored segment, you should be fine.

I used ‘sd’ everywhere for the rx-flow-hash just in case and the distribution was good enough.

I’d also go with the smallest rx ring you can have - start with something like 256 and monitor packet drops. Smaller ring size is way more cache efficient and it improved the performance here.

Hi Vlad,

this is the problem we still have with same card and settings. Can I kindly ask you to post your:
1. Linux OS dist/version
2. NIC driver version: sudo ethtool -i nic_name
3. Ethtool version: sudo ethtool --version
4. Output for: sudo ethtool -k nic_name
or sudo ethtool --show-offload nic_name

Your help will be greatly appreciated.

Thanks

Miro

We’ve stopped using AF_PACKET, and are using DPDK: https://github.com/esnet/zeek-dpdk

The plugin should set all the settings for you. No more need for ethtool.

–Vlad