[LU-16959] LNet: auto-tune ARP-related sysctl setting

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Labels: lnet
• Severity: 3
• Assignee: Frank Sehr
• Reporter: Serguei Smirnov

Description

Default linux settings for net.ipv4.neigh.default.gc_thresh* may be too low:

net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024

For example, this can cause the garbage collector in the kernel to start deleting entries in the table once the table size exceeds gc_thresh2, leading to unnecessary ARP traffic.

If IP to MAC mappings in the system are permanent, setting a higher gc_thresh1 may also help avoid unnecessary ARP traffic as this is the threshold for when entries start timing out. 

In large systems this may cause troubles with evictions, etc.

The memory cost of having larger tables is low, so It is proposed to use the following settings:

net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 16384
net.ipv4.neigh.default.gc_thresh3 = 32768

The user should be able to make the choice to manage the settings on their own, probably via a module parameter.

Note that there's the similar set of IPv6 parameters to consider.