-
Bug
-
Resolution: Fixed
-
Medium
-
None
-
None
-
3
-
9223372036854775807
test_50 enables routing through a yaml import without setting any router buffer parameter, so LNet allocates its default pool. On a node with limited memory the import fails:
/usr/sbin/lnetctl import /tmp/sanity-lnet-50.yaml
---
add:
- import:
errno: -12
descr: ! "Out of memory"
...
sanity-lnet test_50: @@@@@@ FAIL: Import failed rc = 234
The kernel reports the size of the request:
LNetError: lnet_rtrpool_adjust_bufs()) lnet: error allocating 1024x256 page router buffers on CPT 0: rc = -12
That is LNET_NRB_LARGE (1024) buffers of LNET_MTU (1 MiB) each, so about 1 GiB, plus LNET_NRB_SMALL 4 KiB buffers for another 64 MiB. lnet_nrb_*_calculate() divides the count by LNET_CPT_NUMBER and lnet_rtrpools_alloc() then allocates that many per CPT, so the total is about 1.06 GiB whatever the CPU count.
The test passes in isolation on a 1385 MB node and fails when run after the preceding subtests, once the page cache is populated.
Other tests that enable routing do not hit this because setup_router_test() sets the counts as lnet module parameters:
mod_opts+=" large_router_buffers=4" mod_opts+=" small_router_buffers=8" mod_opts+=" tiny_router_buffers=$NTRB"
test_50 should do the same. tiny_router_buffers, small_router_buffers and large_router_buffers are all module_param(..., 0444), so they can only be set when lnet is loaded; the test currently calls reinit_dlc(), which reuses an already configured LNet.