Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20581

sanity-lnet test_50 allocates the default 1 GiB router pool

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • 3
    • 9223372036854775807

      test_50 enables routing through a yaml import without setting any router buffer parameter, so LNet allocates its default pool. On a node with limited memory the import fails:

      /usr/sbin/lnetctl import /tmp/sanity-lnet-50.yaml
      ---
      add:
      -     import:
            errno: -12
            descr: ! "Out of memory"
      ...
       sanity-lnet test_50: @@@@@@ FAIL: Import failed rc = 234
      

      The kernel reports the size of the request:

      LNetError: lnet_rtrpool_adjust_bufs()) lnet: error allocating 1024x256 page router buffers on CPT 0: rc = -12
      

      That is LNET_NRB_LARGE (1024) buffers of LNET_MTU (1 MiB) each, so about 1 GiB, plus LNET_NRB_SMALL 4 KiB buffers for another 64 MiB. lnet_nrb_*_calculate() divides the count by LNET_CPT_NUMBER and lnet_rtrpools_alloc() then allocates that many per CPT, so the total is about 1.06 GiB whatever the CPU count.

      The test passes in isolation on a 1385 MB node and fails when run after the preceding subtests, once the page cache is populated.

      Other tests that enable routing do not hit this because setup_router_test() sets the counts as lnet module parameters:

      	mod_opts+=" large_router_buffers=4"
      	mod_opts+=" small_router_buffers=8"
      	mod_opts+=" tiny_router_buffers=$NTRB"
      

      test_50 should do the same. tiny_router_buffers, small_router_buffers and large_router_buffers are all module_param(..., 0444), so they can only be set when lnet is loaded; the test currently calls reinit_dlc(), which reuses an already configured LNet.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: