Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21040

conf-sanity test_66: check lustre-OST0000.osc.active failed! with many MDTs per MDS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • 3
    • 9223372036854775807

      [Claude] conf-sanity test_66 fails with many MDTs per MDS: after the MGS/MDS restart, the "lctl conf_param $FSNAME-OST0000.osc.active=1" change is not seen on MDT0000 within the 90s that set_conf_param_and_check() waits.

      CMD: onyx-155vm26 /usr/sbin/lctl conf_param lustre-OST0000.osc.active='1'
      Waiting 90s for '1'
      ...
      Update not seen after 90s: want '1' got '0'
       conf-sanity test_66: @@@@@@ FAIL: check lustre-OST0000.osc.active failed!
        = test-framework.sh:6316:set_conf_param_and_check()
        = conf-sanity.sh:6008:test_66()
      

      Configuration: 2 MDS nodes with 6 MDTs each (MDSCOUNT=12), from the custom test session for https://review.whamcloud.com/69303 (LU-14058 >10 MDT script fixes), which removes the old MDT-count limits from the test scripts.

      The MDS console (onyx-155vm26) shows the MGC applying the new record to each local MDT configuration log about 20s apart, highest MDT index first, so MDT0000 is updated last:

      [39035.583513] Lustre: Permanently reactivating lustre-OST0000
      [39035.591801] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client
      [39065.757442] LustreError: osp_process_config()) lustre-OST0000-osc-MDT000a: unknown param osc.active=1
      [39086.237341] LustreError: osp_process_config()) lustre-OST0000-osc-MDT0008: unknown param osc.active=1
      [39106.717188] LustreError: osp_process_config()) lustre-OST0000-osc-MDT0006: unknown param osc.active=1
      [39127.197055] LustreError: osp_process_config()) lustre-OST0000-osc-MDT0004: unknown param osc.active=1
      

      With 6 MDTs on the node, MDT0000 would be processed at about +130s, beyond the 90s wait. With the usual 2-4 MDTs the same ~20s per log stays under 90s, which is why this is not seen in normal DNE testing.

      Questions to answer:

      • Why does the MGC config-log requeue take ~20s per MDT log after the MGS restart (lock enqueue/RPC timeout per log?). Fixing that is preferable to lengthening the test wait.
      • "unknown param osc.active=1" for each osp device: the osc.active conf_param is apparently not handled by osp_process_config(), so the MDT side only picks up activation through a different path.

      conf-sanity test_66 is excluded with MDSCOUNT > 9 by LU-14058 until this is fixed.

            adilger Andreas Dilger
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: