Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20867

sanity-sec test_79: Timeout occurred after 313 minutes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • 3
    • 9223372036854775807

      sanity-sec test_79 TIMEOUTs in the review-dne-selinux-ssk-part-2 test group (lustre-reviews, el8.10 and el9.8 x86_64 servers/clients):

      Timeout occurred after 313 minutes, last suite running was sanity-sec
      

      The test hangs right after the first client remount with the c0 key (gssonly_identification), at "[[ -f $DIR/this_is_c0 ]]". It is failing across unrelated patches (several of which only touch lustre/tests).

      Frequency: 4 TIMEOUT out of 33 non-skipped test_79 runs in lustre-reviews on 2026-10-01..02 (all in review-dne-selinux-ssk-part-2, on both el8.10 and el9.8; test_79 is skipped without shared keys). lustre-master has no sanity-sec runs in the scanned sessions. The code involved (lsvcgssd nodemap lookup, test_79, the full OST remounts in test_70/test_75b) all landed between 2024 and June 2026, so there is no obvious recent landing that started it; a possible but unconfirmed contributor is 14cf8275e232 ("LU-20087 tests: fix sanity-sec/27ad hang with shared key", landed 2026-08-22), which adds another per-nodemap SSK mount before test_70.

      Example sessions:

      Client stat is stuck in a glimpse because GSS context negotiation to the OSTs never succeeds:

      LustreError: gss_keyring.c:1858:gss_kt_update()) lustre-OST0000_UUID at 10.240.38.135@tcp: negotiation: rpc err 0, gss err d0000
      LustreError: gss_keyring.c:1858:gss_kt_update()) Skipped 3813 previous similar messages
      
      task:bash state:I pid:850930
       cl_sync_io_wait+0xda/0x350 [obdclass]
       cl_lock_request+0x1b5/0x1e0 [obdclass]
       cl_glimpse_lock+0x1ef/0x2a0 [lustre]
       __cl_glimpse_size+0x168/0x2a0 [lustre]
       ll_getattr_dentry+0xbec/0x1140 [lustre]
       vfs_statx+0x8a/0xe0
      

      On the OSS, lsvcgssd sends the gss identification nodemap lookup ioctl to the wrong obd device, so the rsi upcall fails:

      LustreError: 9815:0:(ofd_obd.c:1327:ofd_iocontrol()) lustre-OST0000: iocontrol from 'lsvcgssd' cmd=c00866c5 _IOWR('f', 197, 8) unrecognized: rc = -25
      LustreError: 597333:0:(gss_svc_upcall.c:904:gss_svc_upcall_handle_init()) lustre-OST0000: failed to get entry from rsi cache (nid 10.240.38.133@tcp): rc = -43
      

      In session 83e643f3 the same ioctl lands on lustre-OST0006-osd on the OSS and on lustre-MDT0003-osd on MDS2:

      LustreError: 13487:0:(obd_class.h:1208:obd_iocontrol()) lustre-OST0006-osd: no obd_iocontrol operation
      LustreError: 5864:0:(obd_class.h:1208:obd_iocontrol()) lustre-MDT0003-osd: no obd_iocontrol operation
      LustreError: 349718:0:(gss_svc_upcall.c:904:gss_svc_upcall_handle_init()) lustre-MDT0001: failed to get entry from rsi cache (nid 10.240.46.101@tcp): rc = -43
      

      VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
      sanity-sec test_79 - Timeout occurred after 312 minutes, last suite running was sanity-sec

            sebastien Sebastien Buisson
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: