Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-19765

High latency in Lustre operations and Permission Denied errors due to Identity Upcall timeouts (rc = -110)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Medium Medium
    • None
    • Lustre 2.17.0
    • None
    • 3
    • 9223372036854775807

      On a Lustre 2.17.0 cluster, metadata operations (mount, unmount, touch) are experiencing extreme delays (2–4 minutes). While root can eventually complete a touch, non-root users receive "Permission denied" after a significant hang.

      Setting identity_upcall to NONE bypasses the delay and restores normal performance, indicating a bottleneck or failure in the l_getidentity_nss upcall path.

      Environment

      • Lustre Version: 2.17.0_RC4_4_g81318f8
      • Upcall Configuration: mdt.* MDT*.identity_upcall=/usr/sbin/l_getidentity_nss
      • Identity Source: Remote LDAP via NSS

      Symptoms

      1. Metadata Latency: ** A touch as root takes over 4 minutes:

      # time touch /lus/fs_name/test_file
      real    4m1.679s
      user    0m0.001s
      sys     0m0.000s 

      2. Permission Denied for Non-Root: ** Users encounter EACCES after a ~2-minute timeout:

      $ time touch /lus/fs_name/user_file
      touch: cannot touch '/lus/fs_name/user_file': Permission denied
      real    2m1.124s 

      3. MDS Kernel Errors: The MDS nodes report upcall cache acquisition failures with error -110:

      LustreError: [PID]:0:(upcall_cache.c:342:upcall_cache_get_entry()) [FS]-MDT0000: acquire for key [UID] after 30: rc = -110 

      The error rc = -110 in upcall_cache_get_entry suggests that the Lustre MDS is successfully triggering the upcall, but the user space process (l_getidentity_nss) is failing to return identity/group information within the 30-second window.

      Steps to Reproduce

      1. Configure Lustre MDT to use /usr/sbin/l_getidentity_nss.
      1. Ensure an LDAP-backed NSS configuration is active on the MDS.
      1. Perform metadata operations from a client.
      1. Observe rc = -110 in dmesg on the MDS and high real time on the client.

            pravs Praveen Kumar S
            pravs Praveen Kumar S
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: