Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21200

sanity-quota test_73b: MDS deadlock in quota pool lookup (qti_pools_add vs qsd_reint_qpool), suite TIMEOUT

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      sanity-quota test_73b ("default OST Pool Quotas limit for new user") hangs and the sanity-quota test set ends with "Timeout occurred after N minutes, last suite running was sanity-quota". It is a deadlock in the quota pool code on the MDS, not a data path failure.

      The suite_log stops right after "set default quota for qpool1", i.e. at the runas dd in test_73b. The MDS stack_trace log from the timed out subtest shows about 55 mdt_rdpg threads blocked in rwsem_down_read_slowpath from qti_pools_add, one mdt thread blocked in rwsem_down_read_slowpath (cfs_hash_lookup / cfs_hash_multi_bd_lock), and qsd_reint_qpool blocked in rwsem_down_write_slowpath (cfs_hash_find_or_add / cfs_hash_multi_bd_lock). Clients are evicted/lose connection to MDT0000 while the hang lasts.

      Frequency: rainshadow test sanity-quota test_73b shows 121 branch failures in 30 days (lustre-master-next 65, lustre-master 56), all TIMEOUT, first seen 2026-11-02, in full-part-2 (about 13-15% of runs) and full-dne-part-2 (17-25%). Reviews ran 6604 suite runs with a single TIMEOUT, on change 64875 (LU-19783 lfsck, unrelated: the patch does not touch quota or the default pool quota path).

      Example failures:
      master full-part-2 (same stack): https://testing.whamcloud.com/sub_tests/a3490105-320a-441d-b648-88701ce9807a
      review-zfs on change 64875 ps39: https://testing.whamcloud.com/sub_tests/51859b9d-cc9b-42fe-809b-cc42995966c9

      No existing ticket matches (LU-16448, LU-17974 are qsd_reint_qpool crashes, resolved). Likely a lock ordering problem between qsd_reint_qpool and qti_pools_add/lqe pool lookups (see the nested lqe lookup fix for test_73b).

      Raised by Patch Watcher run pw-engineer-64875-ps39-517a61d8daf3 on https://review.whamcloud.com/64875

      VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
      sanity-quota test_73b - Timeout occurred after 939 minutes, last suite running was sanity-quota

            xiwang Xiyang Wang
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: