In a DNE configuration, when one MDT reboots, the LDLM blocking thread pool on other MDTs can become permanently exhausted due to truncate_inode_pages() blocking on PG_locked directory pages in ll_lock_cancel_bits(). This causes a cascading deadlock that requires rebooting all MDTs.

Root Cause{}

  ll_lock_cancel_bits() (lustre/llite/namei.c) calls truncate_inode_pages(inode->i_mapping, 0) unconditionally for directory inodes when MDS_INODELOCK_UPDATE is being canceled. This function waits on PG_locked pages.

  On the MDS internal client, directory pages become PG_locked when mdc_read_page_remote() has an in-flight readdir RPC. During recovery, these  RPCs are placed on imp_delayed_list with rq_waiting=1 because the import is not in FULL state. Delayed RPCs have no timeout:

  - ptlrpc_set_next_timeout() skips rq_waiting=1 requests

  - ptlrpc_expired_set() skips rq_waiting=1 requests

  - ptlrpc_wake_delayed() only fires when imp_state == LUSTRE_IMP_FULL

  The import cannot reach FULL because the recovery state machine needs blocking threads for lock replay/cancel — but all blocking threads are stuck in truncate_inode_pages(). This creates a permanent deadlock with no self-healing mechanism.

  Observed Incident{}

  On a production 16-MDT filesystem:

  - MDT000d auto-rebooted due to load (67 avg on 16 cores) at 22:38:20 UTC

  - MDT000d recovery started at 22:38:56: Will be in recovery for at least 2:30, or until 583 clients reconnect

  - MDT000d recovery completed at 22:41:26: Recovery over after 2:30, of 583 clients 581 recovered and 2 were evicted.

  - At 22:43 UTC (2 min after recovery completed): all 10+ ldlm_bl_* threads on MDT1 blocked for 122+ seconds

  - By 22:44: MDT1's MDC to itself (0@lo) entered recovery

  - By 23:07: MDT1 started evicting customer clients

  - By 23:10: other MDTs timing out trying to get locks from MDT1

  - At 00:55: all 16 MDTs rebooted (2h12m downtime)

  The critical timing: the blocking threads got stuck during the 2.5-minute recovery window (before 22:41:26), and never recovered even after MDT000d's recovery completed — because the MDC import to MDT000d could never reach FULL state (requires blocking threads that are all stuck).