[LU-16146] after dropping mgs/mdt0 for test: mdt_handler.c:7522:mdt_postrecov()) lflood-MDT0000: auto trigger paused LFSCK failed: rc = -6

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 2.15.0
• Labels: llnl
• Environment: TOSS 4.4-4.1 ; 4.18.0-372.19.1.1toss.t4.x86_64 ; lustre 2.15.0_3.llnl
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Gian-Carlo Defazio

Description

After doing a test that involved turning off the mgs/mdt0 node for the test cluster garter, the mdts fail to come back up.  They are stuck "WAITING".

When the mgs node was powered off, an ior test was in progress.

Failover seemed to work, but the ior job was unable to finish. The cluster was eventually rebooted.

After the reboot, the logs show that all 4 mdts are failing an lfsck_start with a line such as

2022-09-07 14:35:12 [ 5791.188613] Lustre: 32474:0:(mdt_handler.c:7522:mdt_postrecov()) lflood-MDT0000: auto trigger paused LFSCK failed: rc = -6

I believe that -6 is coming from lfsck_start()

        lfsck = lfsck_instance_find(key, true, false);                                                           
        if (unlikely(lfsck == NULL))                                                                             
                RETURN(-ENXIO);                                                                                  
                                                                                                                 
        if (unlikely(lfsck->li_stopping))                                                                        
                GOTO(put, rc = -ENXIO); 

For our reference, our local ticket is TOSS5875