[LU-14048] Unable stop all OSTs due to obd_unlinked_exports

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.15.0
• Affects Version/s: Lustre 2.12.5
• Labels: LTS12
• Environment: 2.12.5_7.srcc (https://github.com/stanford-rc/lustre/commits/b2_12_5) on servers + 2.13 clients
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Stephane Thiell

Description

Since our Oak upgrade from Lustre 2.10.8 to 2.12.5, it has not always been possible to stop OSTs anymore (when doing a manual failover for example).

When stopping OSTs, we can see these messages on the servers:

Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST011a is waiting for obd_unlinked_exports more than 512 seconds. The obd refcount = 6. Is it stuck?
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST011e: UNLINKED ffff887ca4215400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST011e: UNLINKED ffff88de7f82bc00 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST011e: UNLINKED ffff88de717a7c00 f04b9922-9d5a-4 10.51.1.63@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST011e: UNLINKED ffff88deb8acd000 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST0112: UNLINKED ffff8833ab100c00 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:12 oak-io6-s2 kernel: Lustre: oak-OST0112: UNLINKED ffff88de6d629400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0

This has never been an issue with 2.10. We tried to manual failback OSTs this morning and we waited for about 10 minutes before rebooting the servers because it was too long and started to impact production. It would be nice to have a short timeout on this. Restart and recovery of the OSTs went well (perhaps worst case 1 client only was evicted).

Attaching some lustre debug logs from 3 different OSS oak-io[4-6]-s2:

• oststopissue_oak-io4-s2.dk.log.gz
• oststopissue_oak-io5-s2.dk.log.gz
• oststopissue_oak-io6-s2.dk.log.gz

Attaching kernel logs from the same 3 servers:

• oststopissue_oak-io4-s2.kern.log
• oststopissue_oak-io5-s2.kern.log
• oststopissue_oak-io6-s2.kern.log

 

This could be similar to the issue reported in LU-12319 which doesn't seem to be fixed at this time.

Thanks for the help!
Stephane