[LU-8177] osp-syn threads in D state

• Type: Bug
• Resolution: Incomplete
• Priority: Blocker
• Affects Version/s: Lustre 2.5.3
• Severity: 1
• Assignee: John Fuchs-Chesney (Inactive)
• Reporter: Mahmoud Hanafi

Description

When trying to mount mdt all osp-syn threads stuck in 'D' state.

Debug logs are filled with these messages

00000004:00080000:8.0:1463850740.016156:0:14081:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebf799800, transno 0
00000004:00080000:8.0:1463850740.016164:0:14081:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebf799800/1, rc -2, transno 0
00000100:00100000:8.0:1463850740.016176:0:14081:0:(client.c:1872:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_3:nbp2-MDT0000-mdtlov_UUID:14081:1534957896521600:10.151.26.98@o2ib:6
00000004:00080000:9.0:1463850740.016219:0:14087:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebed48800, transno 0
00000004:00080000:9.0:1463850740.016226:0:14087:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebed48800/1, rc -2, transno 0

I will upload full debug logs to ftp site.

Issue Links

• is related to: LU-7079 OSP shouldn't discard requests due to imp_peer_committed_transno

Comments

[Jian Yu]

Hello Jay,

Here is the back-ported patch for Lustre b2_5_fe branch: http://review.whamcloud.com/20392

[Jay Lan (Inactive)]

We have a b2_7_fe version of the patch, but need a back port to b2_5_fe. Thanks!

[Mahmoud Hanafi]

Please leave the case open for now.

[John Fuchs-Chesney (Inactive)]

Thank you for the update Mahmoud.

I think we'll keep the priority as it is for the time being (for recording purposes).

Do you want us to keep the ticket open for a while longer? Or do you think this event is now resolved?

Best regards,
~ jfc.

[Mahmoud Hanafi]

Ok thanks. You may lower the priority of the case. It did finish.

[Oleg Drokin]

Generally since I believe your system is now mountable, it's safer to just let the sync threads to run their course. it would put additional load on the system, but should not be too bad.

Once you apply lu7079 patch it should kill those records for good next time you reboot.
Without the patch some of the records would still be killed, but not all of them (and more might be amassed until next reboot) so you are looking at a similar situation next time you remount anyway.

[Mahmoud Hanafi]

no we don't

[Oleg Drokin]

do you use changelogs too?

[Mahmoud Hanafi]

I looked in /O/1/d* and there where files going back to 2015.

should i just delete everything in /0/1/* and remount?

[Oleg Drokin]

if you really need to clear the condition immediately, it's possible to unmount the MDT, mount it as ldiskfs, remove the stale llogs, unmount ldiskfs and remount mdt as lustre.
Perhaps not all of them needs removing but just hte really old ones (you can tell by the date).

[Oleg Drokin]

Alex advises that the condition will clear on it's own after all llogs are reproceessed. the duration of that is hard to tell as it depends on number of those llogs.