[LU-8177] osp-syn threads in D state

• Type: Bug
• Resolution: Incomplete
• Priority: Blocker
• Affects Version/s: Lustre 2.5.3
• Severity: 1
• Assignee: John Fuchs-Chesney (Inactive)
• Reporter: Mahmoud Hanafi

Description

When trying to mount mdt all osp-syn threads stuck in 'D' state.

Debug logs are filled with these messages

00000004:00080000:8.0:1463850740.016156:0:14081:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebf799800, transno 0
00000004:00080000:8.0:1463850740.016164:0:14081:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebf799800/1, rc -2, transno 0
00000100:00100000:8.0:1463850740.016176:0:14081:0:(client.c:1872:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_3:nbp2-MDT0000-mdtlov_UUID:14081:1534957896521600:10.151.26.98@o2ib:6
00000004:00080000:9.0:1463850740.016219:0:14087:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebed48800, transno 0
00000004:00080000:9.0:1463850740.016226:0:14087:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebed48800/1, rc -2, transno 0

I will upload full debug logs to ftp site.

Issue Links

• is related to: LU-7079 OSP shouldn't discard requests due to imp_peer_committed_transno

Comments

[John Fuchs-Chesney (Inactive)]

Thank you for the update Mahmoud.

I think we'll keep the priority as it is for the time being (for recording purposes).

Do you want us to keep the ticket open for a while longer? Or do you think this event is now resolved?

Best regards,
~ jfc.

[Mahmoud Hanafi]

Ok thanks. You may lower the priority of the case. It did finish.

[Oleg Drokin]

Generally since I believe your system is now mountable, it's safer to just let the sync threads to run their course. it would put additional load on the system, but should not be too bad.

Once you apply lu7079 patch it should kill those records for good next time you reboot.
Without the patch some of the records would still be killed, but not all of them (and more might be amassed until next reboot) so you are looking at a similar situation next time you remount anyway.

[Mahmoud Hanafi]

no we don't

[Oleg Drokin]

do you use changelogs too?

[Mahmoud Hanafi]

I looked in /O/1/d* and there where files going back to 2015.

should i just delete everything in /0/1/* and remount?

[Oleg Drokin]

if you really need to clear the condition immediately, it's possible to unmount the MDT, mount it as ldiskfs, remove the stale llogs, unmount ldiskfs and remount mdt as lustre.
Perhaps not all of them needs removing but just hte really old ones (you can tell by the date).

[Oleg Drokin]

Alex advises that the condition will clear on it's own after all llogs are reproceessed. the duration of that is hard to tell as it depends on number of those llogs.

[Mahmoud Hanafi]

This was a remount after power down.
The OST are mounted.
The shutdown was normal.

I unmounted all the OSTs the the mdt got mounted. Then I remounted the OSTs and the mdt got back to osp-sync in 'D' state. but at least i am able to mount it on the client.

So do we need to apply the patch from LU-7079 and remount? or can we some how stop the osp-sync.

[Alex Zhuravlev]

basically some llog cancels got lost by mistake causing lots of IO to rescan llogs at startup.

[Alex Zhuravlev]

I think this can be a dup of LU-7079