[LU-8177] osp-syn threads in D state

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.5.3
• Type: Bug
• Priority: Blocker
• Reporter: Mahmoud Hanafi
• Assignee: John Fuchs-Chesney (Inactive)
• Resolution: Incomplete
• Severity: 1

Description

When trying to mount mdt all osp-syn threads stuck in 'D' state.

Debug logs are filled with these messages

00000004:00080000:8.0:1463850740.016156:0:14081:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebf799800, transno 0
00000004:00080000:8.0:1463850740.016164:0:14081:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebf799800/1, rc -2, transno 0
00000100:00100000:8.0:1463850740.016176:0:14081:0:(client.c:1872:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_3:nbp2-MDT0000-mdtlov_UUID:14081:1534957896521600:10.151.26.98@o2ib:6
00000004:00080000:9.0:1463850740.016219:0:14087:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebed48800, transno 0
00000004:00080000:9.0:1463850740.016226:0:14087:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebed48800/1, rc -2, transno 0

I will upload full debug logs to ftp site.

Issue Links

• is related to: LU-7079 OSP shouldn't discard requests due to...

Comments

[Mahmoud Hanafi]

Uploaded file to ftp:/uploads/LU8177/s600.debug.out.gz

[John Fuchs-Chesney (Inactive)]

Mahmoud,

Can you please clarify if you have a production site down emergency please? We rate that as a SEV-1 event, and you have selected SEV-4.

Thanks,
~ jfc.

[Mahmoud Hanafi]

Sorry it should be level 1.

[John Fuchs-Chesney (Inactive)]

Email from Mahmoud: "Sorry this should be severity1. The production site is down and unusable."

~ jfc.

[John Fuchs-Chesney (Inactive)]

Assigning to me – Oleg is looking.
~ jfc.

[Oleg Drokin]

Are there any messages in dmesg on mds or osts?
Is this a normal mount after a normal shutdown? a failover after something else?
are the OSTs up?

[Alex Zhuravlev]

I think this can be a dup of LU-7079

[Alex Zhuravlev]

basically some llog cancels got lost by mistake causing lots of IO to rescan llogs at startup.

[Mahmoud Hanafi]

This was a remount after power down.
The OST are mounted.
The shutdown was normal.

I unmounted all the OSTs the the mdt got mounted. Then I remounted the OSTs and the mdt got back to osp-sync in 'D' state. but at least i am able to mount it on the client.

So do we need to apply the patch from LU-7079 and remount? or can we some how stop the osp-sync.

[Oleg Drokin]

Alex advises that the condition will clear on it's own after all llogs are reproceessed. the duration of that is hard to tell as it depends on number of those llogs.

[Oleg Drokin]

if you really need to clear the condition immediately, it's possible to unmount the MDT, mount it as ldiskfs, remove the stale llogs, unmount ldiskfs and remount mdt as lustre.
Perhaps not all of them needs removing but just hte really old ones (you can tell by the date).

[Mahmoud Hanafi]

I looked in /O/1/d* and there where files going back to 2015.

should i just delete everything in /0/1/* and remount?

[Oleg Drokin]

do you use changelogs too?

[Mahmoud Hanafi]

no we don't

[Oleg Drokin]

Generally since I believe your system is now mountable, it's safer to just let the sync threads to run their course. it would put additional load on the system, but should not be too bad.

Once you apply lu7079 patch it should kill those records for good next time you reboot.
Without the patch some of the records would still be killed, but not all of them (and more might be amassed until next reboot) so you are looking at a similar situation next time you remount anyway.

[Mahmoud Hanafi]

Ok thanks. You may lower the priority of the case. It did finish.

[John Fuchs-Chesney (Inactive)]

Thank you for the update Mahmoud.

I think we'll keep the priority as it is for the time being (for recording purposes).

Do you want us to keep the ticket open for a while longer? Or do you think this event is now resolved?

Best regards,
~ jfc.

[Mahmoud Hanafi]

Please leave the case open for now.

[Jay Lan (Inactive)]

We have a b2_7_fe version of the patch, but need a back port to b2_5_fe. Thanks!

[Jian Yu]

Hello Jay,

Here is the back-ported patch for Lustre b2_5_fe branch: http://review.whamcloud.com/20392

[Jay Lan (Inactive)]

Thanks!

[John Fuchs-Chesney (Inactive)]

Hello Mahmoud,

Do you want us to continue to keep this ticket open?

Thanks,
~ jfc.

[John Fuchs-Chesney (Inactive)]

Resolving as incomplete.

Please let us know if any further work is required on this ticket.

Thanks,
~ jfc.