https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8177 Lustre <p>When trying to mount mdt all osp-syn threads stuck in 'D' state. </p> <p>Debug logs are filled with these messages </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000004:00080000:8.0:1463850740.016156:0:14081:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebf799800, transno 0 00000004:00080000:8.0:1463850740.016164:0:14081:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebf799800/1, rc -2, transno 0 00000100:00100000:8.0:1463850740.016176:0:14081:0:(client.c:1872:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_3:nbp2-MDT0000-mdtlov_UUID:14081:1534957896521600:10.151.26.98@o2ib:6 00000004:00080000:9.0:1463850740.016219:0:14087:0:(osp_sync.c:317:osp_sync_request_commit_cb()) commit req ffff883ebed48800, transno 0 00000004:00080000:9.0:1463850740.016226:0:14087:0:(osp_sync.c:351:osp_sync_interpret()) reply req ffff883ebed48800/1, rc -2, transno 0 </pre> </div></div> <p>I will upload full debug logs to ftp site.</p> LU-8177

osp-syn threads in D state

Bug Blocker Resolved Incomplete John Fuchs-Chesney Mahmoud Hanafi Sat, 21 May 2016 17:28:28 +0000 Fri, 17 Jun 2016 23:15:17 +0000 Fri, 17 Jun 2016 23:15:17 +0000 Lustre 2.5.3 0 9 <p>Uploaded file to ftp:/uploads/LU8177/s600.debug.out.gz</p> <p>Mahmoud,</p> <p>Can you please clarify if you have a production site down emergency please? We rate that as a SEV-1 event, and you have selected SEV-4.</p> <p>Thanks,<br/> ~ jfc.</p> <p>Sorry it should be level 1. </p> <p>Email from Mahmoud: "Sorry this should be severity1. The production site is down and unusable."</p> <p>~ jfc.</p> <p>Assigning to me &#8211; Oleg is looking.<br/> ~ jfc.</p> <p>Are there any messages in dmesg on mds or osts?<br/> Is this a normal mount after a normal shutdown? a failover after something else?<br/> are the OSTs up?</p> <p>I think this can be a dup of <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a></p> <p>basically some llog cancels got lost by mistake causing lots of IO to rescan llogs at startup.</p> <p>This was a remount after power down. <br/> The OST are mounted. <br/> The shutdown was normal.</p> <p>I unmounted all the OSTs the the mdt got mounted. Then I remounted the OSTs and the mdt got back to osp-sync in 'D' state. but at least i am able to mount it on the client.</p> <p>So do we need to apply the patch from <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> and remount? or can we some how stop the osp-sync.</p> <p>Alex advises that the condition will clear on it's own after all llogs are reproceessed. the duration of that is hard to tell as it depends on number of those llogs.</p> <p>if you really need to clear the condition immediately, it's possible to unmount the MDT, mount it as ldiskfs, remove the stale llogs, unmount ldiskfs and remount mdt as lustre.<br/> Perhaps not all of them needs removing but just hte really old ones (you can tell by the date).</p> <p>I looked in /O/1/d* and there where files going back to 2015. </p> <p>should i just delete everything in /0/1/* and remount?</p> <p>do you use changelogs too?</p> <p>no we don't</p> <p>Generally since I believe your system is now mountable, it's safer to just let the sync threads to run their course. it would put additional load on the system, but should not be too bad.</p> <p>Once you apply lu7079 patch it should kill those records for good next time you reboot.<br/> Without the patch some of the records would still be killed, but not all of them (and more might be amassed until next reboot) so you are looking at a similar situation next time you remount anyway.</p> <p>Ok thanks. You may lower the priority of the case. It did finish. </p> <p>Thank you for the update Mahmoud.</p> <p>I think we'll keep the priority as it is for the time being (for recording purposes).</p> <p>Do you want us to keep the ticket open for a while longer? Or do you think this event is now resolved?</p> <p>Best regards,<br/> ~ jfc.</p> <p>Please leave the case open for now.</p> <p>We have a b2_7_fe version of the patch, but need a back port to b2_5_fe. Thanks!</p> <p>Hello Jay,</p> <p>Here is the back-ported patch for Lustre b2_5_fe branch: <a href="http://review.whamcloud.com/20392" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20392</a></p> <p>Thanks!</p> <p>Hello Mahmoud,</p> <p>Do you want us to continue to keep this ticket open?</p> <p>Thanks,<br/> ~ jfc.</p> <p>Resolving as incomplete.</p> <p>Please let us know if any further work is required on this ticket.</p> <p>Thanks,<br/> ~ jfc.</p> Related LU-7079 Development End date Fri, 3 Jun 2016 17:28:28 +0000 Rank 1|hzycev: Rank (Obsolete) 9223372036854775807 Severity Start date Mon, 23 May 2016 17:28:28 +0000