https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-128 Lustre <p>As suggested by Peter Jones, we open a Jira ticket for this issue in order to get the fix landed in 2.1.<br/> I simply copy here the initial description from bugzilla 24420:</p> <p>We are having this bug when we reboot some OSSs. It's being raised in the recovery phase and it's<br/> provoking a long Lustre service interruption.</p> <p>Each time/crash, the panic'ing thread stack-trace looked like following :<br/> =========================================================================<br/> #0 <span class="error">&#91;ffff881021fd1238&#93;</span> machine_kexec at ffffffff8102e66b<br/> #1 <span class="error">&#91;ffff881021fd1298&#93;</span> crash_kexec at ffffffff810a9ae8<br/> #2 <span class="error">&#91;ffff881021fd1368&#93;</span> panic at ffffffff8145210d<br/> #3 <span class="error">&#91;ffff881021fd13e8&#93;</span> lbug_with_loc at ffffffffa0454eeb<br/> #4 <span class="error">&#91;ffff881021fd1438&#93;</span> libcfs_assertion_failed at ffffffffa04607d6<br/> #5 <span class="error">&#91;ffff881021fd1488&#93;</span> filter_finish_transno at ffffffffa096c825<br/> #6 <span class="error">&#91;ffff881021fd1548&#93;</span> filter_do_bio at ffffffffa098e390<br/> #7 <span class="error">&#91;ffff881021fd15e8&#93;</span> filter_commitrw_write at ffffffffa0990a78<br/> #8 <span class="error">&#91;ffff881021fd17d8&#93;</span> filter_commitrw at ffffffffa09833d5<br/> #9 <span class="error">&#91;ffff881021fd1898&#93;</span> obd_commitrw at ffffffffa093affa<br/> #10 <span class="error">&#91;ffff881021fd1918&#93;</span> ost_brw_write at ffffffffa0943644<br/> #11 <span class="error">&#91;ffff881021fd1af8&#93;</span> ost_handle at ffffffffa094837a<br/> #12 <span class="error">&#91;ffff881021fd1ca8&#93;</span> ptlrpc_server_handle_request at ffffffffa060eb11<br/> #13 <span class="error">&#91;ffff881021fd1de8&#93;</span> ptlrpc_main at ffffffffa060feea<br/> #14 <span class="error">&#91;ffff881021fd1f48&#93;</span> kernel_thread at ffffffff8100d1aa<br/> =========================================================================</p> <p>In a particular analysis from our on-site support we get the following values when the LBUG is<br/> raised on "filter_finish_transno" function:</p> <p>lcd_last_transno=0x4ddebb<br/> oti_transno=last_rcvd=0x4ddeba<br/> lsd_last_transno=0x4de0ee</p> <p>So we have the client (lcd_last_transno) having a bad transaction number with the actual<br/> transaction number being lower than client's one which, according the the ASSERT, is bad. </p> <p>I could see there is a similar bug (bz23296) but I don't think this bug is related with this one,<br/> as in bz23296 the problem comes from a bad initialization in obdecho/echo_client.c which is used<br/> only for tests, not for production as it's our case.</p> <p>Does this sound as a known bug for you? In order to work-around this bug, what would be the<br/> consequences of disabling this LBUG? I mean, I think we would loss some data on a client but I<br/> don't know if there is any other important consequence.</p> <p>I also attach here the patch from bugzilla 24420 that is already landed in 1.8.6.</p> <p>Thanks,<br/> Sebastien.</p> x86_64, RHEL6 LU-128

OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd>=le64_to_cpu(lcd->lcd_last_transno)) failed] in recovery

Bug Major Resolved Fixed Niu Yawei Sebastien Buisson Tue, 15 Mar 2011 01:49:22 +0000 Thu, 25 Aug 2011 23:38:45 +0000 Tue, 14 Jun 2011 08:06:37 +0000 Lustre 2.0.0 Lustre 2.1.0 0 4 <p>&gt; lcd_last_transno=0x4ddebb<br/> &gt; oti_transno=last_rcvd=0x4ddeba<br/> &gt; lsd_last_transno=0x4de0ee<br/> &gt;<br/> &gt; So we have the client (lcd_last_transno) having a bad transaction number with the actual<br/> &gt; transaction number being lower than client's one which, according the the ASSERT, is bad.</p> <p>This is not correct, lcd_last_transno is not client transno but the latest transno written on server, it is stored in 'client_data' per export. Given the fact it is on disk that said it was committed, therefore all lower transactions must be committed as well. But client tries to replay 0x4ddeba transaction which is bad and shouldn't happen.</p> <p>Therefore we have the following possible cases:<br/> 1) bug in client replay code - it issues lower transaction after higher<br/> 2) bug on server side - it should skip such lower transno if server committed trasno is higher</p> <p>I doubt we have case 1) as it is generic code for MDS/OSS. Also this happens always with IO replays, maybe something is wrong in that path. I'd propose to track how such replays are handled on server.</p> <p>the patch is correct but only to avoid assertion on wire data, the bug itself is not fixed and still exists, that is fundamentally wrong to try to execute the lower transaction request. Do you have lustre log about that crash?</p> <p>Thanks for your feedback Mikhail,</p> <p>We asked our client to activate this debug levels:</p> <ul class="alternate" type="square"> <li>On OSS: HA and warning</li> <li>On MDS: HA, inode and warning</li> <li>On clients: nothing in particular</li> </ul> <p>They cannot set full debug level as their fs is in production. Do you think this will be enough trying to trace trnasno's?</p> <p>Hi,</p> <p>The patch is only considered as a work-around for us, but it is vital because without it production is completely blocked at customer site.</p> <p>Unfortunately we do not have the lustre debug logs, all the information we have is in bugzilla 24420.</p> <p>Regards,<br/> Sebastien.</p> <p>Diego, HA should help a lot.</p> <p>Sebastien, am I right that bug was seen only with 1.8.x so far?</p> <p>No, our customer is installed with Lustre 2.0.0.1, this is where we have seen this issue.</p> <p>Niu</p> <p>Could you please look at this one?</p> <p>thanks</p> <p>Peter</p> <p>Hi, tappro</p> <p>Though the 0x4ddebb was already committed, we can't guarantee that the reply with last_commited_trasno &gt;= 0x4ddebb arrived client before server crash, so the replay requests probably were still in the client's replay list, and they should be replayed on recovery, did I miss anything? </p> <p>To my understanding, such kind of assert is incorrect (the first replay request might have lower transno), we can just simply remove it.</p> <p>upon connect the real last_committed is reported back to the client, so it shouldn't send lower replays. That is the first questions why that doesn't work?</p> <p>Also server always can compare them during replay and must drop them because processing them breaks consistency. The assertion is set in filter_finish_transno which is endpoint, it shouldn't go so far.</p> <p>Note that this assertion was there for quite long time and all cases are about bulk replay which was introduced recently. I think this is root of problem, we need to review that paths, what last_committed value client has after reconnect? What server values for next_transno, committed, current at the moment of replay? After that it will be more clear what is wrong. </p> <p>in target_queue_recovery_request() we have exclusion for lower transno requests:</p> <p> if (transno &lt; obd-&gt;obd_next_recovery_transno) </p> { /* Processing the queue right now, don't re-add. */ LASSERT(cfs_list_empty(&amp;req-&gt;rq_list)); cfs_spin_unlock(&amp;obd-&gt;obd_recovery_task_lock); RETURN(1); } <p>that is done for open replays which can be older than committed values, but such requests are specially kept on client. That makes sense only on MDS and I am not sure about bulk replay - maybe they also can be kept even if lower than committed for some reason? Though I can't imagine any.</p> <p>So the only question remains why bulk replays are possible with lower transno, see my previous comment about last_committed which must be applied after reconnect.</p> <p>Right, client should get last_committed upon connect and cleanup all the requests with lower transactions, I missed that part. Will rethink it. Thank you, Tappro.</p> <p>1) Such replays will be replayed during replay phase, there is no mechanism to replay something after replay phase. It is still in replay queue and will be replayed again at first place (not committed on server) or freed if committed. The server last_committed value must be equal to its transno or less.</p> <p>2) The replay errors causes import invalidation, client is evicted from server. Anyway we have no assertion for '==' case, it is allowed.</p> <p>That doesn't look as source of problem. I suspect the source of problem is in client code, which replays requests with transno lesser than last_committed from server. I don't know how yet, but that is good point to start with.</p> <p>The problem is not in assertion, this is just fundamentally wrong to replay lesser transno over greater one. </p> <p>Hmm, whenever a replay request get error or timeout (lost reply), the ptlrpc_replay_interpret() calls ptlrpc_connect_import() to restart recover, if that replay wasn't committed before the reconnect replied, then client will resend it as the first replay request upon reconnect. The transno of this resent replay might be less than the 'obd_next_recovery_transno', so it'll be processed immediately. For filter, this looks like a defect to me, because OSS doesn't check the non-idempotent resent request as MDS does, however, it shouldn't trigger the LBUG, because the transno is at most equal to lcd_last_transno in this case. </p> <p>I can't imagine the case of client sending lower transno niether, maybe some more log would be helpful.</p> <p>Hi, Tappro</p> <p>It looks to me there is a serious race in target_handle_connect() which could casuse this LBUG.</p> <p>In target_handle_connect(), it'll check the obd_recovering to see if we're in recovery state, if it's true, then MSG_CONNECT_RECOVERING will packed in reply to inform the client to start replay, however, recovery might be finished just after this checking/setting done, at the end, this client will be allowed to do replay after recovery window. </p> <p>Do you agree with me on this? If you think it's a race either, I'll make a patch to fix it soon. Thanks.</p> <p>Niu, that shouldn't be. Recovery is waiting for all clients to start and cannot end until connected client will finish recovery. That said if client was connected then 1) it will be added to recovery 2) if recovery stops before that then this client will be evicted as missed recovery window. I see no way how it can do replays after recovery window.</p> <p>Please note that this bug would be seen much often if exits but we start to see that bug only recently and only with bulk replays. I think that OSS recovery is right place to investigate.</p> <p>Hi, Diego &amp; Sebastien</p> <p>Is this LBUG reproduced with the HA and WARNING logs? So far, I don't see how it can be triggered with only code inspection, if there is any logs indicating what's the recovery stage it was on and if any client was evicted, that'll be helpful.</p> <p>Thanks<br/> Niu</p> <p>Hi Niu,</p> <p>We have asked our on-site Support Team for the logs, but unfortunately we have not been sent them.</p> <p>Sebastien.</p> <p>Dropping priority to reflect importance to CEA. Please just attach logs when you are able to gather them and we will see what we can do then.</p> <p>Peter, does it make sense to land 24420 patch on master, it is safe and replaces assertions on wire data. I have updated version of patch so it will require just inspections and testing</p> <p>Mike</p> <p>Yes, Oleg is ok with this approach. Please upload the patch to gerrit so that we can get the inspections underways</p> <p>Thanks</p> <p>Peter</p> <p>Mike</p> <p>Are you able to hand off your patch to Niu so he can complete the testing and landing of this patch?</p> <p>Peter</p> <p>Peter, I've just added patch to gerrit for review. Local tests are passed.</p> <p>thanks Mike!</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_open.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_recovery.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=ofa/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,ofa #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/mdt/mdt_recovery.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_recovery.c</li> <li>lustre/mdt/mdt_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/mdt/mdt_recovery.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/117/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #117</a><br/> <a href="https://jira.whamcloud.com/browse/LU-128" title="OSSs frequent crashes due to LBUG/[ASSERTION(last_rcvd&gt;=le64_to_cpu(lcd-&gt;lcd_last_transno)) failed] in recovery" class="issue-link" data-issue-key="LU-128"><del>LU-128</del></a> Avoid assertion on wire data in last_rcvd update</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=2bb3a7f6b9889af696485267eb254db7980fe193" class="external-link" target="_blank" rel="nofollow noopener">2bb3a7f6b9889af696485267eb254db7980fe193</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> <li>lustre/mdt/mdt_open.c</li> <li>lustre/mdt/mdt_recovery.c</li> </ul> <p>The patch has been installed on our client's cluster, and he started to get MDS crashes<br/> for LBUG/ASSERTION(req_is_replay(req)) failed.</p> <p>The MDS Panic thread stack-trace looks like following :<br/> =======================================================<br/> panic()<br/> lbug_with_loc()<br/> libcfs_assertion_failed()<br/> mdt_txn_stop_cb()<br/> dt_txn_hook_stop()<br/> osd_trans_stop()<br/> mdd_trans_stop()<br/> mdd_create()<br/> cml_create()<br/> mdt_reint_open()<br/> mdt_reint_rec()<br/> mdt_reint_internal()<br/> mdt_intent_reint()<br/> mdt_intent_policy()<br/> ldlm_lock_enqueue()<br/> ldlm_handle_enqueue0()<br/> mdt_enqueue()<br/> mdt_handle_common()<br/> mdt_regular_handle()<br/> ptlrpc_server_handle_request()<br/> ptlrpc_main()<br/> kernel_thread()<br/> =======================================================</p> <p>Looking at the stack trace, the failing ASSERTION(req_is_replay(req)) is likely to come from the fix in /lustre/mdt/mdt_recovery.c.</p> <p>It appears some scenario are still not covered by the patch.</p> <p>Could you have a look ?<br/> Thanks,</p> <p>Grégoire.</p> <p>Can you provide more information about setup, what are versions on clients and servers? When does bug occur - immediately after start or occasionally during normal work?</p> <p>It is the same cluster than initial problem (Lustre 2.0.0, x86_64, RHEL6).</p> <p>Here is the information from the client:<br/> The 1st occurence was during normal operations, and next ones during restart+recovery.<br/> And no msgs of connection problems nor client eviction at that time ...</p> <p>The patch has been removed from the cluster.</p> <p>That reason can be other just 2.0.0 code itself, some bug which causes this assertion. The assert can be removed from patch, in that case there will be client evictions only and more debug info about why that is happening.</p> <p>Adding as 2.1 blocker upon advice of Bull because this patch is landed for 2.1 and caused issues when deployed in production at CEA.</p> <p>Peter, it is not quite correct, patch was landed for 2.1 but problems with it were seen with 2.0.0. I am afraid the differences between 2.1 and 2.0.0 can be the reason for this. So it is not blocker for 2.1 at least we saw no issues with it so far, but we need patch which works with 2.0.0 correctly. </p> <p>I'd propose to cook special patch for Bull with removed assertions just to make it loyal to any error and we will be able to see debug messages if issues will occur again.</p> <p>Ah thanks for clarifying Mike. This can certainly remain an important support issue for CEA and a priority for us without being considered a 2.1 blocker. I have adjusted the status accordingly.</p> <p>Mike</p> <p>Are you still expecting to be able to create a patch based on 2.0 for CEA?</p> <p>CEA,</p> <p>Would you deploy such a patch or is the window until you rebase on 2.1 small enough that it would not be worthwhile?</p> <p>Peter</p> <p>Peter,</p> <p>As suggested by Mike we cooked a patch for CEA with removed assertions just to make it loyal to any error. Also these error messages are deactivated by default, but can be activated via a kernel module option. That way CEA will be able to collect debug messages when the issue reoccurs.</p> <p>Sebastien.</p> <p>ok then I think that we can close this ticket and reopen it if it transpires that we need to take any further action before CEA realign on 2.1</p> <p>I agree.</p> <p>Thank you,<br/> Sebastien.</p> <blockquote> <p>The patch has been installed on our client's cluster, and he started to get MDS crashes<br/> for LBUG/ASSERTION(req_is_replay(req)) failed.</p> <p>The MDS Panic thread stack-trace looks like following :<br/> =======================================================<br/> panic()<br/> lbug_with_loc()<br/> libcfs_assertion_failed()<br/> mdt_txn_stop_cb()<br/> dt_txn_hook_stop()<br/> osd_trans_stop()<br/> mdd_trans_stop()<br/> mdd_create()<br/> cml_create()<br/> mdt_reint_open()<br/> mdt_reint_rec()<br/> mdt_reint_internal()<br/> mdt_intent_reint()<br/> mdt_intent_policy()<br/> ldlm_lock_enqueue()<br/> ldlm_handle_enqueue0()<br/> mdt_enqueue()<br/> mdt_handle_common()<br/> mdt_regular_handle()<br/> ptlrpc_server_handle_request()<br/> ptlrpc_main()<br/> kernel_thread()<br/> =======================================================</p> <p>Looking at the stack trace, the failing ASSERTION(req_is_replay(req)) is likely to come from the fix in /lustre/mdt/mdt_recovery.c.</p></blockquote> <p>This should be fixed in <a href="https://jira.whamcloud.com/browse/LU-617" title="LBUG: (mdt_recovery.c:787:mdt_last_rcvd_update()) ASSERTION(req_is_replay(req)) failed" class="issue-link" data-issue-key="LU-617"><del>LU-617</del></a>.</p> Bugzilla ID 24420.0 Development Rank 1|hzv9an: Rank (Obsolete) 5040 Severity