Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-2232] LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG https://jira.whamcloud.com/browse/LU-2232 Lustre <p>Last night we had two OSSs panic at virtually the same time with and LBUG error being thrown. We just updated our servers and clients to 2.1.2-4chaos from 2.1.2-3chaos releases with the past 2 days and had not experienced this issue with the previous release. Below is a sample of the console log from one of the servers. I have also captured all the console messages up until the system panicked and am attaching it. </p> <p>LustreError: 9044:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock-&gt;l_export == opd-&gt;opd_exp) failed<br/> LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock-&gt;l_export == opd-&gt;opd_exp) failed<br/> LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG<br/> Pid: 9120, comm: ll_ost_io_341</p> <p>Call Trace:<br/> LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock-&gt;l_export == opd-&gt;opd_exp) failed<br/> LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG<br/> <span class="error">&#91;&lt;ffffffffa0440895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Pid: 9083, comm: ll_ost_io_304</p> Dell R710 servers running TOSS-2.0-2 and DDN 10k storage. LU-2232 LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG Bug Major Resolved Cannot Reproduce Lai Siyao Joe Mervini Thu, 25 Oct 2012 12:24:33 +0000 Mon, 24 Aug 2015 21:57:34 +0000 Mon, 24 Aug 2015 21:57:34 +0000 Lustre 2.4.0 Lustre 2.1.2 0 18 <p>Oleg is looking into this one</p> <p>Do you have a crash dump? Can you print the lock content by any chance?</p> <p>Unfortunately, no. Our servers are diskless so when it dumps core it is lost on reboot. I have linked /var/crash to a NFS mounted device so hopefully if the LBUG occurs again it will write to that device before freezing. </p> <p>As far as printing the lock content, I'm not sure what you mean. If you can tell me how to gather that information I will be happy to if I can.</p> <p>Disklessness does not matter.<br/> You just need to configure kdump to dump kernel core to an nfs share (no need to mount it at runtime, the debug kernel will do it separately).<br/> The lock content you will get out of the crashdump once you have it.<br/> How repeatable is this?</p> <p>We should probably close this issue. We decided to roll back to the previous version during a system outage last week to avoid running into the issue again in the short term. We have another system that is preproduction that is running the current version so we will monitor that and report back if we encounter the problem again. </p> <p>ok thanks Joe!</p> <p>Xyratex hit that bug at own branch..</p> <p>Cray just hit this in our 2.5 with SLES11SP3 clients on CentOS 6.4 servers, twice.</p> <p>Both times, it was immediately preceded by a 'double eviction' of a particular client, two callback timers expiring within the same second for the same client:</p> <p>2014-02-01 13:44:35 LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 117s: evicting client at 54@gni1 ns: filter-esfprod-OST0004_UUID lock: ffff8803fa547980/0xd460be1680c1b564 lrc: 3/0,0 mode: PW/PW res: <span class="error">&#91;0x424b478:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;4095) flags: 0x60000000010020 nid: 54@gni1 remote: 0xf4283d6d3305e0e5 expref: 2679 pid: 22955 timeout: 4994587934 lvb_type: 0<br/> 2014-02-01 13:44:35 LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 117s: evicting client at 54@gni1 ns: filter-esfprod-OST0004_UUID lock: ffff8802c669d100/0xd460be1680c1b580 lrc: 3/0,0 mode: PW/PW res: <span class="error">&#91;0x424b479:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;4095) flags: 0x60000000010020 nid: 54@gni1 remote: 0xf4283d6d3305e155 expref: 2679 pid: 22955 timeout: 4994587934 lvb_type: 0</p> <p>And the other instance of it (on a different system, also running Cray's 2.5):<br/> LustreError: 3489:0:(ldlm_lockd.c:433:ldlm_add_waiting_lock()) Skipped 1 previous similar message<br/> LustreError: 53:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 113s: evicting client at 62@gni<br/> ns: filter-scratch-OST0001_UUID lock: ffff880445551180/0xa9b58315ab2d4d0a lrc: 3/0,0 mode: PW/PW res: <span class="error">&#91;0xb5da7e1:0x0:0x0&#93;</span>.0 rrc:<br/> 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x60000000010020 nid: 62@gni remote: 0x330a8b1d9fd47938<br/> expref: 1024 pid: 16057 timeout: 4781830324 lvb_type: 0</p> <p>LustreError: 53:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 113s: evicting client at 62@gni<br/> ns: filter-scratch-OST0001_UUID lock: ffff880387977bc0/0xa9b58315ab2d4d6c lrc: 3/0,0 mode: PW/PW res: <span class="error">&#91;0xb5da7e6:0x0:0x0&#93;</span>.0 rrc:<br/> 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x60000000010020 nid: 62@gni remote: 0x330a8b1d9fd47bfb<br/> expref: 1030 pid: 16082 timeout: 4781830324 lvb_type: 0</p> <p>I'm just speculating, but I suspect this is key. </p> <p>For some reason, the nid_stats struct pointer in the obd_export is zero, so I wasn't able to confirm if this export was actually the double-evicted client. (Perhaps this is always the case?)</p> <p><span class="error">&#91;Edited to clarify server and client versions. Also, we&#39;ve seen this several times since. Not sure why we&#39;re suddenly seeing it now, as indications are the underlying bug has been in the code for some time.&#93;</span></p> <p>Lai, Jodi,</p> <p>I'm happy to see this bug re-opened, but I'm curious why? Was it seen again at Intel or a customer site? Was it re-opened because of the Cray report?</p> <p>Patrick, this is seen again in <a href="https://jira.whamcloud.com/browse/LU-4844" title="ost_prolong_lock_one()) ASSERTION( lock-&gt;l_export == opd-&gt;opd_exp )" class="issue-link" data-issue-key="LU-4844"><del>LU-4844</del></a>, and it's better to re-open this one and mark <a href="https://jira.whamcloud.com/browse/LU-4844" title="ost_prolong_lock_one()) ASSERTION( lock-&gt;l_export == opd-&gt;opd_exp )" class="issue-link" data-issue-key="LU-4844"><del>LU-4844</del></a> as duplicate.</p> <p>Thank you, Lai.</p> <p>Cray can provide a dump of an OST with this crash, if you think that would be helpful. </p> <p>We're currently trying to replicate it with D_RPCTRACE enabled, so if we get a dump with that, we can send that over as well. (We're working with Xyratex on this bug.)</p> <p>We replicated this on an OSS with +rpctrace enabled. I've provided the dump to Xyratex, and it will be uploaded here in about 5 minutes:<br/> ftp.whamcloud.com<br/> uploads/LELUS-2232/LELUS-234_<a href="https://jira.whamcloud.com/browse/LU-2232" title="LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG" class="issue-link" data-issue-key="LU-2232"><del>LU-2232</del></a>.1404081304.tar.gz</p> <p>The backtrace shows it LBUG on the first ost_prolong_lock_one() in ost_prolong_locks(), IMO what happened is like this:<br/> 1. client did IO with lock handle in the request.<br/> 2. IO bulk failed on server, so no reply to client to let it resend.<br/> 3. lock cancelling timed out on server, the client was evicted.<br/> 4. client reconnected and resent previous IO request, however the lock handle is obsolete, so the LASSERT was triggered. (this lock should have been replayed, but this request was simply resent, there is no way to update lock handle in the resent request)</p> <p>I'll provide a patch to check lock-&gt;l_export against opd-&gt;opd_exp other than assert for the first ost_prolong_lock_one().</p> <p>Patches are ready:<br/> 2.4: <a href="http://review.whamcloud.com/#/c/9925/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9925/</a><br/> 2.5: <a href="http://review.whamcloud.com/#/c/9926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9926/</a><br/> master: <a href="http://review.whamcloud.com/#/c/9927/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9927/</a></p> <p>Question - if the client was evicted - the request resending should have failed because the server would reject it.<br/> So there's something else happening that your explanation does not quite explain I think.</p> <p>I'm afraid some requests were not aborted upon eviction on client, so that they were resent through new connection. I'll make some test to find more proof.</p> <p>I did some test but couldn't reproduce this failure, so I updated the patch for 2.4/2.5 to debug patch which will print lock and request export address and crash as before, so that we can dump these two exports to help analyse.</p> <p>Could you apply the patch and reproduce it? and if it crashes upload the crash dump.</p> <p>I think <a href="https://jira.whamcloud.com/browse/LU-5116" title="Race between resend and reply processing" class="issue-link" data-issue-key="LU-5116"><del>LU-5116</del></a> might be related here to explan resend across eviction</p> <p>It would be a sensible approach to try both the fix from <a href="https://jira.whamcloud.com/browse/LU-5116" title="Race between resend and reply processing" class="issue-link" data-issue-key="LU-5116"><del>LU-5116</del></a> and the diagnostic patch from <a href="https://jira.whamcloud.com/browse/LU-2232" title="LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG" class="issue-link" data-issue-key="LU-2232"><del>LU-2232</del></a>. That way perhaps the issue would be fixed by <a href="https://jira.whamcloud.com/browse/LU-5116" title="Race between resend and reply processing" class="issue-link" data-issue-key="LU-5116"><del>LU-5116</del></a> but if there is still a residual problem we will have fuller information to go forward on.</p> <p>Lai - Sorry we didn't update this earlier. Cray tried to reproduce this bug with your debug patch, and were unable to do so. After that, we pulled your patch set 1 in to our Lustre version and haven't seen this bug since.</p> <p>Patrick</p> <p>To be clear, Cray could reliably reproduce this issue, then applied the diagnostic patch and could not? What code line was this on?</p> <p>Thanks</p> <p>Peter</p> <p>Peter,</p> <p>That's correct. It was on b2_5.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>thanks Patrick. Wow. That is odd.</p> <p>Lai - Given that the code in this area was changed a fair bit in 2.6, do you think this bug is likely to still be present?</p> <p>Per Oleg's comment, this may be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-5116" title="Race between resend and reply processing" class="issue-link" data-issue-key="LU-5116"><del>LU-5116</del></a>, I've tried to verify it by adding some code and test, but not have enough time to finish it. I will keep an eye on it, and at the moment we can mark it resolved.</p> <p>Lai - Thanks for the answer. We'll go with that and I'll let you know if Cray sees any issues. (We're just moving in to internal deployment of 2.6.)</p> <p>The debug message hit on an MDS. Here it is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-04-24 09:00:42 Lustre: lsd-OST0000: haven't heard from client 6048f9e4-4557-a727-5b7d-dd9f7cb9c8b1 (at 192.168.116.115@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805cbcd5000, cur 1429891242 expire 1429891092 last 1429891015 2015-04-24 09:14:05 LustreError: 0:0:(ldlm_lockd.c:355:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 172.19.1.85@o2ib100 ns: filter-lsd-OST0000_UUID lock: ffff880bf976a880/0x7e1b5086f6ebf35e lrc: 3/0,0 mode: PR/PR res: [0x2678a59:0x0:0x0].0 rrc: 266 type: EXT [0-&gt;18446744073709551615] (req 468705280-&gt;468717567) flags: 0x60000000000020 nid: 172.19.1.85@o2ib100 remote: 0xa9a53ce40d684d49 expref: 54562 pid: 6158 timeout: 5072055798 lvb_type: 0 2015-04-24 09:14:05 LustreError: 22433:0:(ldlm_lockd.c:2342:ldlm_cancel_handler()) ldlm_cancel from 172.19.1.85@o2ib100 arrived at 1429892045 with bad export cookie 9086945213692644542 2015-04-24 09:14:05 LustreError: 6684:0:(ldlm_lock.c:2448:ldlm_lock_dump_handle()) ### ### ns: filter-lsd-OST0000_UUID lock: ffff880d7abe3680/0x7e1b5086f3c622c3 lrc: 3/0,0 mode: PR/PR res: [0x24bf940:0x0:0x0].0 rrc: 3 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;18446744073709551615) flags: 0x40000000000000 nid: 172.19.1.85@o2ib100 remote: 0xa9a53ce40617fec7 expref: 54547 pid: 6183 timeout: 0 lvb_type: 1 2015-04-24 09:14:05 LustreError: 22433:0:(ldlm_lockd.c:2342:ldlm_cancel_handler()) Skipped 1 previous similar message 2015-04-24 09:18:04 LustreError: 6784:0:(ost_handler.c:1956:ost_prolong_locks()) ### LU-2232: lock export ffff880e2d590400 != req export ffff880d8a2eb400, this lock is obsolete! 2015-04-24 09:18:04 ns: filter-lsd-OST0000_UUID lock: ffff880bf976a880/0x7e1b5086f6ebf35e lrc: 4/0,0 mode: PR/PR res: [0x2678a59:0x0:0x0].0 rrc: 135 type: EXT [0-&gt;18446744073709551615] (req 468705280-&gt;468717567) flags: 0x60000000000020 nid: 172.19.1.85@o2ib100 remote: 0xa9a53ce40d684d49 expref: 35636 pid: 6158 timeout: 5072055798 lvb_type: 0 </pre> </div></div> <p>No assertion, though. Lustre 2.5.3-5chaos.</p> <p>The log shows after client eviction (and quite possibly reconnected), a stale lock handle is packed in client rw request.</p> <p>I added some code to try to simulate this, but failed. According to the code, after eviction, client inflight RPCs will be aborted, and locks be cleaned up. If a full debug log covering this recovery(at least on client) can be obtained upon this error message is seen, it can help move this forward.</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14950" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2232" title="LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG" class="issue-link" data-issue-key="LU-2232"><del>LU-2232</del></a> debug: print debug for prolonged lock<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 1d21596de85b81bb3f98277f8f0b425368ccb187</p> <p>The updated debug patch will print both lock and request details, which can tell us whether this request is new, or resent/replay request.</p> <p>I have pulled change 14950, Patch Set 1, into LLNL's local tree. It is in the queue to go into the next TOSS release and eventually roll out into production.</p> <p>We have not seen any more occurrences of this error since we rolled out our 2.5.4-4chaos version into production.</p> <p>Thanks Marc!</p> Duplicate LU-4844 Related LU-5116 Development Rank 1|hzvatz: Rank (Obsolete) 5290 Severity