https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5264 Lustre <p>While stopping a Lustre filesystem, the following LBUG occurred on an OSS:</p> <p>----8&lt; ----<br/> LustreError: 4581:0:(osd_handler.c:5343:osd_key_exit()) ASSERTION( info-&gt;oti_r_locks == 0 ) failed:<br/> Lustre: server umount scratch3-OST0130 complete<br/> LustreError: 4581:0:(osd_handler.c:5343:osd_key_exit()) LBUG<br/> Pid: 4581, comm: ll_ost00_070</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0d8c895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d8ce97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa14da67b&gt;&#93;</span> osd_key_exit+0x5b/0xc0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e5d9f8&gt;&#93;</span> lu_context_exit+0x58/0xa0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffd749&gt;&#93;</span> ptlrpc_main+0xa59/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Kernel panic - not syncing: LBUG<br/> Pid: 4581, comm: ll_ost00_070 Tainted: G W --------------- 2.6.32-431.11.2.el6.Bull.48.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff81528393&gt;&#93;</span> ? panic+0xa7/0x16f<br/> <span class="error">&#91;&lt;ffffffffa0d8ceeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa14da67b&gt;&#93;</span> ? osd_key_exit+0x5b/0xc0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e5d9f8&gt;&#93;</span> ? lu_context_exit+0x58/0xa0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffd749&gt;&#93;</span> ? ptlrpc_main+0xa59/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ffccf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> ----8&lt; ----</p> <p>There were 15 OSTs mounted on the OSS. One umount process has completed while others were still present at crash time. All umount processes were running in parallel because of Shine (shine stop -f scratch3 -n @io)</p> <p>----8&lt; ----<br/> crash&gt; ps | grep umount<br/> 21639 21636 25 ffff88084445e080 IN 0.0 105176 760 umount<br/> 21642 21638 25 ffff8807e0f86b40 IN 0.0 105176 760 umount<br/> 21643 21637 26 ffff880b6a78e100 IN 0.0 105176 760 umount<br/> 21646 21640 25 ffff880c789fcb40 IN 0.0 106068 756 umount<br/> 21649 21644 2 ffff880f9c9740c0 IN 0.0 105176 760 umount<br/> 21651 21645 17 ffff88083ecfb580 IN 0.0 106068 756 umount<br/> 21653 21648 15 ffff880f97935580 IN 0.0 105176 760 umount<br/> 21655 21650 6 ffff880fc5c554c0 IN 0.0 105176 760 umount<br/> 21657 21652 25 ffff881076294080 IN 0.0 105176 756 umount<br/> 21659 21654 19 ffff880f8f245500 IN 0.0 106068 760 umount<br/> 21661 21656 11 ffff8807ec1214c0 IN 0.0 105176 764 umount<br/> 21663 21660 30 ffff8808122a9500 IN 0.0 106068 764 umount<br/> 21664 21658 3 ffff880b3d1d7500 IN 0.0 105176 764 umount<br/> 21665 21662 5 ffff8807ec120a80 IN 0.0 106068 764 umount<br/> ----8&lt; ----</p> <p>Backtrace of the process:</p> <p>----8&lt; ----<br/> crash&gt; bt<br/> PID: 4581 TASK: ffff881004af1540 CPU: 8 COMMAND: "ll_ost00_070"<br/> #0 <span class="error">&#91;ffff881004af3cb8&#93;</span> machine_kexec at ffffffff8103915b<br/> #1 <span class="error">&#91;ffff881004af3d18&#93;</span> crash_kexec at ffffffff810c5e42<br/> #2 <span class="error">&#91;ffff881004af3de8&#93;</span> panic at ffffffff8152839a<br/> #3 <span class="error">&#91;ffff881004af3e68&#93;</span> lbug_with_loc at ffffffffa0d8ceeb <span class="error">&#91;libcfs&#93;</span><br/> #4 <span class="error">&#91;ffff881004af3e88&#93;</span> osd_key_exit at ffffffffa14da67b <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #5 <span class="error">&#91;ffff881004af3e98&#93;</span> lu_context_exit at ffffffffa0e5d9f8 <span class="error">&#91;obdclass&#93;</span><br/> #6 <span class="error">&#91;ffff881004af3eb8&#93;</span> ptlrpc_main at ffffffffa0ffd749 <span class="error">&#91;ptlrpc&#93;</span><br/> #7 <span class="error">&#91;ffff881004af3f48&#93;</span> kernel_thread at ffffffff8100c20a<br/> crash&gt; bt -l 4581<br/> PID: 4581 TASK: ffff881004af1540 CPU: 8 COMMAND: "ll_ost00_070"<br/> #0 <span class="error">&#91;ffff881004af3cb8&#93;</span> machine_kexec at ffffffff8103915b<br/> /usr/src/debug/kernel-2.6/linux-2.6.32-431.11.2.el6.Bull.48.x86_64/arch/x86/kernel/machine_kexec_64.c: 336<br/> #1 <span class="error">&#91;ffff881004af3d18&#93;</span> crash_kexec at ffffffff810c5e42<br/> /usr/src/debug/kernel-2.6/linux-2.6.32-431.11.2.el6.Bull.48.x86_64/kernel/kexec.c: 1106<br/> #2 <span class="error">&#91;ffff881004af3de8&#93;</span> panic at ffffffff8152839a<br/> /usr/src/debug/kernel-2.6/linux-2.6.32-431.11.2.el6.Bull.48.x86_64/kernel/panic.c: 111<br/> #3 <span class="error">&#91;ffff881004af3e68&#93;</span> lbug_with_loc at ffffffffa0d8ceeb <span class="error">&#91;libcfs&#93;</span><br/> /usr/src/debug/lustre-2.4.3/libcfs/libcfs/linux/linux-debug.c: 176<br/> #4 <span class="error">&#91;ffff881004af3e88&#93;</span> osd_key_exit at ffffffffa14da67b <span class="error">&#91;osd_ldiskfs&#93;</span><br/> /usr/src/debug/lustre-2.4.3/lustre/osd-ldiskfs/osd_handler.c: 5345<br/> #5 <span class="error">&#91;ffff881004af3e98&#93;</span> lu_context_exit at ffffffffa0e5d9f8 <span class="error">&#91;obdclass&#93;</span><br/> /usr/src/debug/lustre-2.4.3/lustre/obdclass/lu_object.c: 1662<br/> #6 <span class="error">&#91;ffff881004af3eb8&#93;</span> ptlrpc_main at ffffffffa0ffd749 <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/debug/lustre-2.4.3/lustre/ptlrpc/service.c: 2514<br/> #7 <span class="error">&#91;ffff881004af3f48&#93;</span> kernel_thread at ffffffff8100c20a<br/> /usr/src/debug//////////////////////////////////////////////////////////////////kernel-2.6/linux-2.6.32-431.11.2.el6.Bull.48.x86_64/arch/x86/kernel/entry_64.S: 1235<br/> ----8&lt; ----</p> <p>You can find attached:</p> <ul class="alternate" type="square"> <li>dmesg.txt: log from the crash</li> <li>bt-all.merged.txt: merged foreach bt from the crash</li> </ul> RHEL6 w/ kernel 2.6.32-431.17.1.el6.x86_64 LU-5264

ASSERTION( info->oti_r_locks == 0 ) at OST umount

Bug Minor Resolved Fixed Bruno Faccini Bruno Travouillon p4b Fri, 27 Jun 2014 14:44:57 +0000 Thu, 14 May 2020 12:06:34 +0000 Wed, 20 May 2015 13:03:59 +0000 Lustre 2.4.3 Lustre 2.8.0 0 10 <p>Hello Bruno, nice to read from you even in a JIRA !!<br/> Is it a one shoot ? Also is there a crash-dump available for this (one of) occurrence ?</p> <p>Hi Bruno,</p> <p>Yes, AFAIK, this is a one shot.<br/> I won't be able to provide the crash-dump because the LBUG occurred on a blacksite. However, please tell me if you need some input, I will try to extract it from the dump.</p> <p>Humm after having a look to the assembly code of the routines in panic/LBUG stack, it will be difficult/painful for me to really detail the actions/crash-subcommands to get what I need from the crash-dump ... But let's try and 1st I would like to get both the "bt -f" and "bt -F" output for the panic/LBUG task, is it possible ?</p> <p>Also, this ticket duplicates <a href="https://jira.whamcloud.com/browse/LU-4776" title="suite sanity-scrub: ASSERTION( info-&gt;oti_r_locks == 0 )" class="issue-link" data-issue-key="LU-4776">LU-4776</a>, generated from auto-tests similar failures...</p> <p>Hi Bruno,</p> <p>You will find attached both 'bt -f' and 'bt -F' from the crash dump.</p> <p>HTH,</p> <p>Bruno</p> <p>Hello Bruno, can you get me more infos out from the crash-dump+site ?</p> <p>If yes, here is what I need :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>p/x lu_keys lu_env 0xffff88041e7c5d80 </pre> </div></div> <p>This last command will print you the lu_env struct containing the context ant its lc_value array of pointers address.</p> <p>And since current index of interest, in both lu_keys[] and lc_value, seems to be #21 ... :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>p/x *lu_keys[21] rd &lt;lc_value address&gt; 40 osd_thread_info &lt;lc_value[21] pointer value&gt; </pre> </div></div> <p>Don't forget to load the Lustre modules containing debuginfo stuff before with "mod -S &lt;<span class="error">&#91;debuginfo,modules&#93;</span> root dir&gt;".</p> <p>Hi Bruno,</p> <p>Here is the requested output. Sorry for the delay.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; p/x lu_keys $8 = {0xffffffffa0ec53e0, 0xffffffffa0ec7aa0, 0xffffffffa0ec6280, 0xffffffffa0ecd720, 0xffffffffa0eb2a20, 0xffffffffa10a39c0, 0xffffffffa0820ac0, 0xffffffffa090d320, 0xffffffffa11b77a0, 0xffffffffa11beac0, 0 xffffffffa11b9c60, 0xffffffffa1254960, 0xffffffffa12549a0, 0xffffffffa12ca080, 0xffffffffa12ca0c0, 0xffffffffa1391f20, 0xffffffffa1391f60, 0xffffffffa1393220, 0xffffffffa1393260, 0xffffffffa1438700, 0xffffff ffa14386c0, 0xffffffffa15197e0, 0xffffffffa1594f60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0} crash&gt; lu_env 0xffff88041e7c5d80 struct lu_env { le_ctx = { lc_tags = 2952790018, lc_state = LCS_LEFT, lc_thread = 0xffff88041227f140, lc_value = 0xffff88041e7c8e00, lc_remember = { next = 0xffff881057a58ac0, prev = 0xffff880848277d58 }, lc_version = 25, lc_cookie = 6 }, le_ses = 0x0 } crash&gt; p/x *lu_keys[21] $9 = { lct_tags = 0x400000c3, lct_init = 0xffffffffa14e0330, lct_fini = 0xffffffffa14db1a0, lct_exit = 0xffffffffa14da620, lct_index = 0x15, lct_used = { counter = 0x2 }, lct_owner = 0xffffffffa1526680, lct_reference = {&lt;No data fields&gt;} } crash&gt; rd 0xffff88041e7c8e00 40 ffff88041e7c8e00: ffff88041e7c8c00 0000000000000000 ..|............. ffff88041e7c8e10: ffff88041e7c9800 0000000000000000 ..|............. ffff88041e7c8e20: 0000000000000000 ffff88041e7cac00 ..........|..... ffff88041e7c8e30: ffff880418c30d40 ffff88041e7c5cc0 @........\|..... ffff88041e7c8e40: ffff88041e7c7dc0 0000000000000000 .}|............. ffff88041e7c8e50: ffff88041e7c8a00 0000000000000000 ..|............. ffff88041e7c8e60: 0000000000000000 0000000000000000 ................ ffff88041e7c8e70: 0000000000000000 0000000000000000 ................ ffff88041e7c8e80: 0000000000000000 0000000000000000 ................ ffff88041e7c8e90: 0000000000000000 0000000000000000 ................ ffff88041e7c8ea0: 0000000000000000 0000000000000000 ................ ffff88041e7c8eb0: 0000000000000000 0000000000000000 ................ ffff88041e7c8ec0: 0000000000000000 0000000000000000 ................ ffff88041e7c8ed0: 0000000000000000 0000000000000000 ................ ffff88041e7c8ee0: 0000000000000000 0000000000000000 ................ ffff88041e7c8ef0: 0000000000000000 0000000000000000 ................ ffff88041e7c8f00: 0000000000000000 0000000000000000 ................ ffff88041e7c8f10: 0000000000000000 0000000000000000 ................ ffff88041e7c8f20: 0000000000000000 0000000000000000 ................ ffff88041e7c8f30: 0000000000000000 0000000000000000 ................ </pre> </div></div> <p>There is no &lt;lc_value<span class="error">&#91;21&#93;</span> pointer value&gt; here... Did I miss something?</p> <p>We had a new occurrence of this bug last week on the same OSS. I will try to get the output from the new dump.</p> <p>Yes, this is strange because you should have crashed with a Kernel Oops/BUG() instead of an LBUG ...<br/> But this may also be the result of a race where lc_value<span class="error">&#91;21&#93;</span> should have still been populated when referenced, causing the LBUG thread to trigger the Assert when the concurent/racing thread may the have zero'ed it ...<br/> BTW, the "good" news is that during the testing of an other patch where I was running MDT/OST mount/umounts in a loop, I have triggered the same LBUG. And its 1st crash analysis steps show the same strange behavior !!! More to come soon.</p> <p>I have spent sometime this week doing more analysis on my own crash-dump and I should be able to push a patch soon now to fix the suspected race during umounts.</p> <p>BTW, can you check in your on-site crash-dump/logs that this was the last OST to be unmounted ?</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13103" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13103</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> obdclass: fix race during key quiescency<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 66dcaa9824f4ae35fb375bc9f9da18c481b7a8f7</p> <p>After more debugging on my own crash-dump, I think problem comes from the fact that upon umount, presumably of last device using same OSD back-end, to prepare for module unload, lu_context_key_quiesce() is run to remove all module's key reference in any context linked on lu_context_remembered list. Thus threads must protect against such transversal processing when exiting from their context. <br/> Master patch to implement this is at <a href="http://review.whamcloud.com/13103" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13103</a>.</p> <p>Hi Bruno,</p> <p>I was just looking at the dumps at the customer site. For the first crash, there were still 14 umount processes running for 15 OSTs. For the second, 3 umount processes were remaining for 8 OSTs.</p> <p>I mean 14 umount processes running for 15 OSTs previously mounted, ie 1 OST has been successfully unmounted for first crash and 5 OSTs have been successfully unmounted for the second one.</p> <p>Thanks Bruno, but I believe this could be the effect of Shine tool running all umounts in parallel ...</p> <p>Bruno,</p> <p>Can we go ahead and backport the patch to b2_5?</p> <p>Humm master patch version, has successfully passed auto-test but definitelly needs to pass the review step. Will try to get more involvment from the reviewers. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13103/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13103/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> obdclass: fix race during key quiescency<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 42fdf8355791cb682c6120f7950bb2ecd50f97aa</p> <p>Landed for 2.8</p> <p>Grégoire Pichon (gregoire.pichon@bull.net) uploaded a new patch: <a href="http://review.whamcloud.com/15647" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15647</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> obdclass: fix race during key quiescency<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: cf26b808c1f1187a600de0674baf59061bdefc30</p> <p>Grégoire,</p> <p>Maybe you should abandon this backport to b2_5 because of <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a>. Backporting <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> would be a better answer to this issue.</p> Duplicate Related LU-4776 LU-6800 Development Rank 1|hzwq33: Rank (Obsolete) 14690 Severity