https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9806 Lustre <p>This seems to be a return of <a href="https://jira.whamcloud.com/browse/LU-7430" title="General protection fault: 0000 upon mounting MDT" class="issue-link" data-issue-key="LU-7430"><del>LU-7430</del></a> and a few other similar bugs, but happening on current master.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[291606.098200] Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 23:53:41 (1501300421) [291616.783248] Lustre: DEBUG MARKER: before: 623720 after_dd: 618600 took 1 seconds [291617.134646] LustreError: 28072:0:(osd_handler.c:2184:osd_ro()) *** setting lustre-OST0000 read-only *** [291617.152901] Turning device loop1 (0x700001) read-only [291617.224927] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [291617.277436] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [291617.590847] Lustre: Failing over lustre-OST0000 [291617.601802] LustreError: 22375:0:(tgt_lastrcvd.c:440:tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed: [291617.602975] LustreError: 22375:0:(tgt_lastrcvd.c:440:tgt_client_free()) LBUG [291617.603578] Pid: 22375, comm: obd_zombid [291617.604096] Call Trace: [291617.606669] [&lt;ffffffffa02857ce&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [291617.607349] [&lt;ffffffffa028585c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [291617.608122] [&lt;ffffffffa05ddde2&gt;] tgt_client_free+0x2a2/0x360 [ptlrpc] [291617.608814] [&lt;ffffffffa0db5b12&gt;] ofd_destroy_export+0x62/0x180 [ofd] [291617.609551] [&lt;ffffffffa0389239&gt;] obd_zombie_impexp_cull+0x549/0x920 [obdclass] [291617.622563] [&lt;ffffffffa038967d&gt;] obd_zombie_impexp_thread+0x6d/0x1c0 [obdclass] [291617.628967] [&lt;ffffffff810b7cc0&gt;] ? default_wake_function+0x0/0x20 [291617.629676] [&lt;ffffffffa0389610&gt;] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] [291617.631230] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [291617.631906] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [291617.632572] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [291617.633236] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [291617.639601] [291617.640036] Kernel panic - not syncing: LBUG [291617.640462] CPU: 4 PID: 22375 Comm: obd_zombid Tainted: P OE ------------ 3.10.0-debug #2 [291617.641354] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [291617.641830] ffffffffa02a4ed2 0000000025d32961 ffff8800a16b3cc0 ffffffff816fd3e4 [291617.642712] ffff8800a16b3d40 ffffffff816f8c34 ffffffff00000008 ffff8800a16b3d50 [291617.644582] ffff8800a16b3cf0 0000000025d32961 0000000025d32961 ffff88033e48d948 [291617.645811] Call Trace: [291617.646408] [&lt;ffffffff816fd3e4&gt;] dump_stack+0x19/0x1b [291617.647142] [&lt;ffffffff816f8c34&gt;] panic+0xd8/0x1e7 [291617.647765] [&lt;ffffffffa0285874&gt;] lbug_with_loc+0x64/0xb0 [libcfs] [291617.648540] [&lt;ffffffffa05ddde2&gt;] tgt_client_free+0x2a2/0x360 [ptlrpc] [291617.649224] [&lt;ffffffffa0db5b12&gt;] ofd_destroy_export+0x62/0x180 [ofd] [291617.649911] [&lt;ffffffffa0389239&gt;] obd_zombie_impexp_cull+0x549/0x920 [obdclass] [291617.651165] [&lt;ffffffffa038967d&gt;] obd_zombie_impexp_thread+0x6d/0x1c0 [obdclass] [291617.652377] [&lt;ffffffff810b7cc0&gt;] ? wake_up_state+0x20/0x20 [291617.653065] [&lt;ffffffffa0389610&gt;] ? obd_zombie_impexp_cull+0x920/0x920 [obdclass] [291617.654285] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [291617.654920] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 [291617.655610] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [291617.656262] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>Crasydump on onyx-68 in /exports/crashdumps/192.168.123.181-2017-07-28-23:53:59<br/> Modules also there.</p> LU-9806

tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed

Bug Major Resolved Fixed Alex Zhuravlev Oleg Drokin Sat, 29 Jul 2017 18:19:41 +0000 Wed, 19 Jul 2023 17:20:40 +0000 Wed, 19 Jul 2023 17:20:40 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.16.0 0 5 <p>Just had another one</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[11716.272157] Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) ================== 23:21:29 (1501384889) [11716.438161] Lustre: Failing over lustre-OST0000 [11716.527043] LustreError: 9005:0:(tgt_lastrcvd.c:440:tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed: [11716.528524] LustreError: 9005:0:(tgt_lastrcvd.c:440:tgt_client_free()) LBUG [11716.529497] Pid: 9005, comm: obd_zombid [11716.530209] Call Trace: [11716.532127] [&lt;ffffffffa02c57ce&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [11716.534315] [&lt;ffffffffa02c585c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [11716.535401] [&lt;ffffffffa061dde2&gt;] tgt_client_free+0x2a2/0x360 [ptlrpc] [11716.536214] [&lt;ffffffffa1412b12&gt;] ofd_destroy_export+0x62/0x180 [ofd] [11716.537110] [&lt;ffffffffa03c9239&gt;] obd_zombie_impexp_cull+0x549/0x920 [obdclass] [11716.551808] [&lt;ffffffffa03c967d&gt;] obd_zombie_impexp_thread+0x6d/0x1c0 [obdclass] [11716.553655] [&lt;ffffffff810b7cc0&gt;] ? default_wake_function+0x0/0x20 [11716.554770] [&lt;ffffffffa03c9610&gt;] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass] [11716.556332] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [11716.557232] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [11716.558361] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [11716.564715] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [11716.567093] [11716.568045] Kernel panic - not syncing: LBUG [11716.568703] CPU: 4 PID: 9005 Comm: obd_zombid Tainted: P OE ------------ 3.10.0-debug #2 [11716.570244] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [11716.570937] ffffffffa02e4ed2 00000000809dbba9 ffff8800b7697cc0 ffffffff816fd3e4 [11716.572370] ffff8800b7697d40 ffffffff816f8c34 ffffffff00000008 ffff8800b7697d50 [11716.573539] ffff8800b7697cf0 00000000809dbba9 00000000809dbba9 ffff88033e48d948 [11716.574459] Call Trace: [11716.574892] [&lt;ffffffff816fd3e4&gt;] dump_stack+0x19/0x1b [11716.575383] [&lt;ffffffff816f8c34&gt;] panic+0xd8/0x1e7 [11716.575862] [&lt;ffffffffa02c5874&gt;] lbug_with_loc+0x64/0xb0 [libcfs] [11716.576514] [&lt;ffffffffa061dde2&gt;] tgt_client_free+0x2a2/0x360 [ptlrpc] [11716.577065] [&lt;ffffffffa1412b12&gt;] ofd_destroy_export+0x62/0x180 [ofd] [11716.577577] [&lt;ffffffffa03c9239&gt;] obd_zombie_impexp_cull+0x549/0x920 [obdclass] [11716.578500] [&lt;ffffffffa03c967d&gt;] obd_zombie_impexp_thread+0x6d/0x1c0 [obdclass] [11716.579429] [&lt;ffffffff810b7cc0&gt;] ? wake_up_state+0x20/0x20 [11716.579917] [&lt;ffffffffa03c9610&gt;] ? obd_zombie_impexp_cull+0x920/0x920 [obdclass] [11716.580827] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [11716.581306] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 [11716.581801] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [11716.582316] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>crashdump is in 192.168.123.146-2017-07-29-23:21:* on onyx-68</p> <p>this still seems to be regularly triggering in my testing</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Lustre: DEBUG MARKER: == recovery-small test 60: Add Changelog entries during MDS failover ================================= 04:12:39 (1573945959) Lustre: lustre-MDD0000: changelog on Lustre: lustre-MDT0001: haven<span class="code-quote">'t heard from client 128ea591-f299-4 (at 192.168.122.22@tcp) in 48 seconds. I think it'</span>s dead, and I am evicting it. exp 000000007725ad20, cur 1573945996 expire 1573945966 last 1573945948 Lustre: lustre-OST0000: haven<span class="code-quote">'t heard from client 128ea591-f299-4 (at 192.168.122.22@tcp) in 48 seconds. I think it'</span>s dead, and I am evicting it. exp 00000000a202a5e3, cur 1573945996 expire 1573945966 last 1573945948 LustreError: 19:0:(tgt_lastrcvd.c:451:tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed: LustreError: 19:0:(tgt_lastrcvd.c:451:tgt_client_free()) LBUG ... Call Trace: ? __schedule+0x2ad/0xb00 schedule+0x34/0x80 lbug_with_loc+0x79/0x80 [libcfs] ? tgt_client_free+0x2b0/0x330 [ptlrpc] ? mdt_destroy_export+0x87/0x2a0 [mdt] ? class_export_destroy+0xe9/0x460 [obdclass] ? process_one_work+0x249/0x5d0 ? worker_thread+0x48/0x3d0 ? kthread+0x100/0x140 umount D 0 24858 24857 0x00000000 Call Trace: ? __schedule+0x2ad/0xb00 schedule+0x34/0x80 schedule_timeout+0x323/0x500 ? wait_for_common+0x3b/0x160 wait_for_common+0xc9/0x160 ? wake_up_q+0x60/0x60 flush_workqueue+0x143/0x4a0 ? obd_exports_barrier+0x43/0x1a0 [obdclass] ? obd_exports_barrier+0x76/0x1a0 [obdclass] mgs_device_fini+0xdb/0x5c0 [mgs] class_cleanup+0x689/0xb50 [obdclass] class_process_config+0x153e/0x30f0 [obdclass] ? cache_alloc_debugcheck_after+0x138/0x150 class_manual_cleanup+0x197/0x670 [obdclass] server_put_super+0x1525/0x1d50 [obdclass] ? evict_inodes+0x138/0x180 generic_shutdown_super+0x5f/0xf0 </pre> </div></div> <p>looks like MDT umount didn't wait for all exports to be gone?</p> <p>there is no any serialization between export destroy and obd destroy:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 00000020:00000080:0.0:1607303759.080403:0:10539:0:(genops.c:984:class_export_put()) <span class="code-keyword">final</span> put 0000000048c8f7e8/7bdf7e52-e46c-4201-82b5-5380be291135 00000020:00000001:1.0:1607303759.082137:0:11815:0:(tgt_main.c:570:tgt_fini()) <span class="code-object">Process</span> entered 00000020:00000001:1.0:1607303759.082148:0:11815:0:(tgt_main.c:610:tgt_fini()) <span class="code-object">Process</span> leaving 00000020:00000080:1.0:1607303759.082811:0:8175:0:(genops.c:943:class_export_destroy()) destroying export 0000000048c8f7e8/7bdf7e52-e46c-4201-82b5-5380be291135 <span class="code-keyword">for</span> lustre-OST0000 00000001:00040000:1.0:1607303759.082843:0:8175:0:(tgt_lastrcvd.c:451:tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed: </pre> </div></div> <p>IMHO, the check for freed OBD is very naive:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* Target may have been freed (see LU-7430) * Slot may be not yet assigned */ <span class="code-keyword">if</span> (exp-&gt;exp_obd-&gt;u.obt.obt_magic != OBT_MAGIC || ted-&gt;ted_lr_idx &lt; 0) <span class="code-keyword">return</span>; </pre> </div></div> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50147" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50147</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9806" title="tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed" class="issue-link" data-issue-key="LU-9806"><del>LU-9806</del></a> obdclass: wait for all exports to go<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8895829088251d37576a01d959689d4d9e9204a7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50147/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50147/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9806" title="tgt_client_free()) ASSERTION( lut &amp;&amp; lut-&gt;lut_client_bitmap ) failed" class="issue-link" data-issue-key="LU-9806"><del>LU-9806</del></a> obdclass: wait for all exports to go<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 08f9ebe93b300c39d2af1fb8e82a22e9c84f401b</p> <p>Landed for 2.16</p> Duplicate LU-11232 Related Development Rank 1|hzzhfr: Rank (Obsolete) 9223372036854775807 Severity