https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7221 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/6824daa8-65d5-11e5-8d98-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6824daa8-65d5-11e5-8d98-5254006e85c2</a>.</p> <p>The sub-test test_3 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>07:42:45:Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification == 00:42:38 (1443426158) 07:42:45:Lustre: DEBUG MARKER: grep -c /mnt/ost1' ' /proc/mounts 07:42:45:Lustre: DEBUG MARKER: umount -d /mnt/ost1 07:42:45:Lustre: Failing over lustre-OST0000 07:42:45:Lustre: lustre-OST0000: Not available for connect from 10.2.4.116@tcp (stopping) 07:42:45:LustreError: 28990:0:(genops.c:815:class_export_put()) ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0 07:42:45:LustreError: 28990:0:(genops.c:815:class_export_put()) LBUG 07:42:45:Pid: 28990, comm: ll_ost00_014 07:42:45: 07:42:45:Call Trace: 07:42:45: [&lt;ffffffffa04a6875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 07:42:45: [&lt;ffffffffa04a6e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 07:42:45: [&lt;ffffffffa05c98b1&gt;] class_export_put+0x271/0x310 [obdclass] 07:42:45: [&lt;ffffffffa05c9aa0&gt;] obd_stale_export_put+0x150/0x290 [obdclass] 07:42:45: [&lt;ffffffffa05c9d01&gt;] class_unlink_export+0x121/0x130 [obdclass] 07:42:45: [&lt;ffffffffa05e4a90&gt;] class_decref+0x350/0x4d0 [obdclass] 07:42:45: [&lt;ffffffffa04b2b61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 07:42:45: [&lt;ffffffffa0531efc&gt;] ? libcfs_nid2str_r+0x11c/0x140 [lnet] 07:42:45: [&lt;ffffffffa07ea6ad&gt;] target_handle_connect+0x23d/0x2bb0 [ptlrpc] 07:42:45: [&lt;ffffffffa04b2523&gt;] ? libcfs_debug_vmsg2+0x5e3/0xbe0 [libcfs] 07:42:46: [&lt;ffffffffa08907d2&gt;] tgt_request_handle+0x5a2/0x12e0 [ptlrpc] 07:42:46: [&lt;ffffffffa0838731&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] 07:42:46: [&lt;ffffffffa08378f0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 07:42:46: [&lt;ffffffff810a101e&gt;] kthread+0x9e/0xc0 07:42:46: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 07:42:46: [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 07:42:46: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 07:42:46: 07:42:46:Kernel panic - not syncing: LBUG 07:42:46:Pid: 28990, comm: ll_ost00_014 Not tainted 2.6.32-573.3.1.el6_lustre.g00880a0.x86_64 #1 07:42:46:Call Trace: 07:42:46: [&lt;ffffffff815384e4&gt;] ? panic+0xa7/0x16f 07:42:47: [&lt;ffffffffa04a6ecb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 07:42:47: [&lt;ffffffffa05c98b1&gt;] ? class_export_put+0x271/0x310 [obdclass] 07:42:47: [&lt;ffffffffa05c9aa0&gt;] ? obd_stale_export_put+0x150/0x290 [obdclass] 07:42:47: [&lt;ffffffffa05c9d01&gt;] ? class_unlink_export+0x121/0x130 [obdclass] 07:42:47: [&lt;ffffffffa05e4a90&gt;] ? class_decref+0x350/0x4d0 [obdclass] 07:42:47: [&lt;ffffffffa04b2b61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 07:42:47: [&lt;ffffffffa0531efc&gt;] ? libcfs_nid2str_r+0x11c/0x140 [lnet] 07:42:47: [&lt;ffffffffa07ea6ad&gt;] ? target_handle_connect+0x23d/0x2bb0 [ptlrpc] 07:42:47: [&lt;ffffffffa04b2523&gt;] ? libcfs_debug_vmsg2+0x5e3/0xbe0 [libcfs] 07:42:47: [&lt;ffffffffa08907d2&gt;] ? tgt_request_handle+0x5a2/0x12e0 [ptlrpc] 07:42:47: [&lt;ffffffffa0838731&gt;] ? ptlrpc_main+0xe41/0x1910 [ptlrpc] 07:42:47: [&lt;ffffffffa08378f0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 07:42:47: [&lt;ffffffff810a101e&gt;] ? kthread+0x9e/0xc0 07:42:47: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 07:42:47: [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 07:42:47: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 test failed to respond and timed out </pre> </div></div> LU-7221

replay-ost-single test_3: ASSERTION( __v > 0 && __v < ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0

Bug Critical Resolved Fixed Bruno Faccini Maloo Mon, 28 Sep 2015 16:29:04 +0000 Fri, 15 Mar 2019 17:19:03 +0000 Tue, 24 Nov 2015 14:40:37 +0000 Lustre 2.8.0 Lustre 2.8.0 0 11 <p>Same LBUG seen hanging replay-single test_7 with logs at <a href="https://testing.hpdd.intel.com/test_sets/9d7825ca-6bf7-11e5-87fb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9d7825ca-6bf7-11e5-87fb-5254006e85c2</a></p> <p>From the MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>22:48:00:Lustre: lustre-MDT0000: Not available for connect from 10.1.4.104@tcp (stopping) 22:48:00:Lustre: Skipped 9 previous similar messages 22:48:00:LustreError: 21346:0:(genops.c:815:class_export_put()) ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0 22:48:00:LustreError: 21346:0:(genops.c:815:class_export_put()) LBUG 22:48:00:Pid: 21346, comm: mdt00_001 22:48:00: 22:48:00:Call Trace: 22:48:00: [&lt;ffffffffa049b875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 22:48:00: [&lt;ffffffffa049be77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 22:48:00: [&lt;ffffffffa05bea21&gt;] class_export_put+0x271/0x310 [obdclass] 22:48:00: [&lt;ffffffffa05bec10&gt;] obd_stale_export_put+0x150/0x290 [obdclass] 22:48:00: [&lt;ffffffffa05bee71&gt;] class_unlink_export+0x121/0x130 [obdclass] 22:48:00: [&lt;ffffffffa05d9c00&gt;] class_decref+0x350/0x4d0 [obdclass] 22:48:00: [&lt;ffffffffa04a7b61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 22:48:00: [&lt;ffffffffa0526efc&gt;] ? libcfs_nid2str_r+0x11c/0x140 [lnet] 22:48:00: [&lt;ffffffffa07df6ad&gt;] target_handle_connect+0x23d/0x2bb0 [ptlrpc] 22:48:00: [&lt;ffffffffa04a7523&gt;] ? libcfs_debug_vmsg2+0x5e3/0xbe0 [libcfs] 22:48:00: [&lt;ffffffffa05f1a45&gt;] ? keys_fill+0x25/0x1b0 [obdclass] 22:48:00: [&lt;ffffffffa08857d2&gt;] tgt_request_handle+0x5a2/0x12e0 [ptlrpc] 22:48:00: [&lt;ffffffffa082d731&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] 22:48:00: [&lt;ffffffffa082c8f0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 22:48:00: [&lt;ffffffff810a101e&gt;] kthread+0x9e/0xc0 22:48:00: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 22:48:00: [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 22:48:00: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 22:48:00: 22:48:00: 22:48:00:Kernel panic - not syncing: LBUG 22:48:00:Pid: 21346, comm: mdt00_001 Not tainted 2.6.32-573.3.1.el6_lustre.g00880a0.x86_64 #1 22:48:00:Call Trace: 22:48:00: [&lt;ffffffff815384e4&gt;] ? panic+0xa7/0x16f 22:48:00: [&lt;ffffffffa049becb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 22:48:00: [&lt;ffffffffa05bea21&gt;] ? class_export_put+0x271/0x310 [obdclass] 22:48:00: [&lt;ffffffffa05bec10&gt;] ? obd_stale_export_put+0x150/0x290 [obdclass] 22:48:00: [&lt;ffffffffa05bee71&gt;] ? class_unlink_export+0x121/0x130 [obdclass] 22:48:00: [&lt;ffffffffa05d9c00&gt;] ? class_decref+0x350/0x4d0 [obdclass] 22:48:00: [&lt;ffffffffa04a7b61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 22:48:00: [&lt;ffffffffa0526efc&gt;] ? libcfs_nid2str_r+0x11c/0x140 [lnet] 22:48:00: [&lt;ffffffffa07df6ad&gt;] ? target_handle_connect+0x23d/0x2bb0 [ptlrpc] 22:48:00: [&lt;ffffffffa04a7523&gt;] ? libcfs_debug_vmsg2+0x5e3/0xbe0 [libcfs] 22:48:00: [&lt;ffffffffa05f1a45&gt;] ? keys_fill+0x25/0x1b0 [obdclass] 22:48:00: [&lt;ffffffffa08857d2&gt;] ? tgt_request_handle+0x5a2/0x12e0 [ptlrpc] 22:48:00: [&lt;ffffffffa082d731&gt;] ? ptlrpc_main+0xe41/0x1910 [ptlrpc] 22:48:00: [&lt;ffffffffa082c8f0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 22:48:00: [&lt;ffffffff810a101e&gt;] ? kthread+0x9e/0xc0 22:48:00: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 22:48:00: [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 22:48:00: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 22:48:00:Initializing cgroup subsys cpuset </pre> </div></div> <p>+1 at <a href="https://testing.hpdd.intel.com/test_sets/9d7825ca-6bf7-11e5-87fb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9d7825ca-6bf7-11e5-87fb-5254006e85c2</a></p> <p>This LBUG seen on replay-single test_53g timeout. Logs at <a href="https://testing.hpdd.intel.com/test_sets/13337056-6c72-11e5-9ae6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/13337056-6c72-11e5-9ae6-5254006e85c2</a></p> <p>seen in replay-single, test 93 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/50f53620-74c1-11e5-b8c1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/50f53620-74c1-11e5-b8c1-5254006e85c2</a></p> <p>console log of OST shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>05:39:24:LustreError: 9188:0:(genops.c:815:class_export_put()) ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0 05:39:24:LustreError: 9188:0:(genops.c:815:class_export_put()) LBUG 05:39:24:Pid: 9188, comm: ll_ost00_012 05:39:24: 05:39:24:Call Trace: 05:39:24: [&lt;ffffffffa079f875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 05:39:24: [&lt;ffffffffa079fe77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 05:39:24: [&lt;ffffffffa08c28b1&gt;] class_export_put+0x271/0x310 [obdclass] 05:39:24: [&lt;ffffffffa08c2aa0&gt;] obd_stale_export_put+0x150/0x290 [obdclass] 05:39:24: [&lt;ffffffffa08c2d01&gt;] class_unlink_export+0x121/0x130 [obdclass] 05:39:24: [&lt;ffffffffa08dd778&gt;] class_decref+0x348/0x4c0 [obdclass] 05:39:24: [&lt;ffffffffa07abb61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 05:39:24: [&lt;ffffffffa082aefc&gt;] ? libcfs_nid2str_r+0x11c/0x140 [lnet] 05:39:24: [&lt;ffffffffa0ae27ed&gt;] target_handle_connect+0x23d/0x2ba0 [ptlrpc] 05:39:24: [&lt;ffffffffa07ab523&gt;] ? libcfs_debug_vmsg2+0x5e3/0xbe0 [libcfs] 05:39:24: [&lt;ffffffffa0b88dc2&gt;] tgt_request_handle+0x5a2/0x12e0 [ptlrpc] 05:39:24: [&lt;ffffffffa0b309c1&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] 05:39:24: [&lt;ffffffffa0b2fb80&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 05:39:24: [&lt;ffffffff810a0fce&gt;] kthread+0x9e/0xc0 05:39:24: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 05:39:24: [&lt;ffffffff810a0f30&gt;] ? kthread+0x0/0xc0 05:39:24: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 05:39:24: 05:39:24:Kernel panic - not syncing: LBUG </pre> </div></div> <p>another in replay-single test 62 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/6c2d9470-7537-11e5-95e7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6c2d9470-7537-11e5-95e7-5254006e85c2</a></p> <p>This appears to be happening all over the place in various tests in replay-single.</p> <p>another seen in replay-single, test_35 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d890aae6-78d2-11e5-9aa5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d890aae6-78d2-11e5-9aa5-5254006e85c2</a></p> <p>Seems there are a variety of problems with unmounting the MDT or OST that need to be investigated.</p> <p>Just got a very similar occurrence during replay-single/test_80d at <a href="https://testing.hpdd.intel.com/test_sets/7466dbda-799a-11e5-a447-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7466dbda-799a-11e5-a447-5254006e85c2</a>.</p> <p>A quick look to the crash-dump shows that 3 MDT threads have triggered the same LBUG at the same time, and their common stack part looks like :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>............. #3 [ffff88007cd7bab8] lbug_with_loc at ffffffffa04a6ecb [libcfs] #4 [ffff88007cd7bad8] class_export_put at ffffffffa05c98b1 [obdclass] #5 [ffff88007cd7baf8] obd_stale_export_put at ffffffffa05c9aa0 [obdclass] #6 [ffff88007cd7bb28] class_unlink_export at ffffffffa05c9d01 [obdclass] #7 [ffff88007cd7bb48] class_decref at ffffffffa05e4778 [obdclass] #8 [ffff88007cd7bbb8] target_handle_connect at ffffffffa07e97ed [ptlrpc] #9 [ffff88007cd7bd48] tgt_request_handle at ffffffffa088fb22 [ptlrpc] #10 [ffff88007cd7bda8] ptlrpc_main at ffffffffa08378c1 [ptlrpc] #11 [ffff88007cd7bee8] kthread at ffffffff810a0fce #12 [ffff88007cd7bf48] kernel_thread at ffffffff8100c28a </pre> </div></div> <p>they all 3 are working on the same "lustre-MDT0000" obd_device, its obd_refcount=1, but some of its other counters (obd_num_exports=-3, obd_conn_inprogress=-5) seem to indicate a possible race condition where multiple threads have executed this same path in the code handling connect requests when the MDT target is being unmounted (obd_stopping).</p> <p>Having a look to the concerned code, I think this has been introduced by patch <a href="http://review.whamcloud.com/#/c/11750/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11750/</a> for <a href="https://jira.whamcloud.com/browse/LU-5569" title="recreating a reverse import produce a various fails." class="issue-link" data-issue-key="LU-5569"><del>LU-5569</del></a>, where class_incref()/class_decref() calls are now done even when obd_stopping ...</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7221" title="replay-ost-single test_3: ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0" class="issue-link" data-issue-key="LU-7221"><del>LU-7221</del></a> ldlm: do not take a reference on target if stopping<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3008de85070ab7ce9388f1fad6ec37f4989dcdd</p> <p>Patch <a href="http://review.whamcloud.com/16940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16940</a> is an attempt to revert one of the changes of patch <a href="http://review.whamcloud.com/#/c/11750/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11750/</a> for <a href="https://jira.whamcloud.com/browse/LU-5569" title="recreating a reverse import produce a various fails." class="issue-link" data-issue-key="LU-5569"><del>LU-5569</del></a>, which allows to take a reference on a target even if it is stopping/umounting, upon new connections requests, finally causing a LBUG during useless <span class="error">&#91;obd_self_&#93;</span>export cleanup operations.</p> <p>Another failure on master at <a href="https://testing.hpdd.intel.com/test_sets/5c507be8-7b46-11e5-9ee6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5c507be8-7b46-11e5-9ee6-5254006e85c2</a></p> <p>2015-10-26 04:00:03 - <a href="https://testing.hpdd.intel.com/test_sets/87e30d88-7bba-11e5-9851-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/87e30d88-7bba-11e5-9851-5254006e85c2</a><br/> 2015-11-11 13:07:17 - <a href="https://testing.hpdd.intel.com/test_sets/622a597e-88a9-11e5-b099-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/622a597e-88a9-11e5-b099-5254006e85c2</a></p> <p>another on master;<br/> <a href="https://testing.hpdd.intel.com/test_sets/fa2e57a4-7c59-11e5-9ca1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fa2e57a4-7c59-11e5-9ca1-5254006e85c2</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e4a06a3a-86a4-11e5-bf92-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e4a06a3a-86a4-11e5-bf92-5254006e85c2</a></p> <p>Another one on master<br/> <a href="https://testing.hpdd.intel.com/test_sets/112b8818-896c-11e5-8ba4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/112b8818-896c-11e5-8ba4-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16940/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16940/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7221" title="replay-ost-single test_3: ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0" class="issue-link" data-issue-key="LU-7221"><del>LU-7221</del></a> ldlm: do not take a reference on target if stopping<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6ff417169a2ad84478cce1b0321e70a030ffed83</p> <p>Landed for 2.8</p> Duplicate LU-7411 Related LU-7038 LU-7172 LU-7256 LU-7326 LU-5569 Development Rank 1|hzxow7: Rank (Obsolete) 9223372036854775807 Severity