https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7157 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/ccc00f28-5b17-11e5-af09-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ccc00f28-5b17-11e5-af09-5254006e85c2</a>.</p> <p>The sub-test test_27z failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>I think an OST crashed &amp; rebooted during test 27z, but I'm not sure. no console logs were captured. console logs might have given better clues.</p> <p>Info required for matching: sanity 27z</p> LU-7157

sanity test_27z: cfs_hash_bd_del_locked()) ASSERTION( bd->bd_bucket->hsb_count > 0 ) failed

Bug Minor Open Unresolved WC Triage Maloo Mon, 14 Sep 2015 19:48:27 +0000 Thu, 23 Nov 2017 17:56:18 +0000 Lustre 2.8.0 0 8 <p>I think the missing console logs have been misplaced onto lustre-init, as has been seen before on el7 test runs. This isn't el7, it's sles11sp4. However the same thing may be happening here.</p> <p>If I look at the OST console log recorded in lustre-init I do in fact see a panic:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16:57:44:Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.1 01-63_lustre.g031dbf9-default (console).16:57:44: 16:57:44: 16:57:44:shadow-7vm11 login: [ 1817.654651] LustreError: 17681:0:(hash.c:554:cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed: 16:57:44:[ 1817.658147] LustreError: 17681:0:(hash.c:554:cfs_hash_bd_del_locked()) LBUG 16:57:44:[ 1817.661508] Kernel panic - not syncing: LBUG 16:57:44:[ 1817.662478] Pid: 17681, comm: umount Tainted: G EN 3.0.101-63_lustre.g031dbf9-default #1 16:57:44:[ 1817.664474] Call Trace: 16:57:44:[ 1817.665214] [&lt;ffffffff81004b95&gt;] dump_trace+0x75/0x300 16:57:44:[ 1817.666278] [&lt;ffffffff81466093&gt;] dump_stack+0x69/0x6f 16:57:44:[ 1817.667349] [&lt;ffffffff8146612c&gt;] panic+0x93/0x201 16:57:44:[ 1817.669684] [&lt;ffffffffa0726db3&gt;] lbug_with_loc+0xa3/0xb0 [libcfs] 16:57:44:[ 1817.672157] [&lt;ffffffffa0737ccd&gt;] cfs_hash_bd_del_locked+0xdd/0x120 [libcfs] 16:57:44:[ 1817.675226] [&lt;ffffffffa0a6516e&gt;] __ldlm_resource_putref_final+0x3e/0xc0 [ptlrpc] 16:57:44:[ 1817.679206] [&lt;ffffffffa0a652d2&gt;] ldlm_resource_putref_locked+0xe2/0x3f0 [ptlrpc] 16:57:44:[ 1817.681626] [&lt;ffffffffa073852a&gt;] cfs_hash_for_each_relax+0x1da/0x330 [libcfs] 16:57:44:[ 1817.683368] [&lt;ffffffffa073a6ba&gt;] cfs_hash_for_each_nolock+0x7a/0x1e0 [libcfs] 16:57:44:[ 1817.685157] [&lt;ffffffffa0a63be9&gt;] ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] 16:57:44:[ 1817.686515] [&lt;ffffffffa0a660f2&gt;] __ldlm_namespace_free+0x52/0x580 [ptlrpc] 16:57:44:[ 1817.687817] [&lt;ffffffffa0a66682&gt;] ldlm_namespace_free_prior+0x62/0x230 [ptlrpc] 16:57:44:[ 1817.689594] [&lt;ffffffffa0fff0a8&gt;] ofd_fini+0x58/0x190 [ofd] 16:57:44:[ 1817.690725] [&lt;ffffffffa0fff211&gt;] ofd_device_fini+0x31/0xf0 [ofd] 16:57:44:[ 1817.692021] [&lt;ffffffffa086872d&gt;] class_cleanup+0x9bd/0xd40 [obdclass] 16:57:44:[ 1817.694026] [&lt;ffffffffa0869c91&gt;] class_process_config+0x11e1/0x1910 [obdclass] 16:57:44:[ 1817.698374] [&lt;ffffffffa086a8bf&gt;] class_manual_cleanup+0x4ff/0x8c0 [obdclass] 16:57:44:[ 1817.701394] [&lt;ffffffffa08a6477&gt;] server_put_super+0x607/0xb00 [obdclass] 16:57:44:[ 1817.702686] [&lt;ffffffff811603fb&gt;] generic_shutdown_super+0x6b/0x100 16:57:44:[ 1817.703907] [&lt;ffffffff81160519&gt;] kill_anon_super+0x9/0x20 16:57:44:[ 1817.705052] [&lt;ffffffff81160b83&gt;] deactivate_locked_super+0x33/0x90 16:57:44:[ 1817.706654] [&lt;ffffffff8117cc0c&gt;] sys_umount+0x6c/0xd0 16:57:44:[ 1817.707718] [&lt;ffffffff814710f2&gt;] system_call_fastpath+0x16/0x1b 16:57:44:[ 1817.708906] [&lt;00007fd1fa0b16f7&gt;] 0x7fd1fa0b16f6 16:57:44:[ 0.000000] Initializing cgroup subsys cpuset 16:57:44:[ 0.000000] Initializing cgroup subsys cpu </pre> </div></div> <p>hoping this will give somebody a clue.</p> <p><a href="https://testing.hpdd.intel.com/test_logs/959bcc54-6bba-11e5-8e3b-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/959bcc54-6bba-11e5-8e3b-5254006e85c2/show_text</a> - not exactly the same, but the same test and at umount.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13:06:41:Lustre: DEBUG MARKER: umount -d /mnt/ost1 13:06:41:Lustre: Failing over lustre-OST0000 13:06:41:Lustre: lustre-OST0000: Not available for connect from 10.1.4.19@tcp (stopping) 13:06:41:LustreError: 6276:0:(genops.c:815:class_export_put()) ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0 13:06:41:LustreError: 6276:0:(genops.c:815:class_export_put()) LBUG </pre> </div></div> <p>Similar failure on OST unmount on master at<br/> 2015-11-05 00:43:03 - <a href="https://testing.hpdd.intel.com/test_sets/008f7a1e-839f-11e5-b1ba-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/008f7a1e-839f-11e5-b1ba-5254006e85c2</a><br/> 2015-11-11 19:37:53 - <a href="https://testing.hpdd.intel.com/test_sets/fa66faee-88ef-11e5-8ba4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fa66faee-88ef-11e5-8ba4-5254006e85c2</a></p> <p>Recent failures of this test:<br/> <a href="https://testing.hpdd.intel.com/test_sets/260b405e-cf8d-11e5-9923-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/260b405e-cf8d-11e5-9923-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:37:21:Lustre: Failing over lustre-OST0000 09:37:21:general protection fault: 0000 [#1] SMP 09:37:21:Pid: 10695, comm: umount Not tainted 2.6.32-573.12.1.el6_lustre.gd68d18b.x86_64 #1 Red Hat KVM 09:37:21:RIP: 0010:[&lt;ffffffffa076151b&gt;] [&lt;ffffffffa076151b&gt;] ldlm_resource_putref_locked+0x1b/0x3f0 [ptlrpc] 09:37:21:Process umount (pid: 10695, threadinfo ffff88005b4d0000, task ffff88005cac8040) 09:37:21:Call Trace: 09:37:21: [&lt;ffffffffa0761902&gt;] ldlm_res_hop_put_locked+0x12/0x20 [ptlrpc] 09:37:21: [&lt;ffffffffa0478779&gt;] cfs_hash_for_each_relax+0x199/0x350 [libcfs] 09:37:21: [&lt;ffffffffa047a6ac&gt;] cfs_hash_for_each_nolock+0x8c/0x1d0 [libcfs] 09:37:21: [&lt;ffffffffa075ff30&gt;] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] 09:37:21: [&lt;ffffffffa0762444&gt;] __ldlm_namespace_free+0x54/0x560 [ptlrpc] 09:37:21: [&lt;ffffffffa07629bf&gt;] ldlm_namespace_free_prior+0x6f/0x220 [ptlrpc] 09:37:21: [&lt;ffffffffa0de85bb&gt;] ofd_device_fini+0x7b/0x260 [ofd] 09:37:21: [&lt;ffffffffa056f282&gt;] class_cleanup+0x572/0xd20 [obdclass] 09:37:21: [&lt;ffffffffa0571906&gt;] class_process_config+0x1ed6/0x2830 [obdclass] 09:37:21: [&lt;ffffffffa057271f&gt;] class_manual_cleanup+0x4bf/0x8e0 [obdclass] 09:37:21: [&lt;ffffffffa05aae3c&gt;] server_put_super+0xa0c/0xed0 [obdclass] 09:37:21: [&lt;ffffffff811944bb&gt;] generic_shutdown_super+0x5b/0xe0 09:37:21: [&lt;ffffffff811945a6&gt;] kill_anon_super+0x16/0x60 09:37:21: [&lt;ffffffffa05755d6&gt;] lustre_kill_super+0x36/0x60 [obdclass] 09:37:21: [&lt;ffffffff81194d47&gt;] deactivate_super+0x57/0x80 09:37:21: [&lt;ffffffff811b4d3f&gt;] mntput_no_expire+0xbf/0x110 09:37:21: [&lt;ffffffff811b588b&gt;] sys_umount+0x7b/0x3a0 </pre> </div></div> <p>and <a href="https://testing.hpdd.intel.com/test_sets/008f7a1e-839f-11e5-b1ba-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/008f7a1e-839f-11e5-b1ba-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:11:57:Lustre: DEBUG MARKER: umount -d /mnt/ost4 01:11:57:LustreError: 15917:0:(hash.c:554:cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed: 01:11:57:LustreError: 15917:0:(hash.c:554:cfs_hash_bd_del_locked()) LBUG 01:11:57:Pid: 15917, comm: umount 01:11:57:Call Trace: 01:11:57: [&lt;ffffffffa046c875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 01:11:57: [&lt;ffffffffa046ce77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 01:11:57: [&lt;ffffffffa047de60&gt;] cfs_hash_bd_del_locked+0xc0/0x100 [libcfs] 01:11:57: [&lt;ffffffffa07663d8&gt;] __ldlm_resource_putref_final+0x48/0xc0 [ptlrpc] 01:11:57: [&lt;ffffffffa076652d&gt;] ldlm_resource_putref_locked+0xdd/0x3f0 [ptlrpc] 01:11:57: [&lt;ffffffffa0766852&gt;] ldlm_res_hop_put_locked+0x12/0x20 [ptlrpc] 01:11:57: [&lt;ffffffffa05782cf&gt;] ? class_manual_cleanup+0x4bf/0x8e0 [obdclass] 01:11:57: [&lt;ffffffffa05559f6&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 01:11:57: [&lt;ffffffffa05b019c&gt;] ? server_put_super+0xa0c/0xed0 [obdclass] 01:11:57: [&lt;ffffffff811b0116&gt;] ? invalidate_inodes+0xf6/0x190 01:11:57: [&lt;ffffffff8119437b&gt;] ? generic_shutdown_super+0x5b/0xe0 01:11:57: [&lt;ffffffff81194466&gt;] ? kill_anon_super+0x16/0x60 01:11:57: [&lt;ffffffffa057b186&gt;] ? lustre_kill_super+0x36/0x60 [obdclass] 01:11:57: [&lt;ffffffff81194c07&gt;] ? deactivate_super+0x57/0x80 01:11:57: [&lt;ffffffff811b4a7f&gt;] ? mntput_no_expire+0xbf/0x110 01:11:57: [&lt;ffffffff811b55cb&gt;] ? sys_umount+0x7b/0x3a0 </pre> </div></div> <p>Another failure: <a href="https://testing.hpdd.intel.com/test_sets/0cc7cb54-4872-11e6-8968-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0cc7cb54-4872-11e6-8968-5254006e85c2</a></p> <p>Andreas, this issue is an oops, your link is for a softlockup. See <a href="https://jira.whamcloud.com/browse/LU-8392" title="sanity test_27z: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:6145]" class="issue-link" data-issue-key="LU-8392"><del>LU-8392</del></a>.</p> <p>Another failure: <a href="https://testing.hpdd.intel.com/test_sets/fb94d724-8420-11e6-a35f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fb94d724-8420-11e6-a35f-5254006e85c2</a></p> <p>This last one seems to be a different failure that's worth filing a separate ticket for</p> Development Rank 1|hzxnjj: Rank (Obsolete) 9223372036854775807 Severity