https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7739 Lustre <p>replay-single test_70b times out. In the MDS 2, MDS 3, MDS 4 console log, we see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13:15:37:LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 13:15:37:LustreError: 25429:0:(mgc_request.c:995:mgc_blocking_ast()) ASSERTION( atomic_read(&amp;cld-&gt;cld_refcount) &gt; 0 ) failed: 13:15:37:LustreError: 25429:0:(mgc_request.c:995:mgc_blocking_ast()) LBUG 13:15:37:Pid: 25429, comm: ldlm_bl_01 13:15:37: 13:15:37:Call Trace: 13:15:37: [&lt;ffffffffa0467875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 13:15:37: [&lt;ffffffffa0467e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 13:15:37: [&lt;ffffffffa0cff9d9&gt;] mgc_blocking_ast+0x6e9/0x810 [mgc] 13:15:37: [&lt;ffffffffa0758b57&gt;] ldlm_cancel_callback+0x87/0x280 [ptlrpc] 13:15:37: [&lt;ffffffffa07779ba&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] 13:15:37: [&lt;ffffffffa077c55c&gt;] ldlm_cli_cancel+0x9c/0x3e0 [ptlrpc] 13:15:37: [&lt;ffffffffa0cff3db&gt;] mgc_blocking_ast+0xeb/0x810 [mgc] 13:15:37: [&lt;ffffffffa0cff2f0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] 13:15:37: [&lt;ffffffffa0780c90&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] 13:15:37: [&lt;ffffffffa0781ba1&gt;] ldlm_bl_thread_main+0x481/0x710 [ptlrpc] 13:15:37: [&lt;ffffffff810672b0&gt;] ? default_wake_function+0x0/0x20 13:15:37: [&lt;ffffffffa0781720&gt;] ? ldlm_bl_thread_main+0x0/0x710 [ptlrpc] 13:15:37: [&lt;ffffffff810a0fce&gt;] kthread+0x9e/0xc0 13:15:37: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 13:15:37: [&lt;ffffffff810a0f30&gt;] ? kthread+0x0/0xc0 13:15:37: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 13:15:37: 13:15:37:Kernel panic - not syncing: LBUG </pre> </div></div> <p>In the past month, I can only find two occurrences of this error for test_70b. Logs at<br/> 2016-01-28 15:21:30 - <a href="https://testing.hpdd.intel.com/test_sets/c296d92c-c620-11e5-b4e1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c296d92c-c620-11e5-b4e1-5254006e85c2</a><br/> 2016-02-03 19:34:24 - <a href="https://testing.hpdd.intel.com/test_sets/e4674cb8-caf7-11e5-be8d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e4674cb8-caf7-11e5-be8d-5254006e85c2</a></p> autotest review-dne-part-2 LU-7739

replay-single test 70b hangs with LBUG '(mgc_request.c:995:mgc_blocking_ast()) ASSERTION( atomic_read(&cld->cld_refcount) > 0 )'

Bug Critical Resolved Cannot Reproduce WC Triage James Nunez Thu, 4 Feb 2016 16:29:16 +0000 Tue, 14 Dec 2021 22:19:07 +0000 Tue, 14 Dec 2021 22:19:07 +0000 Lustre 2.8.0 Lustre 2.9.0 0 6 <p>A similar error on sanity test_133g on unmount of MDS4. The LBUG/ASSERT is the same as above, but the stack trace differs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13:44:11:Lustre: DEBUG MARKER: umount -d -f /mnt/mds4 13:44:11:LustreError: 5639:0:(osp_object.c:588:osp_attr_get()) lustre-MDT0000-osp-MDT0003:osp_attr_get update error [0x200000403:0x1:0x0]: rc = -5 13:44:11:LustreError: 5639:0:(osp_object.c:588:osp_attr_get()) Skipped 2 previous similar messages 13:44:11:LustreError: 5639:0:(mgc_request.c:995:mgc_blocking_ast()) ASSERTION( atomic_read(&amp;cld-&gt;cld_refcount) &gt; 0 ) failed: 13:44:11:LustreError: 5639:0:(mgc_request.c:995:mgc_blocking_ast()) LBUG 13:44:11:Pid: 5639, comm: umount 13:44:11: 13:44:11:Call Trace: 13:44:11: [&lt;ffffffffa0467875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 13:44:11: [&lt;ffffffffa0467e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 13:44:11: [&lt;ffffffffa0cc48b9&gt;] mgc_blocking_ast+0x6e9/0x810 [mgc] 13:44:11: [&lt;ffffffffa074ba87&gt;] ldlm_cancel_callback+0x87/0x280 [ptlrpc] 13:44:11: [&lt;ffffffffa0472cf1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 13:44:11: [&lt;ffffffffa07698da&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] 13:44:11: [&lt;ffffffffa076ccae&gt;] ldlm_cli_cancel_list_local+0xee/0x290 [ptlrpc] 13:44:11: [&lt;ffffffffa076d011&gt;] ldlm_cancel_resource_local+0x1c1/0x290 [ptlrpc] 13:44:11: [&lt;ffffffffa076d177&gt;] ldlm_cli_cancel_unused_resource+0x97/0x280 [ptlrpc] 13:44:11: [&lt;ffffffffa076d545&gt;] ldlm_cli_hash_cancel_unused+0x35/0x40 [ptlrpc] 13:44:11: [&lt;ffffffffa04787cb&gt;] cfs_hash_for_each_relax+0x1eb/0x350 [libcfs] 13:44:11: [&lt;ffffffffa076d510&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0x40 [ptlrpc] 13:44:11: [&lt;ffffffffa076d510&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0x40 [ptlrpc] 13:44:11: [&lt;ffffffffa076d510&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0x40 [ptlrpc] 13:44:11: [&lt;ffffffffa047a6ac&gt;] cfs_hash_for_each_nolock+0x8c/0x1d0 [libcfs] 13:44:11: [&lt;ffffffffa076d4c6&gt;] ldlm_cli_cancel_unused+0x166/0x1b0 [ptlrpc] 13:44:11: [&lt;ffffffffa075ae98&gt;] client_disconnect_export+0x228/0x460 [ptlrpc] 13:44:11: [&lt;ffffffffa057af3a&gt;] lustre_common_put_super+0x28a/0xb00 [obdclass] 13:44:11: [&lt;ffffffffa05a0246&gt;] server_put_super+0x116/0xcd0 [obdclass] 13:44:11: [&lt;ffffffff811946eb&gt;] generic_shutdown_super+0x5b/0xe0 13:44:11: [&lt;ffffffff811947d6&gt;] kill_anon_super+0x16/0x60 13:44:11: [&lt;ffffffffa0572616&gt;] lustre_kill_super+0x36/0x60 [obdclass] 13:44:11: [&lt;ffffffff81194f77&gt;] deactivate_super+0x57/0x80 13:44:11: [&lt;ffffffff811b4f5f&gt;] mntput_no_expire+0xbf/0x110 13:44:11: [&lt;ffffffff811b5aab&gt;] sys_umount+0x7b/0x3a0 13:44:11: [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b 13:44:11: 13:44:11:Kernel panic - not syncing: LBUG 13:44:11:Pid: 5639, comm: umount Not tainted 2.6.32-573.18.1.el6_lustre.g55ae312.x86_64 #1 </pre> </div></div> <p>Log are at<br/> 2016-02-22 19:06:02 - <a href="https://testing.hpdd.intel.com/test_sets/102b46ac-d9e8-11e5-9e9f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/102b46ac-d9e8-11e5-9e9f-5254006e85c2</a></p> <p>Lustre Branch: master<br/> Test Group: review-dne-part-2</p> <p>insanity test 11 hit the same failure on MDS 2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: mkdir -p /mnt/mds2; mount -t lustre /dev/lvm-Role_MDS/P2 /mnt/mds2 LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: LustreError: 31250:0:(mgc_request.c:995:mgc_blocking_ast()) ASSERTION( atomic_read(&amp;cld-&gt;cld_refcount) &gt; 0 ) failed: LustreError: 31250:0:(mgc_request.c:995:mgc_blocking_ast()) LBUG Pid: 31250, comm: ldlm_bl_01 Call Trace: [&lt;ffffffffa0464875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0464e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa044b8b9&gt;] mgc_blocking_ast+0x6e9/0x810 [mgc] [&lt;ffffffffa075ea87&gt;] ldlm_cancel_callback+0x87/0x280 [ptlrpc] [&lt;ffffffffa077c4da&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] [&lt;ffffffffa078107c&gt;] ldlm_cli_cancel+0x9c/0x3e0 [ptlrpc] [&lt;ffffffffa044b2bb&gt;] mgc_blocking_ast+0xeb/0x810 [mgc] [&lt;ffffffffa044b1d0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0785400&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] [&lt;ffffffffa0786291&gt;] ldlm_bl_thread_main+0x461/0x650 [ptlrpc] [&lt;ffffffff81067670&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0785e30&gt;] ? ldlm_bl_thread_main+0x0/0x650 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 31250, comm: ldlm_bl_01 Not tainted 2.6.32-573.22.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff815394d1&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa0464ecb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa044b8b9&gt;] ? mgc_blocking_ast+0x6e9/0x810 [mgc] [&lt;ffffffffa075ea87&gt;] ? ldlm_cancel_callback+0x87/0x280 [ptlrpc] [&lt;ffffffffa077c4da&gt;] ? ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] [&lt;ffffffffa078107c&gt;] ? ldlm_cli_cancel+0x9c/0x3e0 [ptlrpc] [&lt;ffffffffa044b2bb&gt;] ? mgc_blocking_ast+0xeb/0x810 [mgc] [&lt;ffffffffa044b1d0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0785400&gt;] ? ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] [&lt;ffffffffa0786291&gt;] ? ldlm_bl_thread_main+0x461/0x650 [ptlrpc] [&lt;ffffffff81067670&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0785e30&gt;] ? ldlm_bl_thread_main+0x0/0x650 [ptlrpc] [&lt;ffffffff810a138e&gt;] ? kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p><a href="https://testing.hpdd.intel.com/test_sets/595a0f92-030b-11e6-b5f1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/595a0f92-030b-11e6-b5f1-5254006e85c2</a></p> <p>We're sometimes seeing a crash almost identical to the one from James Nunez, at umount.<br/> Using tag 2.8.56 + the fixes for <a href="https://jira.whamcloud.com/browse/LU-6808" title="Interop 2.5.3&lt;-&gt;master sanity test_224c: Bulk IO write error" class="issue-link" data-issue-key="LU-6808"><del>LU-6808</del></a> on the servers, 2.5.41_DDN on the servers</p> <p>This is on a production system.</p> Related LU-7298 LU-6844 Development Rank 1|hzy0b3: Rank (Obsolete) 9223372036854775807 Severity