https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15757 Lustre <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Trace: PID: 622621 TASK: ffff89d05b2117c0 CPU: 1 COMMAND: <span class="code-quote">"ll_imp_inval"</span> #0 [ffff89d04d307b70] panic at ffffffff860af786 /tmp/kernel/kernel/panic.c: 299 #1 [ffff89d04d307c00] ll_md_blocking_ast at ffffffffc1919052 [lustre] /home/lustre/master-mine/lustre/llite/namei.c: 388 #2 [ffff89d04d307c60] ldlm_cancel_callback at ffffffffc0fc074c [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_lock.c: 2445 #3 [ffff89d04d307cb0] ldlm_cli_cancel_local at ffffffffc0fd8ce6 [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_request.c: 1244 #4 [ffff89d04d307cd0] ldlm_cli_cancel at ffffffffc0fde6a8 [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_request.c: 1569 #5 [ffff89d04d307d30] cleanup_resource_queue at ffffffffc0fc278c [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c: 1091 #6 [ffff89d04d307d80] ldlm_resource_cleanup at ffffffffc0fc2914 [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c: 1106 #7 [ffff89d04d307d98] ldlm_resource_clean_hash at ffffffffc0fc294c [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c: 1119 #8 [ffff89d04d307da8] cfs_hash_for_each_relax at ffffffffc0b39cc5 [libcfs] /home/lustre/master-mine/libcfs/libcfs/hash.c: 1644 #9 [ffff89d04d307e20] cfs_hash_for_each_nolock at ffffffffc0b3d40f [libcfs] /home/lustre/master-mine/libcfs/include/libcfs/libcfs_hash.h: 402 #10 [ffff89d04d307e48] ldlm_namespace_cleanup at ffffffffc0fc2ce6 [ptlrpc] /home/lustre/master-mine/lustre/ptlrpc/../../lustre/ldlm/ldlm_resource.c: 1163 #11 [ffff89d04d307e60] mdc_import_event at ffffffffc13456f6 [mdc] /home/lustre/master-mine/lustre/mdc/mdc_request.c: 2712 #12 [ffff89d04d307e90] ptlrpc_invalidate_import at ffffffffc102460d [ptlrpc] /home/lustre/master-mine/libcfs/include/libcfs/libcfs_debug.h: 154 </pre> </div></div> <p>after adding an additional assertiong:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ((bits &amp; (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_PERM))) { LASSERT(inode); LASSERT(inode-&gt;i_sb); LASSERT(inode-&gt;i_sb-&gt;s_root); } </pre> </div></div> <p>it's:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> LustreError: 622621:0:(namei.c:389:ll_lock_cancel_bits()) ASSERTION( inode-&gt;i_sb-&gt;s_root ) failed BUG: unable to handle kernel NULL pointer dereference at 0000000000000030 </pre> </div></div> LU-15757

sanityn test_109: Oops in ll_md_blocking_ast() at umount

Bug Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Tue, 19 Apr 2022 11:26:12 +0000 Tue, 8 Nov 2022 12:57:43 +0000 Thu, 12 May 2022 17:07:21 +0000 Upstream Lustre 2.15.0 Lustre 2.15.0 Lustre 2.15.2 0 5 <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47086" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47086</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> llite: check s_root ll_md_blocking_ast()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dccac47b8b305b99fafaccad66994ac492b7e221</p> <p>not sure how llite's umount can be blocked until all import's invalidations are done, so the patch above just checks s_root to ensure it's not NULL.</p> <p>Do you think this might be related to <a href="https://jira.whamcloud.com/browse/LU-15835" title="sanityn test_102: [20108.951589] WARNING: CPU: 1 PID: 722753 at fs/exportfs/expfs.c:96 reconnect_path+0x22b/0x2c0" class="issue-link" data-issue-key="LU-15835">LU-15835</a>?</p> <p>I looked at several recent sanityn sessions this morning. It appears that when we hit the warning in test_102 then the client crashes in unmount in test_109. And conversely if there was no warning then there is no crash.</p> <p>See <br/> <a href="https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;server_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;horizon=518400&amp;test_set_script_id=570ba67a-4a46-11e0-a7f6-52540025f9af&amp;sub_test_script_id=99ec82f0-f44d-4789-952d-ce0ac311b80e&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;server_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;horizon=518400&amp;test_set_script_id=570ba67a-4a46-11e0-a7f6-52540025f9af&amp;sub_test_script_id=99ec82f0-f44d-4789-952d-ce0ac311b80e&amp;source=sub_tests#redirect</a></p> <p>+2 timeouts on master in sanityn test_109 in the past few days:<br/> <a href="https://testing.whamcloud.com/test_sets/c8c1e14a-226f-4b99-8b4a-0f20ab910488" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c8c1e14a-226f-4b99-8b4a-0f20ab910488</a><br/> <a href="https://testing.whamcloud.com/test_sets/34da2ea2-433f-441b-85a3-b337237c5bfe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/34da2ea2-433f-441b-85a3-b337237c5bfe</a></p> <p>Not sure if this is a regression because of some recent patch landing, or just coincidence.</p> <p>I hadn't seen John's comment that there are also crashes in this test. That would be more of a smoking gun - most crashes are 2022-05-05 or later. There was patch <a href="https://review.whamcloud.com/46693" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46693</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14826" title="enable striped directory as NFS export" class="issue-link" data-issue-key="LU-14826"><del>LU-14826</del></a> tests: fix sanityn.sh test_102 version check</tt>" landed on that day, and part of that test was previously always skipped due to a syntax error.</p> <p>Seems to be reproduced for fstype=zfs, mdtcount=4 (or &gt; 1?) by sanityn test_102 followed by umount. See <a href="https://review.whamcloud.com/#/c/47277/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/47277/</a> and <a href="https://testing.whamcloud.com/test_sets/74867bfb-0bd6-4bec-80f4-16c5f67d8d7c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/74867bfb-0bd6-4bec-80f4-16c5f67d8d7c</a></p> <p>"John L. Hammond &lt;jhammond@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47311" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47311</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> test: disable sanityn test_102() for zfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e280988367b8adf4391402a46a7c90333a7d3a80</p> <p>&gt; Seems to be reproduced for fstype=zfs, mdtcount=4 (or &gt; 1?) by sanityn test_102 followed by umount.</p> <p>Correction. zfs is just more likely to produce the timing needed for this.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47086/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47086/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> llite: check s_root ll_md_blocking_ast()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0095c0d05ca80a2494710e3b4afb1d1e4b5cdcfe</p> <p>This was introduced by patch <a href="https://review.whamcloud.com/40293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40293</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-6142" title="Enforce Linux kernel coding style in all code" class="issue-link" data-issue-key="LU-6142">LU-6142</a> lustre: use is_root_inode()</tt>" landed as commit v2_14_50-100-gfca56be02b.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47311/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47311/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> test: disable sanityn test_102() for zfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a8da1802236c4270dddb8fe88251de4b86d66084</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48406" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48406</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> llite: check s_root ll_md_blocking_ast()<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: bede6b390c455c40a858de0a48d544970892aeb1</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48406/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48406/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15757" title="sanityn test_109: Oops in ll_md_blocking_ast() at umount" class="issue-link" data-issue-key="LU-15757"><del>LU-15757</del></a> llite: check s_root ll_md_blocking_ast()<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: e6dd92e27af98b38ba5bd8e7e818efa82971a145</p> Related LU-15835 LU-6142 LU-15305 Development Rank 1|i02nbz: Rank (Obsolete) 9223372036854775807 Severity