https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7422 Lustre <p>In case of DISP_OPEN_CREATE client waits for valid fid value in reply when it_status == 0.<br/> When reint_open returns ENOENT fid is not set and client gets fid filled by 0. This may cause following panic:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt; LustreError: 19177:0:(llite_lib.c:2178:ll_prep_inode()) ASSERTION( fid_is_sane(&amp;md.body-&gt;fid1) ) failed: &gt; LustreError: 19177:0:(llite_lib.c:2178:ll_prep_inode()) LBUG &gt; Pid: 19177, comm: cp &gt; &gt; Call Trace: &gt; [&lt;ffffffff81006109&gt;] try_stack_unwind+0x169/0x1b0 &gt; [&lt;ffffffff81004b99&gt;] dump_trace+0x89/0x440 &gt; [&lt;ffffffffa036b8c7&gt;] libcfs_debug_dumpstack+0x57/0x80 [libcfs] &gt; [&lt;ffffffffa036be27&gt;] lbug_with_loc+0x47/0xc0 [libcfs] &gt; [&lt;ffffffffa0a21184&gt;] ll_prep_inode+0xc04/0xe40 [lustre] &gt; [&lt;ffffffffa0a34721&gt;] ll_create_nd+0x3d1/0xec0 [lustre] &gt; [&lt;ffffffff81163fe9&gt;] vfs_create+0xa9/0xf0 &gt; [&lt;ffffffff811646aa&gt;] do_last+0x67a/0x7d0 &gt; [&lt;ffffffff81165358&gt;] path_openat+0xc8/0x3d0 &gt; [&lt;ffffffff81165788&gt;] do_filp_open+0x48/0xa0 &gt; [&lt;ffffffff8115651e&gt;] do_sys_open+0x16e/0x240 &gt; [&lt;ffffffff81156630&gt;] sys_open+0x20/0x30 &gt; [&lt;ffffffff815662eb&gt;] system_call_fastpath+0x16/0x1b &gt; [&lt;00007fc9c3726790&gt;] 0x7fc9c3726790</pre> </div></div> <p>We faced the issue on DNE setup. For unknown reason(possibly failover) FLDB on master mdt didn't include OST seq ranges.<br/> We faced above panic every time after trying to create regular file in directory located on mdt1.</p> LU-7422

incorrect ENOENT handling in mdt_intent_reint causes panic on client side

Bug Critical Resolved Fixed WC Triage Sergey Cheremencev patch Thu, 12 Nov 2015 17:02:53 +0000 Thu, 20 Oct 2016 18:11:19 +0000 Tue, 31 May 2016 12:49:26 +0000 Lustre 2.8.0 Lustre 2.9.0 0 8 <p>What version of Lustre are you testing? The fid_is_sane() check should be skipped if -ENOENT is returned:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> ll_prep_inode(struct inode **inode, struct ptlrpc_request *req, struct super_block *sb, struct lookup_intent *it) { : : rc = md_get_lustre_md(sbi-&gt;ll_md_exp, req, sbi-&gt;ll_dt_exp, sbi-&gt;ll_md_exp, &amp;md); <span class="code-keyword">if</span> (rc != 0) GOTO(cleanup, rc); : : /* * At <span class="code-keyword">this</span> point server returns to client's same fid as client * generated <span class="code-keyword">for</span> creating. So using -&gt;fid1 is okay here. */ LASSERT(fid_is_sane(&amp;md.body-&gt;mbo_fid1)); </pre> </div></div> <p>Sergey Cheremencev (sergey_cheremencev@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/17177" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17177</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7422" title="incorrect ENOENT handling in mdt_intent_reint causes panic on client side" class="issue-link" data-issue-key="LU-7422"><del>LU-7422</del></a> mdt: fix ENOENT handling in mdt_intent_reint<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d1aeb6446540ef375bcade3f7d1215ad598d0675</p> <p>The problem faced on lustre 2.5.1.</p> <blockquote><p>The fid_is_sane() check should be skipped if -ENOENT is returned:</p></blockquote> <p>Yes. But the problem here is that 0 returned instead of -ENOENT.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int mdt_intent_reint(enum mdt_it_code opcode, struct mdt_thread_info *info, struct ldlm_lock **lockp, __u64 flags) ... if (rep-&gt;lock_policy_res2 == -ENOENT &amp;&amp; mdt_get_disposition(rep, DISP_LOOKUP_NEG)) rep-&gt;lock_policy_res2 = 0;</pre> </div></div> <p>Error happened also during soak testing of master branch (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20151122" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20151122</a>). DNE is enabled. MDSes are in active-active HA failover configuration.</p> <p>Here the error occurred after the remount of the MDTs on MDS (<tt>lola-10</tt>) completed successful (2015-11-26 00:27:36).<br/> Pasted the stack trace ones more, as context seems to be different than for the one above.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 26 00:29:48 lola-29 kernel: LustreError: 65535:0:(llite_lib.c:2295:ll_prep_inode()) ASSERTION( fid_is_sane(&amp;md.body-&gt;mbo_fid1) ) failed: Nov 26 00:29:48 lola-29 kernel: LustreError: 65535:0:(llite_lib.c:2295:ll_prep_inode()) LBUG Nov 26 00:29:48 lola-29 kernel: Pid: 65535, comm: pct Nov 26 00:29:48 lola-29 kernel: Nov 26 00:29:48 lola-29 kernel: Call Trace: Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa050b875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa050be77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa0abdb62&gt;] ll_prep_inode+0x752/0xc40 [lustre] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa0802c10&gt;] ? lustre_swab_mdt_body+0x0/0x130 [ptlrpc] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa0ad29d2&gt;] ll_new_node+0x682/0x7f0 [lustre] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffffa0ad5224&gt;] ll_mkdir+0x104/0x220 [lustre] Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff8122ec0f&gt;] ? security_inode_permission+0x1f/0x30 Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff8119d759&gt;] vfs_mkdir+0xd9/0x140 Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff811a04e7&gt;] sys_mkdirat+0xc7/0x1b0 Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff8100c6f5&gt;] ? math_state_restore+0x45/0x60 Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff811a05e8&gt;] sys_mkdir+0x18/0x20 Nov 26 00:29:48 lola-29 kernel: [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b Nov 26 00:29:48 lola-29 kernel: Nov 26 00:29:48 lola-29 kernel: LustreError: dumping log to /tmp/lustre-log.1448526588.65535 </pre> </div></div> <p>Chronologically this event can be correlated to the following error on <tt>lola-10</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lola-10.log:Nov 26 00:29:26 lola-10 kernel: Lustre: 3875:0:(client.c:1994:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448526555/real 1448526555] req@ffff88041c873380 x1518889781449540/t0( 0) o400-&gt;soaked-MDT0004-osp-MDT0005@0@lo:24/4 lens 224/224 e 1 to 1 dl 1448526560 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 lola-10.log:Nov 26 00:29:28 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140549951, ql: 10, comp: 6, conn: 16, next: 992140549952 , next_update 992140549968 last_committed: 992140549904) lola-10.log:Nov 26 00:29:28 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140549956, ql: 10, comp: 6, conn: 16, next: 992140549957 , next_update 992140549968 last_committed: 992140549904) etc... etc... lola-10.log:Nov 26 00:29:34 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140550321, ql: 10, comp: 6, conn: 16, next: 992140550322 , next_update 992140550332 last_committed: 992140550276) lola-10.log:Nov 26 00:29:34 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140550327, ql: 10, comp: 6, conn: 16, next: 992140550328 , next_update 992140550332 last_committed: 992140550276) lola-10.log:Nov 26 00:29:35 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140550374, ql: 10, comp: 6, conn: 16, next: 992140550375 , next_update 992140550385 last_committed: 992140550276) lola-10.log:Nov 26 00:29:35 lola-10 kernel: LustreError: 5620:0:(ldlm_lib.c:1883:check_for_next_transno()) soaked-MDT0004: waking for gap in transno, VBR is OFF (skip: 992140550379, ql: 10, comp: 6, conn: 16, next: 992140550380 , next_update 992140550385 last_committed: 992140550276) ... etc... ... lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: 5620:0:(ldlm_lib.c:1946:target_recovery_overseer()) recovery is aborted, evict exports in recovery lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: 5620:0:(ldlm_lib.c:1946:target_recovery_overseer()) Skipped 1 previous similar message lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: soaked-MDT0004: disconnecting 10 stale clients lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: 5620:0:(ldlm_lib.c:1580:abort_req_replay_queue()) @@@ aborted: req@ffff8808216889c0 x1518687394127176/t0(992140550777) o36-&gt;a8187858-2e65-5add-6cdf-c58a76255990@192.168.1.126@o2ib100:210/0 lens 624/0 e 0 to 0 dl 1448526600 ref 1 fl Complete:/4/ffffffff rc 0/-1 lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: 5620:0:(ldlm_lib.c:1580:abort_req_replay_queue()) Skipped 8 previous similar messages lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: soaked-MDT0004: Denying connection for new client 429ceaa9-f9a8-504b-0d85-c997f52f2036(at 192.168.1.130@o2ib100), waiting for 16 known clients (6 recovered, 0 in progress, and 10 evicted) to recover in 2:25 lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: soaked-MDT0004: Recovery over after 2:04, of 16 clients 6 recovered and 10 were evicted. lola-10.log:Nov 26 00:29:40 lola-10 kernel: Lustre: Skipped 9 previous similar messages lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550816 batchid = 987843952824 flags = 0 ops = 5 params = 9 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550834 batchid = 987843952825 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550870 batchid = 987843952826 flags = 0 ops = 5 params = 9 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550885 batchid = 987843952827 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550923 batchid = 987843952828 flags = 0 ops = 6 params = 10 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550934 batchid = 987843952829 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550960 batchid = 987843952830 flags = 0 ops = 5 params = 9 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140550975 batchid = 987843952831 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140551007 batchid = 987843952832 flags = 0 ops = 5 params = 9 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140551023 batchid = 987843952833 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140551054 batchid = 987843952834 flags = 0 ops = 5 params = 9 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140551065 batchid = 987843952835 flags = 0 ops = 5 params = 4 lola-10.log:Nov 26 00:29:40 lola-10 kernel: LustreError: 5620:0:(update_records.c:72:update_records_dump()) master transno = 992140551096 batchid = 987843952836 flags = 0 ops = 5 params = 9 </pre> </div></div> <p>(Gap in trans. is an old bug INTL-156)</p> <p>Attached files: soak.log (to show events); client <tt>lola-29</tt>) kernel debug log file, messages, console logs; mds <tt>lola-10</tt> messages, console logs</p> <p>Maloo set -1 because there are 2 test failures:<br/> 1. sanity 230f - it is marked as known but <a href="https://jira.whamcloud.com/browse/LU-7549" title="sanity test_230f: FAIL: migrate dir fails" class="issue-link" data-issue-key="LU-7549"><del>LU-7549</del></a><br/> 2. conf-sanity 51. I don't see how it can be connected with my patch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:32:55:LustreError: 3731:0:(lov_object.c:278:lov_init_raid0()) ASSERTION( subdev != NULL ) failed: not init ost 1 15:32:55:LustreError: 3731:0:(lov_object.c:278:lov_init_raid0()) LBUG 15:32:55:Pid: 3731, comm: touch 15:32:55: 15:32:55:Call Trace: 15:32:55: [&lt;ffffffffa03e3875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 15:32:55: [&lt;ffffffffa03e3e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 15:32:55: [&lt;ffffffffa0ab1239&gt;] lov_init_raid0+0xde9/0x1140 [lov] 15:32:55: [&lt;ffffffffa0aadea4&gt;] lov_object_init+0x124/0x300 [lov] 15:32:55: [&lt;ffffffffa078621c&gt;] ? lu_object_add+0x2c/0x30 [obdclass] 15:32:55: [&lt;ffffffffa0788e48&gt;] lu_object_alloc+0xd8/0x320 [obdclass] 15:32:55: [&lt;ffffffffa078a1ad&gt;] lu_object_find_try+0xc03e&gt;] ? do_filp_open+0xa8e/0xd20 15:32:55: [&lt;ffffffff81156a00&gt;] ? __vma_link_rb+0x30/0x40 15:32:55: [&lt;ffffffff81156aab&gt;] ? vma_link+0x9b/0xf0 15:32:55: [&lt;ffffffff8129dc3a&gt;] ? strncpy_from_user+0x4a/0x90 15:32:55: [&lt;ffffffff811b1d52&gt;] ? alloc_fd+0x92/0x160 15:32:55: [&lt;ffffffff8118e847&gt;] ? do_sys_open+0x67/0x130 15:32:55: [&lt;ffffffff8118e950&gt;] ? sys_open+0x20/0x30 15:32:55: [&lt;ffffffff8100b0d2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Used build '20160104' from branch master (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160104" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160104</a>).<br/> DNE is enabled and MDSes are in active-active HA configuration. MDTs have been formatted using <em>ldiskfs</em>, OSTs using <em>zfs</em>.</p> <p>Original error happens again for two client nodes at almost the same time during failback of MDS resources to primary node while soak testing the build specified above.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;LustreError: 75738:0:(llite_lib.c:2295:ll_prep_inode()) ASSERTION( fid_is_sane(&amp;md.body-&gt;mbo_fid1) ) failed: &lt;0&gt;LustreError: 75738:0:(llite_lib.c:2295:ll_prep_inode()) LBUG &lt;4&gt;Pid: 75738, comm: mdtest ... &lt;0&gt;Kernel panic - not syncing: LBUG &lt;4&gt;Pid: 75738, comm: mdtest Not tainted 2.6.32-504.30.3.el6.x86_64 #1 </pre> </div></div> <p>Crash dump files have been written for both nodes (<tt>lola-26,29</tt>) and have been saved to <tt>lola-1:/scratch/crashdumps/lu-7422/lola-26-127.0.0.1-2016-01-05-19:02:53 , lola-29-127.0.0.1-2016-01-05-19:02:56</tt>. Log files can be provided on demand.</p> <p>Sergey Cheremencev (sergey_cheremencev@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/17985" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17985</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7422" title="incorrect ENOENT handling in mdt_intent_reint causes panic on client side" class="issue-link" data-issue-key="LU-7422"><del>LU-7422</del></a> llite: don't panic when fid is insane<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 77306b00cd1b3536f8d6b4e9b25f7b065f737ec7</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17177/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17177/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7422" title="incorrect ENOENT handling in mdt_intent_reint causes panic on client side" class="issue-link" data-issue-key="LU-7422"><del>LU-7422</del></a> mdt: fix ENOENT handling in mdt_intent_reint<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ec9078afb635e8a64a4906b09dc99a2ab90e321b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17985/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17985/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7422" title="incorrect ENOENT handling in mdt_intent_reint causes panic on client side" class="issue-link" data-issue-key="LU-7422"><del>LU-7422</del></a> llite: don't panic when fid is insane<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e4eb90ecdc09740d90834cb1e95b5693e6637173</p> <p>Landed for 2.9</p> Duplicate LU-7798 Related Development Rank 1|hzxswn: Rank (Obsolete) 9223372036854775807 Severity