https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16872 Lustre <p>This issue was created by maloo for Frank Sehr &lt;fsehr@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/46d1b101-8e4f-4415-bb49-ee39963275fe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/46d1b101-8e4f-4415-bb49-ee39963275fe</a></p> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/95319" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/95319</a> - 4.18.0-425.10.1.el8_7.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/95319" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/95319</a> - 4.18.0-425.10.1.el8_lustre.x86_64</p> <p>== sanity test 27M: test O_APPEND striping =============== 03:40:09 (1685936409)<br/> CMD: trevis-129vm4 /usr/sbin/lctl get_param -n version 2&gt;/dev/null<br/> striped dir -i3 -c2 -H crush2 /mnt/lustre/d27M.sanity<br/> CMD: trevis-129vm4 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.append_pool<br/> CMD: trevis-129vm4 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.append_stripe_count<br/> CMD: trevis-129vm4,trevis-129vm5 /usr/sbin/lctl set_param mdd.*.append_stripe_count=0<br/> mdd.lustre-MDT0000.append_stripe_count=0<br/> mdd.lustre-MDT0002.append_stripe_count=0<br/> mdd.lustre-MDT0001.append_stripe_count=0<br/> mdd.lustre-MDT0003.append_stripe_count=0<br/> CMD: trevis-129vm4,trevis-129vm5 /usr/sbin/lctl set_param mdd.*.append_stripe_count=2<br/> mdd.lustre-MDT0000.append_stripe_count=2<br/> mdd.lustre-MDT0002.append_stripe_count=2<br/> mdd.lustre-MDT0001.append_stripe_count=2<br/> mdd.lustre-MDT0003.append_stripe_count=2<br/> CMD: trevis-129vm4,trevis-129vm5 /usr/sbin/lctl set_param mdd.*.append_stripe_count=-1<br/> mdd.lustre-MDT0000.append_stripe_count=-1<br/> mdd.lustre-MDT0002.append_stripe_count=-1<br/> mdd.lustre-MDT0001.append_stripe_count=-1<br/> mdd.lustre-MDT0003.append_stripe_count=-1<br/> /usr/lib64/lustre/tests/sanity.sh: line 3101: /mnt/lustre/d27M.sanity/f27M.sanity.5: Invalid argument<br/> lfs: getstripe for '/mnt/lustre/d27M.sanity/f27M.sanity.5' failed: No such file or directory<br/> /usr/lib64/lustre/tests/sanity.sh: line 3103: [: -eq: unary operator expected<br/> sanity test_27M: @@@@@@ FAIL: (5) stripe count , should be 8 for append <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:6585:error()<br/> = /usr/lib64/lustre/tests/sanity.sh:3104:test_27M()<br/> = /usr/lib64/lustre/tests/test-framework.sh:6925:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:6974:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:6811:run_test()<br/> = /usr/lib64/lustre/tests/sanity.sh:3181:main()<br/> Dumping lctl log to /autotest/autotest-2/2023-06-05/lustre-reviews_review-ldiskfs-dne_95319_27_26b27a74-2421-4453-9c33-cd237feca413//sanity.test_27M.*.1685936415.log<br/> CMD: trevis-129vm1.trevis.whamcloud.com,trevis-129vm2,trevis-129vm3,trevis-129vm4,trevis-129vm5 /usr/sbin/lctl dk &gt; /autotest/autotest-2/2023-06-05/lustre-reviews_review-ldiskfs-dne_95319_27_26b27a74-2421-4453-9c33-cd237feca413//sanity.test_27M.debug_log.\$(hostname -s).1685936415.log;<br/> dmesg &gt; /autotest/autotest-2/2023-06-05/lustre-reviews_review-ldiskfs-dne_95319_27_26b27a74-2421-4453-9c33-cd237feca413//sanity.test_27M.dmesg.\$(hostname -s).1685936415.log<br/> CMD: trevis-129vm4,trevis-129vm5 /usr/sbin/lctl set_param mdd.*.append_stripe_count=1<br/> mdd.lustre-MDT0000.append_stripe_count=1<br/> mdd.lustre-MDT0002.append_stripe_count=1<br/> mdd.lustre-MDT0001.append_stripe_count=1<br/> mdd.lustre-MDT0003.append_stripe_count=1<br/> CMD: trevis-129vm4,trevis-129vm5 /usr/sbin/lctl set_param mdd.*.append_pool=none<br/> mdd.lustre-MDT0000.append_pool=none<br/> mdd.lustre-MDT0002.append_pool=none<br/> mdd.lustre-MDT0001.append_pool=none<br/> mdd.lustre-MDT0003.append_pool=none</p> LU-16872

sanity: test_27M Error: '(5) stripe count , should be 8 for append'

Bug Critical Resolved Fixed Thomas Bertschinger Maloo Mon, 5 Jun 2023 17:19:26 +0000 Tue, 16 Jan 2024 21:34:00 +0000 Sat, 19 Aug 2023 12:05:24 +0000 Lustre 2.16.0 0 7 <p>Andreas - I'm away on vacation through June 25 so if this bug needs a quick resolution, you may want to have someone else look into it. Otherwise I'll continue to work on this when I get back next week.</p> <p>+1 on master (<a href="https://testing.whamcloud.com/sub_tests/399cef35-a321-43da-944a-84f8ce67c9f1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/399cef35-a321-43da-944a-84f8ce67c9f1</a>)</p> <p>Here's a status update since the bug is still causing failures. I'm still looking at this but have not found the root cause yet, but I have a hypothesis.</p> <p>The origin of the <tt>EINVAL</tt> response to the <tt>open()</tt> call is <tt>lod_alloc_ost_list()</tt>. From the debug logs, it seems it must be one of these:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">               <span class="code-keyword">if</span> (!test_bit(ost_idx, m-&gt;lod_ost_bitmap)) {                         rc = -EINVAL;                         <span class="code-keyword">break</span>;                 } ...                 <span class="code-keyword">if</span> (lod_qos_is_tgt_used(env, ost_idx, stripe_count) &amp;&amp;                     !(lod_comp-&gt;llc_pattern &amp; LOV_PATTERN_OVERSTRIPING)) {                         rc = -EINVAL;                         <span class="code-keyword">break</span>;                 } </pre> </div></div> <p>However, I am fairly confident that <tt>lod_alloc_ost_list()</tt> should NOT be getting called in the append case at all, because this function appears to be for files with specifically set stripes, e.g., it gets called if a file inherits custom striping from its parent directory.</p> <p>Therefore, I believe the actual problem is that <tt>lod_comp-&gt;llc_ostlist.op_array</tt> and <tt>lod_comp-&gt;llc_ostlist.op_count</tt> are incorrectly non-zero when the failure occurs. My current hypothesis for the cause is this: the <tt>struct lu_tgt_pool</tt> inside <tt>struct lod_layout_component</tt> (where the op_array and op_count fields are) is preceded by <tt>char *llc_pool</tt> so it's possible that <tt>char *</tt> is being overflowed and putting garbage values into the array and count fields. So currently I'm looking at where <tt>llc_pool</tt> is set and if any of these spots could plausibly have an overflow.</p> <p>Oops, I wasn't thinking straight this morning but overflowing llc_pool wouldn't affect op_array and op_count since the buffer wouldn't be adjacent to these fields. So I'm still looking for what could cause op_array and op_count to have bad values. (I still think that's the most likely explanation for the issue.)</p> <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51559" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51559</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16872" title="sanity: test_27M Error: &#39;(5) stripe count , should be 8 for append&#39;" class="issue-link" data-issue-key="LU-16872"><del>LU-16872</del></a> lod: do not stripe O_APPEND files on specific OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 81e2439da705dd35dc2c8c687be21cf7dc952eba</p> <p>How to reproduce:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-bash"> <span class="code-comment"># setup: </span>$ lctl set_param mdd.*.append_stripe_count=-1 $ lfs setstripe -o 1,3 /mnt/lustre <span class="code-comment"># touch enough files with the default striping so that every mdt kernel thread probably has the defaults stored <span class="code-object">in</span> its memory </span>$ <span class="code-object">for</span> i <span class="code-object">in</span> {0..100}; <span class="code-object">do</span> touch /mnt/lustre/x<span class="code-object">$i</span>; <span class="code-object">done</span> <span class="code-comment"># now an append should return EINVAL as long as it gets handled by a kernel thread that previously did a create with default stripes </span>$ echo 1 &gt;&gt; /mnt/lustre/f -bash: /mnt/lustre/g: Invalid argument</pre> </div></div> <p>A closely related problem occurs when an append_pool is set, but in this case, the create succeeds, but the append file is created with the default stripes rather than the pool.</p> <p>I haven't identified which patch caused (or uncovered) the issue yet – I didn't see anything obvious in the patches merged shortly before the first test failure occurred. So I'll attempt a git bisect to try to find what caused this and will update if I get that answer.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51602" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51602</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16872" title="sanity: test_27M Error: &#39;(5) stripe count , should be 8 for append&#39;" class="issue-link" data-issue-key="LU-16872"><del>LU-16872</del></a> tests: exercise sanity test_27M more fully<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 221a2d05d5d4ec2b39c88c6a5d84df2ba3f177dc</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51559/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51559/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16872" title="sanity: test_27M Error: &#39;(5) stripe count , should be 8 for append&#39;" class="issue-link" data-issue-key="LU-16872"><del>LU-16872</del></a> lod: reset llc_ostlist when using O_APPEND stripes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 766b35a9700f36aa08b652fa9d18b890d34bf4a5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51602/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51602/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16872" title="sanity: test_27M Error: &#39;(5) stripe count , should be 8 for append&#39;" class="issue-link" data-issue-key="LU-16872"><del>LU-16872</del></a> tests: exercise sanity test_27M more fully<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7bb1685048bf999df03ceadab39faa09b8a5560d</p> <p>Landed for 2.16</p> Duplicate LU-16014 LU-16147 LU-16648 Related LU-11279 LU-15727 LU-16648 Development Rank 1|i03n6f: Rank (Obsolete) 9223372036854775807 Severity