https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14658 Lustre <p>This issue was created by maloo for Chris Horn &lt;hornc@cray.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/8d081b66-0a6e-453a-b3b7-a74c663621c6" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8d081b66-0a6e-453a-b3b7-a74c663621c6</a></p> <p>test_122b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>failed to write file </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-205vm14.trevis.whamcloud.com,trevis-205vm15 mcreate /mnt/lustre/fsa-\$(hostname); rm /mnt/lustre/fsa-\$(hostname) CMD: trevis-205vm14.trevis.whamcloud.com,trevis-205vm15 if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-\$(hostname); rm /mnt/lustre2/fsa-\$(hostname); fi CMD: trevis-205vm16 /usr/sbin/lctl --device lustre-OST0000 notransno CMD: trevis-205vm16 /usr/sbin/lctl --device lustre-OST0000 readonly CMD: trevis-205vm16 /usr/sbin/lctl mark ost1 REPLAY BARRIER on lustre-OST0000 dd: failed to open '/mnt/lustre/d122b.conf-sanity/f122b.conf-sanity': Numerical result out of range conf-sanity test_122b: @@@@@@ FAIL: failed to write file </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> conf-sanity test_122b - failed to write file</p> LU-14658

conf-sanity test_122b: failed to write file

Bug Minor Resolved Fixed Alexander Boyko Maloo Fri, 30 Apr 2021 15:48:49 +0000 Thu, 2 Jun 2022 03:35:10 +0000 Thu, 6 Jan 2022 23:24:04 +0000 Lustre 2.12.9 Lustre 2.15.0 0 7 <p>Hi Alexander,<br/> it looks like the new conf-sanity.sh test_122b that was added with patch <a href="https://review.whamcloud.com/43248" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43248</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14598" title="Too many FIDs to precreate OST replaced. Wrong IDIF logic." class="issue-link" data-issue-key="LU-14598"><del>LU-14598</del></a> ofd: fix for IDIF sequence at ofd_preprw_write</tt>" has been failing intermittently since it was landed on 2021-04-28 (37 times in the past 35 days):<br/> <a href="https://testing.whamcloud.com/search?server_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;status%5B%5D=FAIL&amp;test_set_script_id=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;sub_test_script_id=59bb51ec-a2ae-11e6-8b77-5254006e85c2&amp;start_date=2021-03-01&amp;end_date=2021-06-02&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?server_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;status%5B%5D=FAIL&amp;test_set_script_id=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;sub_test_script_id=59bb51ec-a2ae-11e6-8b77-5254006e85c2&amp;start_date=2021-03-01&amp;end_date=2021-06-02&amp;source=sub_tests#redirect</a></p> <p>for example, the most recent two failures are at:<br/> <a href="https://testing.whamcloud.com/sub_tests/00c3de0d-a963-4913-b117-9b69ef27b604" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/00c3de0d-a963-4913-b117-9b69ef27b604</a><br/> <a href="https://testing.whamcloud.com/sub_tests/a7dc2041-7315-416b-8472-fbe86791bc9f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/a7dc2041-7315-416b-8472-fbe86791bc9f</a></p> <p>Could you please take a look and see how the test/code can be changed so that the test is more robust. Thanks.</p> <p>Note that all of the actual failures of this subtest have been with ZFS, so it may be something specific with osd-zfs, or just a timing issue because it is slower?</p> <p>There are two failures for the same subtest with ldiskfs, but they reported a different error message and were both caused by the (same) patch that was being tested.</p> <p>About 12x failures of conf-sanity test_122b in the past week, of 129 test runs:<br/> <a href="https://testing.whamcloud.com/sub_tests/06b6cedf-12ce-4873-8c73-2b1cdf3e6e5d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/06b6cedf-12ce-4873-8c73-2b1cdf3e6e5d</a><br/> <a href="https://testing.whamcloud.com/sub_tests/dcec0d91-1cbe-40ad-a142-1043b7bb9f41" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/dcec0d91-1cbe-40ad-a142-1043b7bb9f41</a></p> <p>Looking at the debug logs, it appears that the OSTs are being marked with <tt>state=2=OS_STATFS_READONLY</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1635276069.251190:0:2262865:0:(osp_precreate.c:1050:osp_pre_update_msfs()) lustre-OST0000-osc-MDT0002: blocks=95744 free=94720 avail=89344 avail_mb=349 hwm_mb=3 files=16043 ffree=11840 state=2: rc = 0 1635276069.759773:0:2262864:0:(osp_precreate.c:1050:osp_pre_update_msfs()) lustre-OST0000-osc-MDT0000: blocks=95744 free=94720 avail=89344 avail_mb=349 hwm_mb=3 files=16043 ffree=11840 state=2: rc = 0 </pre> </div></div> <p>and then the object allocation fails because there are no OSTs available:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1635276069.784058:0:2432715:0:(lod_qos.c:142:lod_statfs_and_check()) lustre-OST0000-osc-MDT0000: turns inactive 1635276069.784058:0:2432715:0:(lod_qos.c:170:lod_statfs_and_check()) Process leaving (rc=18446744073709551586 : -30 : ffffffffffffffe2) 1635276069.784059:0:2432715:0:(lod_qos.c:231:lod_qos_statfs_update()) Process leaving 1635276069.784066:0:2432715:0:(lod_qos.c:2541:lod_qos_prep_create()) Process leaving via out (rc=18446744073709551582 : -34 : 0xffffffffffffffde) 1635276069.784067:0:2432715:0:(lod_qos.c:2639:lod_qos_prep_create()) Process leaving (rc=18446744073709551582 : -34 : ffffffffffffffde) 1635276069.784067:0:2432715:0:(lod_qos.c:2695:lod_prepare_create()) Process leaving (rc=18446744073709551582 : -34 : ffffffffffffffde) 1635276069.784071:0:2432715:0:(lod_object.c:5804:lod_declare_striped_create()) Process leaving via out (rc=18446744073709551582 : -34 : 0xffffffffffffffde) 1635276069.784072:0:2432715:0:(lod_lov.c:492:lod_free_comp_buffer()) kfreed 'entries': 128 at 00000000b6ba0656. 1635276069.784073:0:2432715:0:(lod_object.c:5839:lod_declare_striped_create()) Process leaving (rc=18446744073709551582 : -34 : ffffffffffffffde) 1635276069.784074:0:2432715:0:(lod_object.c:5985:lod_declare_create()) Process leaving (rc=18446744073709551582 : -34 : ffffffffffffffde) 1635276069.784075:0:2432715:0:(mdd_object.c:576:mdd_declare_create_object_internal()) Process leaving (rc=18446744073709551582 : -34 : ffffffffffffffde) 1635276069.784076:0:2432715:0:(mdd_dir.c:2167:mdd_declare_create_object()) Process leaving via out (rc=18446744073709551582 : -34 : 0xffffffffffffffde) 1635276069.784077:0:2432715:0:(mdd_dir.c:2267:mdd_declare_create()) Process leaving via out (rc=18446744073709551582 : -34 : 0xffffffffffffffde) 1635276069.784078:0:2432715:0:(mdd_dir.c:2684:mdd_create()) Process leaving via out_stop (rc=18446744073709551582 : -34 : 0xffffffffffffffde) </pre> </div></div> <p>The <tt>-34 = -ERANGE = Numerical result out of range</tt> is because:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> __u16 lod_get_stripe_count(struct lod_device *lod, struct lod_object *lo, <span class="code-object">int</span> comp_idx, __u16 stripe_count, bool overstriping) { : <span class="code-keyword">if</span> (!stripe_count) stripe_count = lod-&gt;lod_ost_descs.ltd_lov_desc.ld_default_stripe_count; <span class="code-keyword">if</span> (!stripe_count) stripe_count = 1; <span class="code-keyword">if</span> (stripe_count &gt; lod-&gt;lod_ost_descs.ltd_lov_desc.ld_active_tgt_count &amp;&amp; !overstriping) stripe_count = lod-&gt;lod_ost_descs.ltd_lov_desc.ld_active_tgt_count; </pre> </div></div> <p>so it appears that <tt>ld_active_tgt_count = 0</tt>, and this is converted to <tt>-ERANGE</tt> in the caller:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> lod_qos_prep_create(<span class="code-keyword">const</span> struct lu_env *env, struct lod_object *lo, struct lu_attr *attr, struct thandle *th, <span class="code-object">int</span> comp_idx, __u64 reserve) { : stripe_len = lod_get_stripe_count(d, lo, comp_idx, lod_comp-&gt;llc_stripe_count, lod_comp-&gt;llc_pattern &amp; LOV_PATTERN_OVERSTRIPING); <span class="code-keyword">if</span> (stripe_len == 0) GOTO(out, rc = -ERANGE); </pre> </div></div> <p>With a fleeting glance the problem starts at readonly simulation. For ldiskfs it is a drop write IO, and for zfs lctl --device readonly. I have no idea how others tests with replay_barrier() works, the logic is too diferrent between zfs/ldiskfs. For ldiskfs environment OS_STATFS_READONLY flag is not set after replay_barrier(). The quick fix is skipping test for zfs env. I'm not familiar with zfs, does it support readonly mode similar to flakey?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># # Set the device of a given facet as "read-only". # set_dev_readonly() { local facet=$1 local svc=${facet}_svc if [[ $(facet_fstype $facet) = zfs ]] || ! dm_flakey_supported $facet; then do_facet $facet $LCTL --device ${!svc} readonly else dm_set_dev_readonly $facet fi }</pre> </div></div> <p>I think ZFS supports read-only, otherwise that would cause a lot more tests to fail on ZFS?</p> <p>"Alexander Boyko &lt;alexander.boyko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45841" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45841</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14658" title="conf-sanity test_122b: failed to write file" class="issue-link" data-issue-key="LU-14658"><del>LU-14658</del></a> tests: fix conf-sanity 122b test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b447c4fea70d799bf13ac1e5236ca2d83b148436</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45841/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45841/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14658" title="conf-sanity test_122b: failed to write file" class="issue-link" data-issue-key="LU-14658"><del>LU-14658</del></a> tests: fix conf-sanity 122b test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 276c22ae3869c1f775cded248d53b9c2a829c801</p> <p>Landed for 2.15</p> <p>+1 on b2_12: <a href="https://testing.whamcloud.com/test_sets/865cf735-e67b-4701-abc3-91d0c97013eb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/865cf735-e67b-4701-abc3-91d0c97013eb</a><br/> <a href="https://testing.whamcloud.com/sub_tests/f1a6c894-d8ef-49ec-89bd-c33d57fdfafa" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/f1a6c894-d8ef-49ec-89bd-c33d57fdfafa</a></p> <p>The message is different:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd: failed to open '/mnt/lustre/d122b.conf-sanity/f122b.conf-sanity': No space left on device 00000002:00100000:1.0:1647298582.780569:0:3559:0:(mdc_locks.c:709:mdc_finish_enqueue()) @@@ op: 3 disposition: 17, status: -28 req@ffff9132ca930d80 x1727317208645248/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff913275c8c000@10.240.44.32@tcp:12/10 lens 664/568 e 0 to 0 dl 1647298589 ref 1 fl Complete:R/0/0 rc 301/301 </pre> </div></div> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46873" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46873</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14658" title="conf-sanity test_122b: failed to write file" class="issue-link" data-issue-key="LU-14658"><del>LU-14658</del></a> tests: fix conf-sanity 122b test<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 6af0c0f659a1d91785aa53be7c3ff056b44885c4</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46873/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46873/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14658" title="conf-sanity test_122b: failed to write file" class="issue-link" data-issue-key="LU-14658"><del>LU-14658</del></a> tests: fix conf-sanity 122b test<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 3f18d493c0aca46982596fefcc798811680d2bd1</p> Related LU-14598 Development Rank 1|i01ti7: Rank (Obsolete) 9223372036854775807 Severity