https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14202 Lustre <p>sanity-sec test_16 fails for RHEL 8.3 client/server testing. We’ve only seen this test fail once with this error; <a href="https://testing.whamcloud.com/test_sets/1c1e2670-df76-498e-a3a9-d74b490d63a9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1c1e2670-df76-498e-a3a9-d74b490d63a9</a> . </p> <p>Looking at the suite_log for this failure, we see several “pdsh@trevis-8vm7: trevis-8vm8: ssh exited with exit code 2”</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>… pdsh@trevis-8vm7: trevis-8vm8: ssh exited with exit code 2 CMD: trevis-8vm8 runas -u60005 -g60005 -G60005 rm /mnt/lustre/d16.sanity-sec/f16.sanity-sec trevis-8vm8: running as uid/gid/euid/egid 60005/60005/60005/60005, groups: 60005 trevis-8vm8: [rm] [/mnt/lustre/d16.sanity-sec/f16.sanity-sec] CMD: trevis-24vm6 /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1 CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-24vm6 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* CMD: trevis-8vm8 runas -u60005 -g60005 -G60005 lfs quota -q /mnt/lustre trevis-8vm8: running as uid/gid/euid/egid 60005/60005/60005/60005, groups: 60005 trevis-8vm8: [lfs] [quota] [-q] [/mnt/lustre] pdsh@trevis-8vm7: trevis-8vm8: ssh exited with exit code 2 CMD: trevis-8vm7.trevis.whamcloud.com chmod 000 /mnt/lustre/d16.sanity-sec CMD: trevis-8vm7.trevis.whamcloud.com chmod 003 /mnt/lustre/d16.sanity-sec CMD: trevis-8vm7.trevis.whamcloud.com chmod 300 /mnt/lustre/d16.sanity-sec CMD: trevis-8vm7.trevis.whamcloud.com chmod 303 /mnt/lustre/d16.sanity-sec CMD: trevis-8vm7.trevis.whamcloud.com chmod 777 /mnt/lustre/d16.sanity-sec CMD: trevis-8vm8 runas -u60006 -g60006 -G60006 lfs quota -q /mnt/lustre trevis-8vm8: running as uid/gid/euid/egid 60006/60006/60006/60006, groups: 60006 trevis-8vm8: [lfs] [quota] [-q] [/mnt/lustre] pdsh@trevis-8vm7: trevis-8vm8: ssh exited with exit code 2 sanity-sec test_16: @@@@@@ FAIL: unable to write quota test file Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6257:error() = /usr/lib64/lustre/tests/sanity-sec.sh:1290:do_fops_quota_test() = /usr/lib64/lustre/tests/sanity-sec.sh:1483:test_fops() = /usr/lib64/lustre/tests/sanity-sec.sh:1567:test_16() </pre> </div></div> <p>In the client2 (8vm8) console log, we see the following error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[23001.757304] Lustre: DEBUG MARKER: runas -u60006 -g60006 -G60006 dd if=/dev/zero of=/mnt/lustre/d16.sanity-sec/f16.sanity-sec oflag=sync bs=1M count=1 [23002.132686] LustreError: 8532:0:(osc_request.c:1947:osc_brw_fini_request()) lustre-OST0002-osc-ffff8a310b0e7800: unexpected positive size 1 [23002.680119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-sec test_16: @@@@@@ FAIL: unable to write quota test file </pre> </div></div> RHEL 8.3 clients/servers LU-14202

sanity-sec test 16 fails with 'unable to write quota test file'

Bug Minor Resolved Duplicate Sebastien Buisson James Nunez rhel8.3 Tue, 8 Dec 2020 21:03:17 +0000 Mon, 14 Dec 2020 13:28:26 +0000 Mon, 14 Dec 2020 13:28:26 +0000 Lustre 2.14.0 0 4 <p>Sebastien</p> <p>Could you please advise on this one? Will this be addressed by your unhanded patches in this area?</p> <p>Peter</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/40934" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40934</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14202" title="sanity-sec test 16 fails with &#39;unable to write quota test file&#39;" class="issue-link" data-issue-key="LU-14202"><del>LU-14202</del></a> tests: try to reproduce<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e9d63ad0ed558f578ce868e65b2250b26568bfa4</p> <p>This is not an issue I was aware of, and it is not going to be addressed by any pending patch I have.</p> <p>Although the issue is uncovered during sanity-sec, I am not sure it is a security related issue. sanity-sec test_16 just plays with nodemaps, and checks that quotas set are correctly enforced. In the case reported by James, neither SHARED_KEY nor SELinux are being used.</p> <p>The error finally returned by the test comes from a failed attempt to write to a file with dd, as user uid <tt>60006</tt>. The relevant messages on client side are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[23001.757304] Lustre: DEBUG MARKER: runas -u60006 -g60006 -G60006 dd if=/dev/zero of=/mnt/lustre/d16.sanity-sec/f16.sanity-sec oflag=sync bs=1M count=1 [23002.132686] LustreError: 8532:0:(osc_request.c:1947:osc_brw_fini_request()) lustre-OST0002-osc-ffff8a310b0e7800: unexpected positive size 1 [23002.680119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-sec test_16: @@@@@@ FAIL: unable to write quota test file </pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000008:00020000:0.0:1607377928.061849:0:8532:0:(osc_request.c:1947:osc_brw_fini_request()) lustre-OST0002-osc-ffff8a310b0e7800: unexpected positive size 1 00000080:00200000:0.0:1607377928.098285:0:382201:0:(vvp_io.c:1299:vvp_io_write_start()) f16.sanity-sec: write position mismatch: ki_pos 1048576 vs. pos 0, written 1048576, commit 0: rc = -71 00000080:00200000:0.0:1607377928.098314:0:382201:0:(file.c:1639:ll_file_io_generic()) f16.sanity-sec: 2 io complete with rc: -71, result: 0, restart: 0 00000080:00200000:0.0:1607377928.098315:0:382201:0:(file.c:1695:ll_file_io_generic()) iot: 2, result: 0 </pre> </div></div> <p>Unfortunately, on OSS side, the debug level does not leave us with many details:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1607377928.044135:0:602125:0:(service.c:2299:ptlrpc_server_handle_request()) Handling RPC req@0000000066b395ba pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:66c97887-330e-4bd1-930b-29b4c28d8385+6:8532:x1685434418579008:12345-10.9.4.88@tcp:4:dd.60006 00000100:00100000:0.0:1607377928.061461:0:602125:0:(service.c:2348:ptlrpc_server_handle_request()) Handled RPC req@0000000066b395ba pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_io00_002:66c97887-330e-4bd1-930b-29b4c28d8385+7:8532:x1685434418579008:12345-10.9.4.88@tcp:4:dd.60006 Request processed in 17325us (17368us total) trans 0 rc 1/1 </pre> </div></div> <p>At least we can see that the 'unexpected positive 1' reported by the client is in fact returned by the server.</p> <p>As noticed by James, even before the error, there are suspicious ssh problems in the form of <tt>pdsh@trevis-8vm7: trevis-8vm8: ssh exited with exit code 2</tt> in the test log. I have pushed patch #40934 to add more traces and try to reproduce the issue, in the hope the the debug level will be higher (but maybe the default used in Maloo testing is not enough in this case).</p> <p>This may be the same as or related to <a href="https://jira.whamcloud.com/browse/LU-14200" title="ost-pools test 23b fails with &#39;dd did not fail with ENOSPC&#39;" class="issue-link" data-issue-key="LU-14200"><del>LU-14200</del></a> since we see similar errors on the client.</p> <p>Similar issue seen on sanity-sec test 21 at <a href="https://testing.whamcloud.com/test_sets/9f7fdf4a-6ec8-4cd2-9a15-631ece42d85b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9f7fdf4a-6ec8-4cd2-9a15-631ece42d85b</a>.</p> <p>OK, so I think we can definitely take the nodemap feature out of the equation.</p> <p>Could we mark this ticket as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-14200" title="ost-pools test 23b fails with &#39;dd did not fail with ENOSPC&#39;" class="issue-link" data-issue-key="LU-14200"><del>LU-14200</del></a>, as it already references multiple tests like ost-pools, sanity-quota and sanity-flr?</p> <p>Thanks,<br/> Sebastien.</p> <p>Sébastien</p> <p><a href="https://jira.whamcloud.com/browse/LU-14200" title="ost-pools test 23b fails with &#39;dd did not fail with ENOSPC&#39;" class="issue-link" data-issue-key="LU-14200"><del>LU-14200</del></a> has in turn been marked a duplicate of <a href="https://jira.whamcloud.com/browse/LU-14205" title="RHEL8.3: sanity test 398c fails with &#39;fio write error&#39;" class="issue-link" data-issue-key="LU-14205"><del>LU-14205</del></a> so should we mark this one as a duplicate of that?</p> <p>Peter</p> <p>Yes, I think we should mark this one as dup of <a href="https://jira.whamcloud.com/browse/LU-14205" title="RHEL8.3: sanity test 398c fails with &#39;fio write error&#39;" class="issue-link" data-issue-key="LU-14205"><del>LU-14205</del></a>, as it is exactly the same error. Thanks.</p> <p>thanks Sébastien!</p> Duplicate LU-14205 Related LU-14200 Development Rank 1|i01gov: Rank (Obsolete) 9223372036854775807 Severity