https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13400 Lustre <p>This issue was created by maloo for Olaf Faaland &lt;faaland1@llnl.gov&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/150c0a5d-19ca-4d0e-bbd4-5d98ae175f9a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/150c0a5d-19ca-4d0e-bbd4-5d98ae175f9a</a></p> <p>test_300d failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>create 10 files failed </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> == sanity test 300d: check default stripe under striped directory ==================================== 12:17:4 3 (1585595863) 2 open(/mnt/diane/d300d.sanity/striped_dir/f1) error: Permission denied 3 total: 1 open/close in 0.18 seconds: 5.62 ops/second 4 sanity test_300d: @@@@@@ FAIL: create 10 files failed 5 Trace dump: 6 = /home/olaf/lustre/lustre/tests/test-framework.sh:6138:error() 7 = /home/olaf/lustre/lustre/tests/sanity.sh:19623:test_300d() 8 = /home/olaf/lustre/lustre/tests/test-framework.sh:6441:run_one() 9 = /home/olaf/lustre/lustre/tests/test-framework.sh:6490:run_one_logged() 10 = /home/olaf/lustre/lustre/tests/test-framework.sh:6315:run_test() 11 = /home/olaf/lustre/lustre/tests/sanity.sh:19639:main() </pre> </div></div> <p>I can locally reproduce reliably with:<br/> FSTYPE=zfs (zfs-0.7.11 or zfs-0.8.3)<br/> MDSCOUNT=2<br/> lustre 2.12.4, 2.13.0, and master<br/> Run 20 times: sudo lustre/tests/auster sanity --only 300</p> <p>I've seen the issue on average about 15% of runs with zfs-0.8 and less often with zfs-0.7.11</p> <p>I've reproduced the issue with lustre 2.12.4, 2.13.0, master, and master-next.<br/> Earlier lustre versions reliably LBUG before getting this far in test_300, in this configuration, so I don't know if the bug existed in earlier versions.</p> <p>There are no warnings or errors in dmesg, emitted while the test is running (on any node).</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_300d - create 10 files failed</p> LU-13400

sanity test_300d: createmany 10 under striped dir fails with Permission denied

Bug Minor Resolved Duplicate Lai Siyao Maloo llnl Mon, 30 Mar 2020 19:38:56 +0000 Mon, 28 Sep 2020 15:13:32 +0000 Sun, 27 Sep 2020 04:04:34 +0000 0 6 <p>I've replaced the createmany with a loop that does mkdirs, and find that the mkdirs fail for only one of the two MDTs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.1': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.1 failed mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.3': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.3 failed mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.5': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.5 failed mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.7': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.7 failed mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.9': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.9 failed mkdir: cannot create directory '/mnt/diane/d300d.sanity/striped_dir/f.10': Permission denied mkdir /mnt/diane/d300d.sanity/striped_dir/f.10 failed </pre> </div></div> <p>I took a look and the subdirs that were created successfully were on MDT0, so the ones that failed were on MDT1.</p> <p>I'm attempting to compare debug logs for a subdirectory created successfully (f.2) and one which failed (f.1).</p> <p>In both cases, I see lmv_create() correctly identifies the MDT, a metadata modify RPC slot is assigned, an XID is assigned, and the RPC is sent successfully. I believe all that is on the client, but if I'm wrong please let me know.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lmv_create()) CREATE name '&lt;dirname&gt;' [&lt;FID&gt;] on [&lt;SHARDFID&gt;] -&gt; mds #&lt;MDT_INDEX&gt; obd_get_mod_rpc_slot()) &lt;MDT_EXPORT&gt;: modify RPC slot &lt;SLOT_INDEX&gt; is alloca ptlrpc_reassign_next_xid()) @@@ reassign xid req@&lt;reqid&gt; x&lt;xid&gt; ptlrpc_send_new_req()) Sending RPC req@&lt;reqid&gt; and pname:cluuid:pid:xid:nid: ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0)</pre> </div></div> <p>For the successful mkdir, I then see the request is enqueued (which as I understand it occurs in the MDT):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ldlm_cli_enqueue_local()) ### client-side local enqueue handler, new lock created ns: mdt-diane-MDT0000_UUID lock: ffff972604887180/0x56189f42e7cba373 lrc: 3/0,1 mode: CW/CW res: </pre> </div></div> <p>But for the failed mkdir I do not see that.</p> <p>And for the failed mkdir I see ptlrpc_check_status() indicates -EACCES as one might expect</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(client.c:1343:ptlrpc_check_status()) @@@ check status: rc = -13 req@ffff9728e9f40480 x1662629166884800/t0(0) o36-&gt;diane-MDT0001-mdc-ffff9725fdff3800@10.0.2.15@tcp:12/10 lens 568/448 e 0 to 0 d l 1585607163 ref 2 fl Rpc:RQU/0/0 rc 0/-13 job:'mkdir.0'</pre> </div></div> <p> <br/> But I'm having trouble figuring out what key functions to look at on the server side, on MDT1, that cause the request to fail.<br/>  </p> <p>I added the topllnl label. This is not something we are seeing in production (we are currently just testing DNE2) but I would like to track it down or at least help narrow the scope. This test seems to fail fairly frequently.</p> <p>Also, in the debug logs for the successful mkdir I see mdt_handler.c, mdt_reint.c, and mdd_permission.c (e.g. mdd_create() and mdt_object_new()).</p> <p><del>But in the failed mkdir I see none of those.</del></p> <p>I made some minor tweaks to the test (e.g. added lctl mark after each mkdir) and in the new logs I do see mdt_object_new(), mdd_create(), and __mdd_permission_internal() calls, and that last returns -13 as one would expect.</p> <p>I'm not sure whether I overlooked those in the original log, or the failure mode was different.</p> <p>Lai</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>The -13 is coming from here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* * Nobody gets write access to an immutable file. */ if (mask &amp; MAY_WRITE &amp;&amp; la-&gt;la_flags &amp; LUSTRE_IMMUTABLE_FL) RETURN(-EACCES); </pre> </div></div> <p>in __mdd_permission_internal()</p> <p>Thanks Olaf, this is quite helpful, I'm reviewing the code now.</p> <p><a href="https://jira.whamcloud.com/browse/LU-10753" title="sanity test 300c fails with &#39;create 5k files failed&#39; " class="issue-link" data-issue-key="LU-10753"><del>LU-10753</del></a> reported the similar failure, and it looks like this failed on zfs system only. I don't see LFSCK related messages in the logs, I suspect it's because ZFS_IMMUTABLE is set the parent directory during the test, do you have any clue?</p> <blockquote><p>I suspect it's because ZFS_IMMUTABLE is set the parent directory during the test, do you have any clue?</p></blockquote> <p>No, not really.</p> <p>I have started looking into whether one of the other tests was not cleaning up properly, but I think bad cleanup does not explain it. In my testing, the subdirs on one MDT succeed, and the subdirs on the other one fail. So apparently one shard has the bit set, and the other does not.</p> <p>I did notice one other thing.  I altered sanity.sh to replace "run_test 300d ..." with this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>for xx in $(seq 1 30); do run_test 300d "check default stripe under striped directory" done</pre> </div></div> <p>I found that every time I ran auster --only 300, either 300d failed every time (all xx in 1,2.3,...,30), or not at all.</p> <p>I believe that means that the problem is occurring in an earlier subtest. I'll see what more I can learn.</p> <p>Seen on Master  Run : <a href="https://testing.whamcloud.com/test_sets/50b3635a-8810-4f18-b3fe-da0bf0eac4fe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/50b3635a-8810-4f18-b3fe-da0bf0eac4fe</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/75fa672b-9224-4efc-8223-fa6663f6b220" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/75fa672b-9224-4efc-8223-fa6663f6b220</a></p> <p>Side note for Olaf - if submitting patches for debugging/fixing intermittent issues, it is possible to use:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: testlist=sanity env=ONLY=300d,ONLY_REPEAT=N </pre> </div></div> <p>to have the specific test run N times in a loop. The test is considered a failure if any of the iterations fails. You can specify a range of subtests like "<tt>ONLY=290-300</tt>" but "<tt>ONLY_REPEAT=N</tt>" will run each subtest N times in a row before moving to the next subtest.</p> <p>It looks like auster also has a command-line option "<tt>-i N</tt>" to rerun a subtest N times without having to edit .</p> <p>This is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-10753" title="sanity test 300c fails with &#39;create 5k files failed&#39; " class="issue-link" data-issue-key="LU-10753"><del>LU-10753</del></a>, and it's fixed there.</p> Related LU-10753 Development Rank 1|i00wmf: Rank (Obsolete) 9223372036854775807 Severity