https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6528 Lustre <p>(This might be related to <a href="https://jira.whamcloud.com/browse/LU-5730" title="intermittent I/O errors for some directories" class="issue-link" data-issue-key="LU-5730"><del>LU-5730</del></a> but as the symptoms are different, I'm opening a new ticket.)</p> <p>Our users are </p> LU-6528

permission denied over NFS (intermittent

Bug Major Resolved Fixed Lai Siyao Frederik Ferner Mon, 27 Apr 2015 12:34:02 +0000 Wed, 20 Dec 2023 00:51:20 +0000 Sat, 6 Jun 2015 13:53:41 +0000 Lustre 2.5.3 Lustre 2.8.0 0 9 <p>Apologies, this ticket got created early, I'll put my information in this comment as I can't edit the original description...</p> <p>Our users are reporting intermittent permission denied problems when accessing directory on lustre over NFS. They trying to change to a directory returns permission denied and when retrying a few minutes later it is permitted. As far as I can tell nothing modifies the directory in the mean time. There are no problems of this sort when using native Lustre clients.</p> <p>For one user this seems to happen when creating directories over NFS with mode 0700 but not (or much less frequently) when using a more permissive mode like 0755 for the directories. In this case we have a reproducer, which I'll attach. The reproducer creates directories below your current directory and attempts to write one file per directory. It will report permission denied eventually when we run it, depending on the file system we are trying it on, we need to run the reproducer with a different directory count to reproduce the problem.</p> <p>This is a sample output for a failing case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@ws104 frederik-test]$ ~/bin/LU-6528-reproducer1.sh 10000 Traceback (most recent call last): File "/home/bnh65367/bin/LU-6528-reproducer1.sh", line 15, in &lt;module&gt; sub_dir = tempfile.mkdtemp(prefix='sub_', dir=top_dir) File "/usr/lib64/python2.6/tempfile.py", line 318, in mkdtemp _os.mkdir(file, 0700) OSError: [Errno 13] Permission denied: '/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_erQIXk/sub_2aRBS4' [bnh65367@ws104 frederik-test]$ </pre> </div></div> <p>Reports from other users are less clear, so there might be other failing cases.</p> <p>I will try to run the reproducer with debug enabled on the NFS server/lustre client and the MDS server on our test file system....</p> <p>Lai</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>I tested locally with nfsv3 and nfsv4, but couldn't reproduce. Once you have debug logs, please put it somewhere, and then I'll look into them.</p> <p>This is an odd question but are you using LDAP?<br/> If so are you using SSSD?<br/> If so do you have enumerate enabled?<br/> We ran into this exact issue when enumerate was not enabled as it breaks the secondary groups upcall.</p> <p>Malcolm,</p> <p>yes we are using LDAP, SSSD is not installed on our MDS servers (or the NFS servers), it might be installed on the clients but as you mention secondary groups upcall, I assume the MDS is what you were after.</p> <p>Cheers,<br/> Frederik</p> <p>Lai,</p> <p>I'm struggling slightly to gather the logs.</p> <p>Initially I was going to use one of our (more powerful) test clients as NFS server, but failed to reproduce the issue with that machine. On the much less powerful machine I used earlier to reproduce the issue on our test system, I failed to trigger it after enabling the debug daemon. I'm currently trying without debug daemon again, hoping to catch it in time to use lctl dk to gather debug logs which contain the interesting information.</p> <p>One difference between the NFS servers where I was able to reproduce the issue and the one where I couldn't is that the later use o2ib for lnet and the others ethernet only. I don't see how this would matter but thought I'd mention it.</p> <p>Cheers,<br/> Frederik</p> <p>I have just managed to reproduce it, triggering 'lctl dk' on both lustre client/aka NFS server and MDS server directly after the permission denied message, both files are attached, so I'm hoping there might be some information in the dumps.</p> <p>This was the error:</p> <p> File "/home/bnh65367/bin/<a href="https://jira.whamcloud.com/browse/LU-6528" title="permission denied over NFS (intermittent" class="issue-link" data-issue-key="LU-6528"><del>LU-6528</del></a>-reproducer1.sh", line 15, in &lt;module&gt;<br/> sub_dir = tempfile.mkdtemp(prefix='sub_', dir=top_dir)<br/> File "/usr/lib64/python2.6/tempfile.py", line 318, in mkdtemp<br/> _os.mkdir(file, 0700)<br/> OSError: <span class="error">&#91;Errno 13&#93;</span> Permission denied: '/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_39JlTW/sub_ju6qhK'</p> <p>I don't see any failure in both logs, but I see there are create/setattr in MDS logs, while not found in client logs. Actually client log shows it's idle (only ping is sent to MDS).</p> <p>Is there other lustre clients which is creating files simultaneously? And when does client log starts? If it does contain all the logs of this creation, I'm afraid _os.mkdir() failed before it's really done on lustre client, that is, NFS server returned -EACCES before calling into lustre code. If this is true, could you `echo 32767 &gt; /proc/sys/sunrpc/nfsd_debug`, and tried to reproduce, and then upload messages which contains NFSD logs? </p> <p>This is on our test file system, I believe it should be fairly quite at most times, so I don't think there are other clients creating files but I'm not absolutely sure.</p> <p>I've noticed (now) that on the lustre client it seems the lnet/debug_mb was quite small (only 8) during the previous test, not sure if this meant that some of the interesting information has been overwritten, so I've increased this now (to 32) and repeated the test. I've verified that the string "setattr" now exists in the client dump file as well.</p> <p>I'll upload the new dump files and /var/log/message from the lustre client/nfs server after reproducing it again: The error was</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Traceback (most recent call last): File "/home/bnh65367/bin/LU-6528-reproducer1.sh", line 15, in &lt;module&gt; sub_dir = tempfile.mkdtemp(prefix='sub_', dir=top_dir) File "/usr/lib64/python2.6/tempfile.py", line 318, in mkdtemp _os.mkdir(file, 0700) OSError: [Errno 13] Permission denied: '/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_D19Rnh/sub_4GFmt6' </pre> </div></div> <p>Also, we are receiving an increasing number of reports from users who are trying to access files and directories written days ago and are receiving intermittent permission denied (for example when running ls). This is obviously much harder to debug as we are not (yet) able to reproduce at will...</p> <p>The log shows lustre client .lookup_nd failed with -13, and it looks quite possible to be inside kernel function reconnect_path() when it finds a disconnected dentry:<br/> 1. -&gt;get_parent() found parent.<br/> 2. -&gt;get_name() found child dentry name.<br/> 3. -&gt;lookup_one_len() on parent for this name failed with -EACCES, MDS log shows it doesn't have EXEC right to parent.</p> <p>I'm not clear of the cause yet, but first could you verify that the test user has EXEC access right to all directories in "/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_D19Rnh"?</p> <blockquote> <p> could you verify that the test user has EXEC access right to all directories in "/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_D19Rnh"?</p></blockquote> <p>Yes, I've have checked and this is the case.</p> <p>Thanks,<br/> Frederik</p> <p>This is a debug patch which will print mode and id information upon permission deny, could you apply it on MDS and reproduce? BTW it looks like only "trace" is enabled in debug log, could you `lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck trace" during reproduce?</p> <p>The debug log lacks information, and I just uploaded a debug patch <a href="https://jira.whamcloud.com/browse/LU-6528" title="permission denied over NFS (intermittent" class="issue-link" data-issue-key="LU-6528"><del>LU-6528</del></a>_debug.diff, could you reproduce it again?</p> <p>After applying the patch we initially failed to reproduce it until today (possibly related to restarting the MDS?)</p> <p>I have just reproduced it again, though /var/log/messages on the MDS server doesn't quite contain what I was expecting:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 11 11:23:59 cs04r-sc-mds02-03 kernel: format at ldlm_pool.c:628:ldlm_pool_recalc doesn't end in newline May 11 11:43:42 cs04r-sc-mds02-03 kernel: format at ldlm_pool.c:628:ldlm_pool_recalc doesn't end in newline May 11 11:45:49 cs04r-sc-mds02-03 kernel: format at ldlm_pool.c:628:ldlm_pool_recalc doesn't end in newline May 11 11:51:10 cs04r-sc-mds02-03 kernel: Lustre: MGS: non-config logname received: params May 11 11:51:23 cs04r-sc-mds02-03 kernel: Lustre: MGS: non-config logname received: params May 11 11:54:11 cs04r-sc-mds02-03 kernel: Lustre: MGS: non-config logname received: params May 11 11:56:43 cs04r-sc-mds02-03 kernel: Lustre: MGS: non-config logname received: params May 11 12:01:35 cs04r-sc-mds02-03 kernel: format at mdd_permission.c:310:__mdd_permission_internal doesn't end in newline </pre> </div></div> <p>Ah, dmesg contained this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 2723:0:(mdd_permission.c:310:__mdd_permission_internal()) permission denied, mode 45c0, fsuid 1200328, uid 37058 </pre> </div></div> <p>I'll upload the dumps as well, but in case it matters, here is the output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Traceback (most recent call last): File "/home/bnh65367/bin/LU-6528-reproducer1.sh", line 16, in &lt;module&gt; fd, pth = tempfile.mkstemp(prefix='script', dir=sub_dir) File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp return _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 13] Permission denied: '/dls/p45/data/2015/cm12185-2/tmp/frederik-test/top_hT52pM/sub_GYZ7QL/script1IzmRI' </pre> </div></div> <p>And this is what seems to be the relevant bit of syslog entries on the NFS server:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 11 12:01:34 p45-storage kernel: nfsd: MKDIR(3) 40: 97010001 000038a5 0000bb99 00000002 00000674 00000000 sub_GYZ7QL May 11 12:01:34 p45-storage kernel: nfsd: fh_verify(40: 97010001 000038a5 0000bb99 00000002 00000674 00000000) May 11 12:01:34 p45-storage kernel: nfsd: fh_compose(exp 32c:b8e6/4555387 top_hT52pM/sub_GYZ7QL, ino=0) May 11 12:01:34 p45-storage kernel: nfsd_dispatch: vers 3 proc 2 May 11 12:01:34 p45-storage kernel: nfsd: fh_verify(40: 97010001 000038a5 0000bb99 00000002 00004999 00000000) May 11 12:01:35 p45-storage kernel: nfsd_dispatch: vers 3 proc 4 May 11 12:01:35 p45-storage kernel: nfsd: ACCESS(3) 40: 97010001 000038a5 0000bb99 00000002 00004999 00000000 0x1f May 11 12:01:35 p45-storage kernel: nfsd: fh_verify(40: 97010001 000038a5 0000bb99 00000002 00004999 00000000) </pre> </div></div> <p>As always, if you need more information, please ask.</p> <p>Latest dumps, collected with these debug flags on both MDS and client/NFS server:</p> <p>trace super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 2723:0:(mdd_permission.c:310:__mdd_permission_internal()) permission denied, mode 45c0, fsuid 1200328, uid 37058 </pre> </div></div> <p>shows uid doesn't match, which is the cause permission is denied.</p> <p>Could you help verify '37058' is owner of "top_hT52pM"? and which should be current test user too. Besides, could you find the user with uid "1200328"?</p> <p>BTW, what's the status of NFS root_squash? and if it's enabled, what's the uid for root_squash? </p> <p>Yes, 37058 is the owner of top_hT52pM and is the user id that I'm running the tests as:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bnh65367@ws104 frederik-test]$ ls -lnd top_hT52pM drwx--S---+ 8597 37058 111733 266240 May 11 12:01 top_hT52pM [bnh65367@ws104 frederik-test]$ id uid=37058(bnh65367) gid=37058(bnh65367) groups=37058(bnh65367),500(dcs),37900(dls_staff),37907(dls_sysadmin),37910(dls-softinst),37911(dls-i15),109216(nt11000_2) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [bnh65367@ws104 frederik-test]$ </pre> </div></div> <p>uid "1200328" is a random user where I don't recognise the name. Re-running the tests again shows the same mismatch but a different fsuid.</p> <p>NFS root_squash is enabled, root user (uid=0) is mapped to nfsnobody (uid=65534).</p> <p>Could you confirm which way round it is, is fsuid the user (apparently) trying to access the file system or is it the apparent owner of the file?</p> <p>yes, it's fsuid that's used in permission check, and it looks like it's mistakenly set somewhere. I'm still checking related code, will update you later.</p> <p>Is "no_subtree_check" set for the NFS export? It looks like if this feature is set (it's set by default), nfsd_set_fh_dentry() won't set fsuid correctly, but raise capability to allow reconnect_path(), but MDS cleared all FS privileges for lookup (see old_init_ucred()).</p> <p>As the NFS export (on a RHEL6 machine) currently doesn't contain either "no_subtree_check" or "subtree_check" and by my reading of the documentation, it seems no_subtree_check is in force, yes. (also confirmed by running exportfs -v.) </p> <p>I can try (at least on our test system) to re-export with explicit subtree_check and re-test.</p> <p>Cheers,<br/> Frederik</p> <p>This patch will keep FS capability for intent getattr, this is because NFSD will raise FS capability to allow exportfs_decode_fh() to reconnect dentry into dcache, if MDS dropped this capability, reconnect path may fail with -EACCES.</p> <p>I just uploaded a possible patch <a href="https://jira.whamcloud.com/browse/LU-6528" title="permission denied over NFS (intermittent" class="issue-link" data-issue-key="LU-6528"><del>LU-6528</del></a>.diff for this, could you apply it on MDS and test again?</p> <p>I've applied the proposed patch on the MDS and I'm now running the tests.... I guess we'll see tomorrow, though I'll keep running those tests for a t least a day continuously...</p> <p>I've had the reproducer running over night so far it hasn't triggered the issue, so looking good. I'll leave the reproducer running...</p> <p>I noticed the patch had been directly attached to the ticket and not to review.whamcloud.com as I would have expected, does this mean it's not yet a good idea to consider applying it to production systems?</p> <p>Once it works(I'll wait until tomorrow), I'll put it on gerrit later.</p> <p>FWIW, I've been running our reproducer for nearly 24h after applying the patch. So far we have not seen the permission denied issue at all, so it looks good AFAICT.</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14978" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14978</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6528" title="permission denied over NFS (intermittent" class="issue-link" data-issue-key="LU-6528"><del>LU-6528</del></a> mdt: keep FS capability for intent_getattr<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b288a18408a59cf3ac8e3d9bd3ed074547689c57</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14978/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14978/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6528" title="permission denied over NFS (intermittent" class="issue-link" data-issue-key="LU-6528"><del>LU-6528</del></a> mdt: keep FS capability for intent_getattr<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a55aebd70e9073ef1d1bd7de280c8393af0ea30d</p> <p>Landed for 2.8</p> Duplicate Related LU-5730 LU-13791 LU-7630 Development Rank 1|hzxbtb: Rank (Obsolete) 9223372036854775807 Severity