https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7093 Lustre <p>In a a set of directories on ldne, /p/ldne/dinatale/count4_index*, Joe is able to create a file, but not a directory. Joe owns the directories and the mode is set to 700.</p> <p>In other directories, for example /p/ldne/dinatale/count3_index*, Joe is able to create both files and directories successfully. They have the same ownership and mode.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zwicky96@dinatale:pwd /p/ldne/dinatale/count4_index0 zwicky96@dinatale:ls -al total 54 drwx------ 2 dinatale lcstaff 43008 Sep 1 14:13 . dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 .. zwicky96@dinatale:date &gt; something zwicky96@dinatale:ls -l something -rw------- 1 dinatale dinatale 29 Sep 1 15:45 something zwicky96@dinatale:rm something zwicky96@dinatale:ls -al total 54 drwx------ 2 dinatale lcstaff 43008 Sep 1 15:46 . dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 .. zwicky96@dinatale:mkdir something mkdir: cannot create directory `something': Operation not permitted zwicky96@dinatale:ls -al total 54 drwx------ 2 dinatale lcstaff 43008 Sep 1 15:46 . dr-xr-xr-x 18 dinatale lcstaff 11776 Aug 31 11:32 .. zwicky96@dinatale:rpm -q lustre lustre-2.7.57-2.6.32_504.16.2.1chaos.ch5.3.x86_64_gea38322.x86_64 zwicky96@dinatale:distro_version toss 2.3-6.ch5.3 zwicky96@dinatale:lfs getdirstripe . . lmv_stripe_count: 4 lmv_stripe_offset: 0 mdtidx FID[seq:oid:ver] 0 [0x200000453:0x1ff:0x0] 1 [0x280000411:0x1ff:0x0] 2 [0x24000041c:0x1fe:0x0] 3 [0x2c000040b:0x1fd:0x0] zwicky96@dinatale:pwd /p/ldne/dinatale/count4_index0 zwicky96@dinatale:ls -ld ../count4_index* drwx------ 2 dinatale lcstaff 43008 Sep 1 15:46 ../count4_index0 drwx------ 2 dinatale lcstaff 43008 Sep 1 14:00 ../count4_index1 drwx------ 2 dinatale lcstaff 43008 Sep 1 14:00 ../count4_index2 drwx------ 2 dinatale lcstaff 43008 Sep 1 14:00 ../count4_index3 zwicky96@dinatale:ls -ld ../count3_index* drwx------ 3 dinatale lcstaff 32256 Sep 1 14:06 ../count3_index0 drwx------ 3 dinatale lcstaff 32256 Sep 1 14:06 ../count3_index1 drwx------ 3 dinatale lcstaff 32256 Sep 1 14:06 ../count3_index2 drwx------ 3 dinatale lcstaff 32256 Sep 1 14:06 ../count3_index3 </pre> </div></div> <p>We are running 4 MDSs each w/ 1 MDT, 2 OSSs each w/ 1 OST, and clients are running the same lustre version as the servers. Backend is zfs.</p> lustre-2.7.57-2.6.32_504.16.2.1chaos.ch5.3.x86_64_gea38322.x86_64<br/> Redhat 2.6.32-504.16.2.1chaos.ch5.3.x86_64 LU-7093

mkdir: cannot create directory: Operation not permitted

Bug Critical Resolved Fixed Di Wang Giuseppe Di Natale dne llnl zfs Wed, 2 Sep 2015 20:52:45 +0000 Tue, 20 Sep 2016 02:57:57 +0000 Thu, 17 Sep 2015 16:53:20 +0000 Lustre 2.8.0 Lustre 2.8.0 0 10 <p>Linked <a href="https://jira.whamcloud.com/browse/LU-7013" title="replay-single test_110c: FAIL: create striped dir failed " class="issue-link" data-issue-key="LU-7013"><del>LU-7013</del></a>. That issue may or may not be related, but it is a test involving a mkdir.</p> <p>Attached debug log output from the MDS nodes, dumped after "mkdir something" which failed as Joe described.</p> <p>See call to mdt_md_create() on line 29 of dne-count4.debug.zwicky-ldne-mds1.after</p> <p>We can gather other information as required - just let us know.</p> <p>Some FIDs in case they help with interpreting logs:</p> <p>/p/ldne/dinatale/count4_index0<br/> lmv_stripe_count: 4 lmv_stripe_offset: 0<br/> mdtidx FID<span class="error">&#91;seq:oid:ver&#93;</span><br/> 0 <span class="error">&#91;0x200000453:0x1ff:0x0&#93;</span><br/> 1 <span class="error">&#91;0x280000411:0x1ff:0x0&#93;</span><br/> 2 <span class="error">&#91;0x24000041c:0x1fe:0x0&#93;</span><br/> 3 <span class="error">&#91;0x2c000040b:0x1fd:0x0&#93;</span></p> <p>/p/ldne: <span class="error">&#91;0x200000007:0x1:0x0&#93;</span> <br/> /p/ldne/dinatale: <span class="error">&#91;0x200000404:0x1:0x0&#93;</span> <br/> /p/ldne/dinatale/count4_index0: <span class="error">&#91;0x200000457:0x1208:0x0&#93;</span> </p> <p>zwicky96@dinatale:sudo lfs getdirstripe /p/ldne/dinatale <br/> /p/ldne/dinatale <br/> lmv_stripe_count: 0 lmv_stripe_offset: 0</p> <p>Hmm, did you set default directory stripeEA on count4_index0 ? Please check. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs getdirstripe -D count4_index0 </pre> </div></div> <p> According to the debug log, it seems it try to create a remote directory under the count4_index0</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000002:9.0:1441148647.363955:0:21353:0:(mdt_reint.c:308:mdt_md_create()) @@@ Create (something-&gt;[0x200000479:0x583a:0x0]) in [0x2c000040b:0x1fd:0x0] req@ffff880ea0a31cc0 x1510173626909920/t0(0) o36-&gt;548c6f4a-55be-90f6-da7e-6f3b85da262e@192.168.120.96@o2ib6:178/0 lens 616/568 e 0 to 0 dl 1441148708 ref 1 fl Interpret:/0/0 rc 0/0 </pre> </div></div> <p>Something seems on MDT0, but its name entry being assigned to MDT3. If that is what you want, then you need do this on all of your MDS</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode tests]# ../utils/lctl set_param mdt.*.enable_remote_dir_gid=-1 </pre> </div></div> <p>This will allow all of non-sysadmin users to create cross-MDT objects (striped, remote dir etc) on all of MDTs. </p> <p>btw: for normal user to test striped directory, you better include the fix <a href="http://review.whamcloud.com/13990" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13990</a>, which seems not in 2.7.57 yet. So either upgrade to 2.7.58, or apply the fix in your build. thanks.</p> <p>Here is the output from lfs regarding default stripe for count4_index0.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zwicky99@dinatale:lfs getdirstripe -D count4_index0 (Default)count4_index0 lmv_stripe_count: 4 lmv_stripe_offset: 0 </pre> </div></div> <p>Here is the same lfs output for count3_index0 for comparison.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zwicky99@dinatale:lfs getdirstripe -D count3_index0 (Default)count3_index0 lmv_stripe_count: 3 lmv_stripe_offset: 0 </pre> </div></div> <p>Again, mkdir fails in count4_index0.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zwicky99@dinatale:pwd /p/ldne/dinatale/count4_index0 zwicky99@dinatale:mkdir something mkdir: cannot create directory `something': Operation not permitted </pre> </div></div> <p>But, in count3_index0 it is successful.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zwicky99@dinatale:pwd /p/ldne/dinatale/count3_index0 zwicky99@dinatale:mkdir something zwicky99@dinatale:ll total 12 drwx------ 2 dinatale dinatale 11776 Sep 3 10:05 something </pre> </div></div> <p>enable_remote_dir_gid was never set on any of the MDTs and directory creation seems to work in other directories. User dinatale is a non-sysadmin user already. The build we are using to do our testing also appears to include the fix you linked according to our git log.</p> <p>The reason it did not fail on count3_index0 is that its name and directory itself are happen to be assigned both MDT0, so it is not a remote directory, then this remote_dir_gid check will not operate on this operation. But for count4_index0, the name and directory are being assigned to different MDT0, and then this remote_dir_gid check needs to step in.</p> <p>Note for striped directory, when creating dir or files under it, </p> <ul> <li>their name entries will be assigned according its name hash value (FNV-1a hash algorithm by default), stripe_index = hash_value % stripe_count;</li> <li>If there are not default stripeEA, then the directory itself will be assigned to the same MDT as its name entry, otherwise it will be assigned to the MDT where default stripeEA indicate, which means creating the directory (mkdir something) under a striped directory, which also have default stripeEA (like your case), might create a remote directory.</li> </ul> <p>So without enable_remote_dir_gid , you will also meet this problem under count3_index0, try to create some directories with other name, like mkdir something1. </p> <p>Di, thank you for the explanation. It makes sense now.</p> <p>I still find it odd that a user may be allowed to make a striped directory even though they are not allowed to actually create remote directories. Wouldn't it be better behavior to not permit users to make striped directories if remote directories is disabled? Likewise, if remote directories are enabled and a striped directory exists in the filesystem, then remote directories cannot be disabled?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>I still find it odd that a user may be allowed to make a striped directory even though they are not allowed to actually create remote directories. </pre> </div></div> <p>hmm, this maybe a problem. I will work on a fix, thanks for finding this. </p> <p>I think the check needs to be that if a striped directory exists, the users can create subdirectories under the striped directory regardless of the extra /proc setting. Also, setting the default directory striping itself needs to check the extra /proc setting, but once it is set then the regular user can create subdirectories according to this policy. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16286" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16286</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7093" title="mkdir: cannot create directory: Operation not permitted" class="issue-link" data-issue-key="LU-7093"><del>LU-7093</del></a> mdt: Remote operation permission check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 482a09045f2d8be0681146c01c7c7a248ed0f559</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16286/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16286/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7093" title="mkdir: cannot create directory: Operation not permitted" class="issue-link" data-issue-key="LU-7093"><del>LU-7093</del></a> mdt: Remote operation permission check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 703195acc2157969e66cb07946522b918dfd7dea</p> <p>Landed for 2.8.</p> Related LU-7013 LU-6341 Development Rank 1|hzxm87: Rank (Obsolete) 9223372036854775807 Severity