https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4185 Lustre <p>One of our customers is hitting what appears to be <a href="https://jira.whamcloud.com/browse/LU-1101" title="ncorrect permission handling when creating existing directories" class="issue-link" data-issue-key="LU-1101"><del>LU-1101</del></a>. They are also running torque, here is the message from the customer:</p> <blockquote> <p>We use torque as resource manager and job start fails because of mkdir() returns EPERM. The only workaround we have for this is to not set the tmpdir in torque, which is suboptimal. Other applications might suffer the same bug. Quantum espresso is reported to fail as well (in the lustre bug report) and we have users running it.</p></blockquote> <p>The bug was originally reported in bz 23459:<br/> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23459" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=23459</a></p> <p>The patch which appears to be the root cause was added as part of bz 18534:<br/> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=18534" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=18534</a></p> <p>Thanks.</p> LU-4185

Incorrect permission handling when creating existing directories at ICHEC

Bug Major Resolved Fixed Zhenyu Xu Rajeshwaran Ganesan Tue, 29 Oct 2013 22:05:42 +0000 Fri, 3 Nov 2017 15:38:37 +0000 Tue, 6 Dec 2016 03:39:04 +0000 Lustre 2.4.0 Lustre 2.1.4 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.9.0 2 14 <p>Bobijam</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>This seems relate to certain kernel version, I tried linux-2.6.32-358.23.2.el6 with current master, which also has the same ll_lookup_nd code as in 2.1.4 lustre, and I've tried both reproducer c code in bz23459, both test passed without the error, all of the mkdir return the expected errno 17 (EEXIST).</p> <p>What kernel are you using? I'll try it with 2.1.4 lustre code.</p> <p>It looks like it's actually 2.1.5 on the servers, but 2.1.4 on the clients. Here are the kernel versions:<br/> mds: 2.6.32-279.19.1.el6_lustre.es52.x86_64<br/> client: 3.0.74-0.6.6-default</p> <p>client:<br/> lustre-client-modules-kmp-default-2.1.4_3.0.74_0.6.8-5.1</p> <p>mds:<br/> lustre-modules-2.1.5-ddn1.0_2.6.32_279.19.1.el6_lustre.es52.x86_64_ES.x86_64</p> <p>can you collect -1 logs of the client and the mds?</p> <p>I downloaded 3.0.74 kernel and failed build 2.1.4 lustre client upon it. (We don't support 3.0 kernel on 2.1.x lustre yet). Can you teach me how to build it? </p> <p>Also I'd still like to have -1 logs of the client and the mds to do log analysis. </p> <p>Thank you.</p> <p>I think it's actually the SLES11 SP2 kernel:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-3292" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-3292</a></p> <p>We are working on getting the lnet.debug=-1 logs. </p> <p>Hi I'm the ICHEC Systems Administrator reporting the problem. I want to point out the directory torque tries to create doesn't exist:</p> <p>The lustre filesystem on the client is mounted at /ichec/work and the torque tmpdir is set to /ichec/work/scratch/tmp/ which means torque will try to create /ichec/work/scratch/tmp/$JOB_ID which is unique, thus not existing. The C code snipped failing from the version of torque we are using (4.2.5) should be:</p> <p> part = strtok(path, "/");</p> <p> if (part == NULL)</p> { rc = -1; goto done; } <p> <b>(part - 1) = '/'; /</b> leading '/' */</p> <p> while ((part = strtok(NULL, "/")) != NULL)<br/> {<br/> if (mkdir(path, mode) == -1)<br/> {<br/> if (errno != EEXIST)</p> { rc = errno; goto done; } <p> }</p> <p> *(part - 1) = '/';<br/> }</p> <p> /* very last component */</p> <p> if (mkdir(path, mode) == -1)<br/> {<br/> if (errno != EEXIST)</p> { rc = errno; goto done; } <p> }</p> <p>This code tries to create the whole tree first, in case it doesn't exist. In our case only the very last component is the only one not existing.</p> <p>Note that if I tried too the reproduces from LU #1101 and I failed reproducing the bug with it for now. The only way to be 100% sure the bug will show itself is to reboot the node and submit a torque job.</p> <p>Update: I found a way to reproduce the issue using the reproducer code from the original bz entry, and I was wrong about the direcotry triggering the problem: it the already existing one. Sorry for the confusion. I guess I run the command the wrong way in my previous attempts.</p> <p>enrico@r3i1n14:~/mkdirbug/reproduce&gt; ./reproducer /ichec/work/scratch/tmp/<br/> Iteration: 1<br/> mkdir(/ichec/work/scratch, 493) : Permission denied<br/> mkdirtree /ichec/work/scratch/tmp/1804289383 : failed: rc=13 (Permission denied)</p> <p>Python can reproduce this easily as well:</p> <p>enrico@r3i1n14:~/mkdirbug/reproduce&gt; python<br/> Python 2.6.8 (unknown, May 29 2012, 22:30:44) <br/> [GCC 4.3.4 <span class="error">&#91;gcc-4_3-branch revision 152973&#93;</span>] on linux2<br/> Type "help", "copyright", "credits" or "license" for more information.<br/> &gt;&gt;&gt; from os import mkdir<br/> &gt;&gt;&gt; mkdir('/ichec/work/scratch/tmp')<br/> Traceback (most recent call last):<br/> File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br/> OSError: <span class="error">&#91;Errno 13&#93;</span> Permission denied: '/ichec/work/scratch/tmp'<br/> &gt;&gt;&gt; mkdir('/ichec/work/scratch/tmp')<br/> Traceback (most recent call last):<br/> File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br/> OSError: <span class="error">&#91;Errno 13&#93;</span> Permission denied: '/ichec/work/scratch/tmp'<br/> &gt;&gt;&gt; from os import stat<br/> &gt;&gt;&gt; stat('/ichec/work/scratch/tmp')<br/> posix.stat_result(st_mode=17407, st_ino=144122436235821058, st_dev=2903758926L, st_nlink=5, st_uid=0, st_gid=0, st_size=4096, st_atime=1383316840, st_mtime=1383316845, st_ctime=1383316845)<br/> &gt;&gt;&gt; mkdir('/ichec/work/scratch/tmp')<br/> Traceback (most recent call last):<br/> File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br/> OSError: <span class="error">&#91;Errno 17&#93;</span> File exists: '/ichec/work/scratch/tmp'</p> <p>I'm trying to reproduce but can't get it to happen. No matter what I do I only see errno 17, EEXIST. I never get errno 13, EACCESS. I've tried both sles11sp2 clients and centos6.4 clients and see the same behavior.</p> <p>What are the permissions in all the intermediate dirs in your path? I note that all your stats show st_uid=0, st_gid=0. I assume that means you are running and creating all your dirs as root?</p> <p>If you could just be a little more clear of the exact sequence of cmds to reproduce the problem it would be a big help.</p> <p>Sorry, following are more details. If you need something else ask</p> <ol> <li>directories permissions after the example</li> <li>node just booted</li> </ol> <p>enrico@r3i0n0:~&gt; mount | grep 'type lustre'<br/> 10.148.100.60@o2ib:10.148.100.61@o2ib:/home on /ichec/home type lustre (rw,_netdev,localflock)<br/> 10.148.100.60@o2ib:10.148.100.61@o2ib:/work on /ichec/work type lustre (rw,_netdev,localflock)<br/> enrico@r3i0n0:~&gt; # note it happens without localflock as well<br/> enrico@r3i0n0:~&gt; pwd<br/> /ichec/home/staff/enrico<br/> enrico@r3i0n0:~&gt; python<br/> Python 2.6.8 (unknown, May 29 2012, 22:30:44) <br/> [GCC 4.3.4 <span class="error">&#91;gcc-4_3-branch revision 152973&#93;</span>] on linux2<br/> Type "help", "copyright", "credits" or "license" for more information.<br/> &gt;&gt;&gt; from os import mkdir<br/> &gt;&gt;&gt; mkdir('/ichec/work/scratch/tmp')<br/> Traceback (most recent call last):<br/> File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br/> OSError: <span class="error">&#91;Errno 13&#93;</span> Permission denied: '/ichec/work/scratch/tmp'</p> <p>&gt;&gt;&gt; from os import stat<br/> &gt;&gt;&gt; stat('/ichec/work/scratch/tmp')<br/> posix.stat_result(st_mode=17407, st_ino=144122436235821058, st_dev=2903758926L, st_nlink=4, st_uid=0, st_gid=0, st_size=4096, st_atime=1383319699, st_mtime=1383320272, st_ctime=1383320272)<br/> &gt;&gt;&gt; mkdir('/ichec/work/scratch/tmp')<br/> Traceback (most recent call last):<br/> File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br/> OSError: <span class="error">&#91;Errno 17&#93;</span> File exists: '/ichec/work/scratch/tmp'</p> <p>This is the simplest way to reproduce the issue.</p> <p>Permission of the tmpdir<br/> enrico@r3i0n0:~&gt; ls -ld /ichec/work/scratch/tmp<br/> drwxrwxrwt 4 root root 4096 Nov 1 15:37 /ichec/work/scratch/tmp<br/> enrico@r3i0n0:~&gt; ls -ld /ichec/work/scratch/<br/> drwxr-xr-x 3 root root 4096 Oct 29 14:21 /ichec/work/scratch/<br/> enrico@r3i0n0:~&gt; ls -ld /ichec/work/<br/> drwxr-xr-x 10 root root 4096 Oct 24 15:46 /ichec/work/<br/> enrico@r3i0n0:~&gt; ls -ld /ichec/<br/> drwxr-xr-x 4 root root 61 Sep 30 17:14 /ichec/<br/> enrico@r3i0n0:~&gt; ls -ld /<br/> drwxr-xr-x 24 root root 4096 Oct 22 11:20 /</p> <p>Using the C program reproducer (updated version, more readable) from the original BZ:</p> <p>enrico@r3i0n1:~/mkdirbug/reproduce&gt; pwd<br/> /ichec/home/staff/enrico/mkdirbug/reproduce<br/> enrico@r3i0n1:~/mkdirbug/reproduce&gt; ./reproducer /ichec/work/scratch/tmp/<br/> Iteration: 1<br/> mkdir(/ichec/work/scratch, 493) : Permission denied<br/> mkdirtree /ichec/work/scratch/tmp/1804289383 : failed: rc=13 (Permission denied) <br/> Iteration: 2<br/> doing stat before creating directory</p> <p>ERROR: inconsistency detected: previous rc: 13 vs current rc: 0</p> <p>The error doesn't happen if you are executing the program from the tmpdir (obviously):<br/> enrico@r3i0n2:/ichec/work/scratch/tmp&gt; pwd<br/> /ichec/work/scratch/tmp<br/> enrico@r3i0n2:/ichec/work/scratch/tmp&gt; ./reproducer /ichec/work/scratch/tmp/<br/> Iteration: 1<br/> Iteration: 2<br/> doing stat before creating directory<br/> Iteration: 3<br/> Iteration: 4<br/> doing stat before creating directory</p> <p>Thank you<br/> Enrico</p> <p>Could you please attach the source of your C program reproducer? I might find that easier to work with and a stronger guarantee that I'm trying the exact same thing as you.</p> <p>I note your mount flags show _netdev. I'm not familiar with that option. What is it? Doubt it's related but I want to pin down all the diffs between you and me.</p> <p>This is the source code for the reproducer</p> <p>Thank you for your reproducer. Seems to run fine for me, no error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ./reproducer /mnt/lustre/tmp Iteration: 1 Iteration: 2 doing stat before creating directory Iteration: 3 Iteration: 4 doing stat before creating directory Iteration: 5 Iteration: 6 doing stat before creating directory Iteration: 7 Iteration: 8 doing stat before creating directory Iteration: 9 </pre> </div></div> <p>I am running as root. I am running on a current sles11sp2 client. Will try rolling back to a 2.1 lustre version client to see if the problem is version specific.</p> <p>Here are the debug=-1 logs that requested previously:<br/> <a href="http://ddntsr.com/ftp/2013-11-01-SR-27367_r3i1n15.dk-log.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2013-11-01-SR-27367_r3i1n15.dk-log.tar.gz</a><br/> <a href="http://ddntsr.com/ftp/2013-11-01-SR-27367_mkd2.dk-log.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2013-11-01-SR-27367_mkd2.dk-log.tar.gz</a></p> <p>Is there any reason to think that it's not the same issue identified by Hongchao in bz 23459? It appears that the potentially incorrect namedata lookup shortcut is still in llite/namei.c.</p> <p>I think that patch can cure this issue. Just want to understand further.</p> <p>Ok cool, just want to make sure we are all on the same page. I was looking at the client dk and I think that code path is hit here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000080:00002000:0.0:1383306592.567518:0:4982:0:(dcache.c:116:ll_dcompare()) found name tmp(ffff880840b64500) flags 0x20e010 refc 1 00000080:00000001:0.0:1383306592.567519:0:4982:0:(dcache.c:123:ll_dcompare()) <span class="code-object">Process</span> leaving (rc=1 : 1 : 1) 00000080:00000001:0.0:1383306592.567520:0:4982:0:(namei.c:571:ll_lookup_nd()) <span class="code-object">Process</span> entered 00000080:00000001:0.0:1383306592.567521:0:4982:0:(dcache.c:199:ll_set_dd()) <span class="code-object">Process</span> entered 00000080:00002000:0.0:1383306592.567521:0:4982:0:(dcache.c:204:ll_set_dd()) ldd on dentry tmp (ffff88083d4152c0) parent ffff88084c088180 inode (<span class="code-keyword">null</span>) refc 1 00000080:00000010:0.0:1383306592.567522:0:4982:0:(dcache.c:209:ll_set_dd()) kmalloced <span class="code-quote">'lld'</span>: 104 at ffff88084c721dc0. 00000080:00000001:0.0:1383306592.567523:0:4982:0:(dcache.c:222:ll_set_dd()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) 00000080:00000001:0.0:1383306592.567523:0:4982:0:(namei.c:593:ll_lookup_nd()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) </pre> </div></div> <p>I can't tell if it's returning EPERM because of that though.</p> <p>I think it could relates to the discretionary access control of the fs, if the running task does not has CAP_DAC_OVERRIDE capability (i.e. override DAC access), this issue could happen. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> generic_permission(struct inode *inode, <span class="code-object">int</span> mask, <span class="code-object">int</span> (*check_acl)(struct inode *inode, <span class="code-object">int</span> mask)) { <span class="code-object">int</span> ret; /* * Do the basic POSIX ACL permission checks. */ ret = acl_permission_check(inode, mask, check_acl); <span class="code-keyword">if</span> (ret != -EACCES) <span class="code-keyword">return</span> ret; /* * Read/write DACs are always overridable. * Executable DACs are overridable <span class="code-keyword">if</span> at least one exec bit is set. */ <span class="code-keyword">if</span> (!(mask &amp; MAY_EXEC) || execute_ok(inode)) <span class="code-keyword">if</span> (capable(CAP_DAC_OVERRIDE)) <span class="code-keyword">return</span> 0; /* * Searching includes executable on directories, <span class="code-keyword">else</span> just read. */ mask &amp;= MAY_READ | MAY_WRITE | MAY_EXEC; <span class="code-keyword">if</span> (mask == MAY_READ || (S_ISDIR(inode-&gt;i_mode) &amp;&amp; !(mask &amp; MAY_WRITE))) <span class="code-keyword">if</span> (capable(CAP_DAC_READ_SEARCH)) <span class="code-keyword">return</span> 0; <span class="code-keyword">return</span> -EACCES; } </pre> </div></div> <p>with patch in bz 23459, the -EEXIST would be detected before this permission checking.</p> <p>Hi Oleg,</p> <p>Could you take a look, I am still confused what the root cause of the -EACCES (I think its from the checking of the parent of the last dir dentry, but I don't know what the relationship it has with the existing dir, if we are to create non-existing dir, the same permission checking path still will go through).</p> <p>I think the issue is that the application code (Torque) uses mkdir to check for existence, even if it does not have permission to create the directory. Lustre believes that the directory doesn't exist, even though it does, so it does a permissions check to see if Torque can create the directory, which fails. These applications require that mkdir return EEXIST before checking permissions. </p> <p>Bob, I think you need to run the reproducer as a non-privileged user in order for it to work. The directory should exist, but the user should not have permission to create it. The expected error is EEXIST, but because of the bug you will get EACCESS. Here are the instructions for how to use the reproducer from the bugzilla ticket linked earlier:</p> <blockquote> <p>Short instruction:<br/> to reproduce the bug:</p> <ol> <li>as root<br/> mkdir &lt;lustre_root&gt;/test -m755<br/> chown root:root &lt;lustre_root&gt;/test<br/> mkdir &lt;lustre_root&gt;/scratch -m1777</li> <li>as user (su - someuser)<br/> gcc lustre-test.c -o test<br/> ./test &lt;lustre_root&gt;/scratch/</li> </ol> </blockquote> <p>Any updates? Should we try the patch from bz 23459?</p> <p>yes, I think you can try that patch. But I'm not sure to include it in the source tree, since it would kill a little open &amp; create performance, while just to return different error code for creating existing directory scenario.</p> <p>Let's try it first, to see whether it can handle the issue which torque meets.</p> <p>Ok, sounds good. I wonder what the performance hit is, is there any automated mdtest'ing done? I'll let you know how it goes. </p> <p>Just a quick note - We're seeing this as well at Cray, both on 2.4.1 on SLES and master on CentOS.</p> <p>I've got a simpler reproducer as well.</p> <p>As root:<br/> mkdir test<br/> cd test<br/> mkdir test2<br/> su <span class="error">&#91;another_user&#93;</span><br/> mkdir test2<br/> ^-- This hits EPERM, when it should be EEXIST.</p> <p>If you then do ls -l and try again, you get EEXIST as expected.<br/> (By the way, this hints at a work around that our application code is planning to try - stat the directory to be 'created' first.)</p> <p>I'll test the suggested patch against master.</p> <p>Patrick,<br/> Using your simple command line reproducer I can see the problem with a 2.4.1 sles11sp2 client, but not with a current master sles11sp2 client. I have to assume it was fixed somewhere along the way since 2.4.1</p> <p>Bob,</p> <p>Interesting! The version of master I tested against was slightly out of date. I'll rebuild and test again with the latest.</p> <p>For what it's worth, my testing of master was on CentOS.</p> <p>Also, I forgot to specify, but the SLES I used with 2.4.1 was SLES11SP2 (as in your test).</p> <p>Bob,</p> <p>I just tested with a fresh pull of master on CentOS and I'm still seeing the issue as described.<br/> Perhaps you accidentally stated the directory when you were testing on master? Or there could be some interaction between master and SLES11SP2.</p> <p>The patch from bz 23459 does fix my reproducer. Still, as noted about, there's a possible performance impact.</p> <p>Patrick,</p> <p>You are entirely correct. I must have mistyped something during my master test. Reran the experiment. Now seeing the problem reproduce with both 2.4.1 and master sles11sp2 clients. EPERM reported instead of EEXIST in both cases.</p> <p>Have also confirmed the same failure can be seen with current master on sles11sp3 clients as well. EPERM instead of EEXIST reported there too. I believe this is good evidence that the problem really is in lustre client code, is still there in current master, and the specific distro or kernel version doesn't matter.</p> <p>The patch is tracking at <a href="http://review.whamcloud.com/8257" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8257</a>, but I'm not sure to include it in the source tree, since it would kill a little open &amp; create performance, while just to return a different error code for creating existing directory scenario.</p> <p>Hi Zhenyu,</p> <p>mkdir() of an existing directory return EEXIST on common filesystems, whatever the permission is. I have no idea if there is a standard mandating that or it is just common usage, but if lustre is going to make something different it will break a lot of workflows, for the good of performances. A lot of applications will need to add a stat before the creation useful only when lustre is the underlaying file system, leading to a performance hit anyway.</p> <p>From the performance PoV this is indeed not the ideal solution, but if you are trying to make a directory I think you really want to know first if it exists already, before knowing if you have the right permission to create it or not.</p> <p>Kind regards<br/> Enrico</p> <p>One additional thing about this that bothers me:<br/> It seems very undesirable that the value returned for an operation in a directory would change depending on whether or not you've read the directory contents recently.</p> <p>It would be nice to have this addressed:<br/> Is this issue that Intel expects to fix, either with the patch already suggested, or some other way, or is this a minor divergence from POSIX that Intel's planning to keep in order to achieve better performance?</p> <p>Yuck - I recommend abandoning this change. I just took a closer look at the performance impact, and it's <b>bad</b>.</p> <p>Below are single node mdtest results. The directory creation is what's interesting - As that is create without open, which is the case we're optimizing away (and the patch would undo that optimization). File creates are create+open, and as expected, the performance doesn't change.<br/> I don't think sacrificing 30-50% of our create performance is worth fixing a quite obscure bit of aberrant behavior.</p> <p>Without the fix in place:<br/> SUMMARY: (of 10 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 1799.957 1636.083 1736.426 55.974<br/> Perf_Data mdtest 1 Dir_create 1736.426 ops/sec<br/> File creation : 1253.821 985.786 1130.595 65.472<br/> Perf_Data mdtest 1 File_create 1130.595 ops/sec<br/> Tree creation : 1968.233 1540.891 1771.710 126.015<br/> Perf_Data mdtest 1 Tree_create 1771.710 ops/sec</p> <p>With the fix in place, same node, same system:<br/> SUMMARY: (of 10 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 1252.500 1027.566 1196.570 61.338<br/> Perf_Data mdtest 1 Dir_create 1196.570 ops/sec<br/> File creation : 1182.596 1070.822 1118.642 35.296<br/> Tree creation : 1009.946 864.270 919.236 43.143<br/> Perf_Data mdtest 1 Tree_create 919.236 ops/sec</p> <p>Like I said, 30-50% drop in performance. Yuck.</p> <p>Hi,</p> <p>We are a DDN customer, who recently hit this issue with (I believe) 2 packages.</p> <p>Firstly, running MATLAB 2014b and later we receive 'Permission Denied' errors when using MATLAB's built-in mkdir function to create a nested directory structure on our lustre fs. Using MATLAB's system command to call out to shell 'mkdir -p' always works without issue. Mathworks investigated for us, but indicated they can't support lustre. MATLAB's in-built mkdir works prior to 2014a, so (after coming across the 2nd example) I think it's related to this <a href="https://jira.whamcloud.com/browse/LU-4185" title="Incorrect permission handling when creating existing directories at ICHEC" class="issue-link" data-issue-key="LU-4185"><del>LU-4185</del></a>, I've contacted them again with some more info - to see if EPERM instead of EEXIST makes sense. (I'm guessing it does since MATLAB is reporting a Permission Denied error for the failing mkdir calls).</p> <p>The second occurrence thing that led me here was failing output directory creation for the cufflinks bioinformatics tool on the lustre fs. Code is available, and I was able to take their implementation of mkpath (for recursive directory creation) and see that it failed trying to run mkdir on an existing path - receiving EPERM not EEXIST. The relevant code is here:</p> <p><a href="https://github.com/cole-trapnell-lab/cufflinks/blob/master/src/common.cpp" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/cole-trapnell-lab/cufflinks/blob/master/src/common.cpp</a></p> <p>Search mkpath, currently line 262.</p> <p>I can make a patch to cufflinks to run a stat before attempting a mkdir, rather than just relying on mkdir returning EEXIST. However, it would be nice to know if there is ongoing work to try and address this.</p> <p>Thanks.</p> <p>Issue ref for the cufflinks package we have the problem with: <a href="https://github.com/cole-trapnell-lab/cufflinks/issues/53" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/cole-trapnell-lab/cufflinks/issues/53</a></p> <p>updated the <a href="http://review.whamcloud.com/8257" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8257</a> to address this issue while not impacting the performance.</p> <p>Though the updated patch can solve this issue, it raises another issue which is that it will ignore ACL permission failure opon directories. </p> <p>Updated the patch which adds a client proc entry to control whether bypass this optimization, by default, this optimization is turned on. And with special applications which only check EEXIST return value of mkdir, we can turn it off by "lctl set_param" command.</p> <p>$ lctl set_param llite.&lt;client_uuid&gt;.create_no_open_optimization=0</p> <p>It would be worthwhile to check if patch <a href="http://review.whamcloud.com/20354" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20354</a> "<a href="https://jira.whamcloud.com/browse/LU-8019" title="Openlock breakage" class="issue-link" data-issue-key="LU-8019"><del>LU-8019</del></a> llite: Restore proper opencache operations" fixes this problem.</p> <p>I tried to read the comments to get a better handle on this issue.</p> <p>The reproducer from Cray is basically:<br/> in a place that user1 owns and user2 does not have write permissions: create a directory as user1 ; then with cold caches try to create same directory as user2 - that fails with EPERM because we do not know that directory already exists. (I imagine if user1 created a non-directory then we get the same issue too).</p> <p>This seems to be a pretty narrow case, though.<br/> So the other EPERM cases - do they happen when all processes have write permissions in the dir where we are trying to create? Was that source of the failures distilled to a testcase oreven better - understood yet?</p> <p>I guess if somebody confirm that the case is distilled to the apps that basically get a work path in the form of /mnt/lustre/homes/user/somedir/app_folder where user only has write permissions from user onward, but the app does the equivalent of:<br/> mkdir /mnt ; mkdir /mnt/lustre ; mkdir /mnt/lustre/homes ; mkdir /mnt/lustre/homes/user ; .... and stops on !-EEXIST<br/> Then we can probably add a lot more cheaper workaround in the ll_lookup_nd in the form of:<br/> if this CREATE with but NOT OPEN ( == good proxy this is mkdir, mknod and so on)<br/> AND if this user does NOT have write permissions in the parent, THEN go on with the lookup.</p> <p>This way we do not penalize all the important fast cases being:<br/> 1. User has write permissions - great, we'll do create and if it fails - that would be on the server with the correct error.<br/> 2. User does not have write permissions, but the directory exists - that's ok too, we go to the server anyway, get our dentry and never get into create in the end.</p> <p>The <br/> 3. User does not have permissions and directory does not exist - this will get slower, we jump from 0 to 1 RPC in this case. But I assume this is unimportant enough that nobody would notice or care?</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21169" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21169</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4185" title="Incorrect permission handling when creating existing directories at ICHEC" class="issue-link" data-issue-key="LU-4185"><del>LU-4185</del></a> llite: Revise create with no open optimization.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 571495cff60cab20d687402524eb051474b3e32b</p> <p>Oleg - I think you've captured the problem as I understand end users are reporting it (as well as the Cray reproducer).</p> <p>One question - Which Gerrit link are we using? Bobi Jam has updated to a patch like yours as well.</p> <p>Bobi's patch has a test which is better than mine that does not.</p> <p>Patrick: It would be great if you can verify the current patch (Either one) is ok in majority of operations like mdtest and such.</p> <p>BTW, interesting thing, apparently NFS has the same problem as Lustre and I have not heard any complaints.<br/> Time to fill some more bugs, I guess.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[green@fedora1 crash]$ mkdir aaa mkdir: cannot create directory <span class="code-quote">'aaa'</span>: Permission denied [green@fedora1 crash]$ mkdir lost+found mkdir: cannot create directory <span class="code-quote">'lost+found'</span>: Permission denied [green@fedora1 crash]$ ls -ld lost+found drwx------ 2 root root 16384 May 25 2013 lost+found [green@fedora1 crash]$ mkdir lost+found mkdir: cannot create directory <span class="code-quote">'lost+found'</span>: File exists </pre> </div></div> <p>After some discussion with the kernel guys, it looks like EEXIST is not a requirement, so the applications that depend on it are broken.</p> <p>See <a href="http://marc.info/?l=linux-kernel&amp;m=146803277310677&amp;w=2" class="external-link" target="_blank" rel="nofollow noopener">http://marc.info/?l=linux-kernel&amp;m=146803277310677&amp;w=2</a> and <a href="http://marc.info/?l=linux-nfs&amp;m=146803381011004&amp;w=2" class="external-link" target="_blank" rel="nofollow noopener">http://marc.info/?l=linux-nfs&amp;m=146803381011004&amp;w=2</a></p> <p>So I guess it's a good idea to file bugreports with those apps even if we have this also adjusted in Lustre.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/8257/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8257/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4185" title="Incorrect permission handling when creating existing directories at ICHEC" class="issue-link" data-issue-key="LU-4185"><del>LU-4185</del></a> llite: Revise create with no open optimization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a2d5b2e83c0a512a3ea59698e8481621ab5856c2</p> <p>In discussion with Bobijam, this issue is resolved with the landing of <a href="https://review.whamcloud.com/#/c/8257/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/8257/</a></p> Duplicate LU-1101 Related LU-8019 Development End date Fri, 3 Jun 2016 22:05:42 +0000 Rank 1|hzw79r: Rank (Obsolete) 11322 Severity Start date Tue, 29 Oct 2013 22:05:42 +0000