https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1101 Lustre <p>Lustre seems to handle permissions on mkdir incorectlly in some cases. This issue makes it hard (or impossible) to use Torque scheduler directly on top of a Lustre filesystem. This is in fact copy of bugzilla bug #23459, which was reported by us some time ago for 1.8 branch, however it looks like the bug is still there even in 2.1. All the symptoms described in bugzilla are identical and the reproducer code provided by Lukasz Flis still works for this issue.</p> Lustre 2.1 on clients and servers, Scientific Linux 5 LU-1101

ncorrect permission handling when creating existing directories

Bug Critical Resolved Duplicate WC Triage Marek Magrys Tue, 14 Feb 2012 11:23:24 +0000 Wed, 6 Nov 2013 17:20:58 +0000 Wed, 6 Nov 2013 17:20:58 +0000 Lustre 2.1.0 1 4 <p>To clarify:<br/> The problem occurs when Torque (pbs_mom) has the $tmpdir config (/var/torque/mom_priv/config) var set to Lustre filesystem (in our case $tmpdir /mnt/lustre/scratch/jobs). We occasionally get errors like:</p> <p>Feb 14 13:56:23 n6-4-16 pbs_mom: LOG_ERROR::Permission denied (13) in TMakeTmpDir, Unable to make job transient directory: /mnt/lustre/scratch/jobs/18555647.batch.grid.cyf-kr.edu.pl<br/> Feb 14 14:37:35 n6-4-16 pbs_mom: LOG_ERROR::Permission denied (13) in TMakeTmpDir, Unable to make job transient directory: /mnt/lustre/scratch/jobs/18557701.batch.grid.cyf-kr.edu.pl<br/> Feb 14 14:38:17 n6-4-16 pbs_mom: LOG_ERROR::Permission denied (13) in TMakeTmpDir, Unable to make job transient directory: /mnt/lustre/scratch/jobs/18557716.batch.grid.cyf-kr.edu.pl<br/> Feb 14 14:50:46 n6-4-16 pbs_mom: LOG_ERROR::Permission denied (13) in TMakeTmpDir, Unable to make job transient directory: /mnt/lustre/scratch/jobs/18559037.batch.grid.cyf-kr.edu.pl<br/> Feb 14 15:01:44 n6-4-16 pbs_mom: LOG_ERROR::Permission denied (13) in TMakeTmpDir, Unable to make job transient directory: /mnt/lustre/scratch/jobs/18559949.batch.grid.cyf-kr.edu.pl</p> <p>An example output of the reproducer:</p> <p><span class="error">&#91;b14flis@n6-4-16 repro&#93;</span>$ ./a.out /mnt/lustre/scratch/jobs/<br/> Iteration: 1<br/> Creating directory: /mnt/lustre/scratch/jobs/1804289383<br/> mkdir(/mnt,mode) errno: 17<br/> mkdir(/mnt/lustre,mode) errno: 17<br/> mkdir(/mnt/lustre/scratch,mode) errno: 17<br/> mkdir(/mnt/lustre/scratch/jobs,mode) errno: 13<br/> mkdirtree: failed: rc=13 <br/> sleeping for 2 seconds</p> <p>Iteration: 2<br/> doing stat before creating directory<br/> Creating directory: /mnt/lustre/scratch/jobs/846930886<br/> mkdir(/mnt,mode) errno: 17<br/> mkdir(/mnt/lustre,mode) errno: 17<br/> mkdir(/mnt/lustre/scratch,mode) errno: 17<br/> mkdir(/mnt/lustre/scratch/jobs,mode) errno: 17<br/> mkdirtree: successful: rc=0 </p> <p>ERROR: inconsistency detected: previous rc: 13 vs current rc: 0</p> <p>Hi,</p> <p>One of our users using Quantum Espresso application hit the bug today.<br/> user has set outdir variable to her directory on lustre filesystem.</p> <p> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br/> task # 39<br/> from parallel_mkdir : error # 1<br/> /mnt/lustre/scratch/people/xuser/ non existent or non writable<br/> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</p> <p> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br/> task # 14<br/> from parallel_mkdir : error # 1<br/> /mnt/lustre/scratch/people/xuser/ non existent or non writable<br/> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br/> task # 44<br/> from parallel_mkdir : error # 1<br/> /mnt/lustre/scratch/people/xuser/ non existent or non writable<br/> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</p> <p>strace dump shown that mkdir result was:<br/> wrapper.6736.3477:mkdir("/mnt/lustre/scratch/people/xuser/", 0777) = -1 EACCES (Permission denied)</p> <p>After doing stat on the directory before invoking application problem dissapeared:<br/> wrapper.23170.3574:mkdir("/mnt/lustre/scratch/people/xuser/", 0777) = -1 EEXIST (File exists)</p> <p>Cheers,</p> <p>To answer the last question in the bugzilla report, the code that causes this bug was added here as an MDS optimization:<br/> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=18534" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=18534</a></p> <p>Hi, </p> <p>Just to update:</p> <p>We have tested and it appeared this is not a problem in 2.2.0 clients.<br/> However 2.1.1 clients with 2.2 servers are still affected by the issue.</p> <p>Hello,</p> <p>2.2.0 clients are not usable yet for us (one unreported LBUG)</p> <p>Is there any plan to include fix for the issue in upcoming 2.1.2?</p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4185" title="Incorrect permission handling when creating existing directories at ICHEC" class="issue-link" data-issue-key="LU-4185"><del>LU-4185</del></a>.</p> Duplicate LU-4185 Bugzilla ID 23459.0 Development Rank 1|hzv37r: Rank (Obsolete) 4020 Severity