https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9208 Lustre <p>We have an IO test called Miranda which is written in Fortran (attached). It intermittently fails when running in Lustre because getcwd() returns NULL with errno set to ENOENT. This behavior is similar to what was reported in <a href="https://jira.whamcloud.com/browse/LU-645" title="getcwd fails" class="issue-link" data-issue-key="LU-645"><del>LU-645</del></a>. We've reproduced the problem on a single node on a Lustre 2.8 client server environment as well as Lustre 2.5. Typically the problem will occur within an hour or two when running Miranda continuously in a loop. Typical invocation is something like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cd /p/lquake/some_lustre_dir srun -N 1 -n 36 -pplustre28 /path/to/miranda_io 100 </pre> </div></div> <p>The failing run prints out something like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>forrtl: severe (121): Cannot access current working directory for unit 10739, file "Unknown" Image PC Routine Line Source miranda_io 000000000040FEE9 Unknown Unknown Unknown miranda_io 000000000041C992 Unknown Unknown Unknown miranda_io 000000000040A4F1 Unknown Unknown Unknown miranda_io 0000000000408F9E Unknown Unknown Unknown libc.so.6 00002AAAABBAEB35 Unknown Unknown Unknown miranda_io 0000000000408EA9 Unknown Unknown Unknown srun: error: opal93: task 735: Exited with exit code 121 srun: First task exited 30s ago srun: tasks 0-734,736-1259: running srun: task 735: exited abnormally srun: Terminating job step 1166343.0 slurmd[opal38]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 *** slurmd[opal40]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 *** slurmd[opal35]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 *** slurmd[opal36]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 *** slurmd[opal41]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 *** ... </pre> </div></div> <p>I haven't collected debug logs yet but the bug shouldn't be hard to reproduce in a test environment.</p> LU-9208

getcwd() sometimes fails

Bug Minor Resolved Duplicate Zhenyu Xu Ned Bass Mon, 13 Mar 2017 22:20:52 +0000 Thu, 5 Apr 2018 19:07:01 +0000 Thu, 5 Apr 2018 19:07:01 +0000 Lustre 2.5.3 Lustre 2.8.0 0 7 <p>Would you please grab some debug logs and dmesg when you rehit it?</p> <p>I attached a -1 debug log from the client when it hit this bug.</p> <p>No Lustre messages appear in dmesg at the time.</p> <p>What system call returns what error in this case? I see a file ioctrl cmd (0x5401 TCGETS) upon miranda.log failure in the log.</p> <p>The getcwd() system call returns ENOENT.</p> <p>I was appending miranda_io output to miranda.log. I suspect the ioctl failure is unrelated to the getcwd() problem.</p> <p>I didn't see miranda.log in the attachment in this ticket.</p> <p>There's nothing useful in miranda.log other than the error message that I included in the description.</p> <p>Our testing team just reported that they may have reproduced this bug on an NFS filesystem. So this may not be a Lustre-specific bug. I'll close the ticket for now, and will re-open it if we get any further evidence that it's a Lustre problem.</p> <p>Thanks for your time on this issue.</p> <p>Reopening to allow attaching a file.</p> <p>I attached a test case Red Hat provided for a similar issue seen on NFS. They are interested to know if we can reproduce this on Lustre and NFS. LLNL doesn't have cycles to spend on this right now so I am uploading here in case others want to take a stab at it. I'd be happy to feed any results back to Red Hat.</p> <p>We have recently upgraded to Sles12sp2 and lustre 2.9 clients and started to see this issues. </p> <p> </p> <p>Ned, are you also running Intel MPI when this problem is hit? According to <a href="https://jira.hpdd.intel.com/browse/LU-9735?focusedCommentId=206925" class="external-link" target="_blank" rel="nofollow noopener">the comment</a> in <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a>, this problem only appears to be relevant to Intel MPI.</p> <blockquote><p>Ned, are you also running Intel MPI when this problem is hit? According to the comment in <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a>, this problem only appears to be relevant to Intel MPI.</p></blockquote> <p>No. I can reproduce it on a single node non-MPI program. I wrote a simple C program to simulate what Miranda is doing and just ran a bunch of copies of it in the background. Eventually getcwd() fails with errno=ENOENT. I'll dig up the program and attach it here.</p> <p>It appears that the <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> fix is relevant to all kernels after 2.6.37. That patch was landed for 2.11 and 2.10.3.</p> Duplicate LU-9735 Related Development Rank 1|hzz6tb: Rank (Obsolete) 9223372036854775807 Severity