[LU-9208] getcwd() sometimes fails

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Affects Version/s: Lustre 2.5.3, Lustre 2.8.0
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Ned Bass (Inactive)

Description

We have an IO test called Miranda which is written in Fortran (attached). It intermittently fails when running in Lustre because getcwd() returns NULL with errno set to ENOENT. This behavior is similar to what was reported in LU-645. We've reproduced the problem on a single node on a Lustre 2.8 client server environment as well as Lustre 2.5. Typically the problem will occur within an hour or two when running Miranda continuously in a loop. Typical invocation is something like

cd /p/lquake/some_lustre_dir
srun -N 1 -n 36 -pplustre28 /path/to/miranda_io 100

The failing run prints out something like

forrtl: severe (121): Cannot access current working directory for unit 10739, file "Unknown"
Image              PC                Routine            Line        Source             
miranda_io         000000000040FEE9  Unknown               Unknown  Unknown
miranda_io         000000000041C992  Unknown               Unknown  Unknown
miranda_io         000000000040A4F1  Unknown               Unknown  Unknown
miranda_io         0000000000408F9E  Unknown               Unknown  Unknown
libc.so.6          00002AAAABBAEB35  Unknown               Unknown  Unknown
miranda_io         0000000000408EA9  Unknown               Unknown  Unknown
srun: error: opal93: task 735: Exited with exit code 121
srun: First task exited 30s ago
srun: tasks 0-734,736-1259: running
srun: task 735: exited abnormally
srun: Terminating job step 1166343.0
slurmd[opal38]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 ***
slurmd[opal40]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 ***
slurmd[opal35]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 ***
slurmd[opal36]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 ***
slurmd[opal41]: *** STEP 1166343.0 KILLED AT 2017-03-02T08:52:13 WITH SIGNAL 9 ***
...

I haven't collected debug logs yet but the bug shouldn't be hard to reproduce in a test environment.

Issue Links

• is duplicated by: LU-9735 Sles12Sp2 and 2.9 getcwd() sometimes fails

Comments

[Zhenyu Xu]

I didn't see miranda.log in the attachment in this ticket.

[Ned Bass (Inactive)]

I was appending miranda_io output to miranda.log. I suspect the ioctl failure is unrelated to the getcwd() problem.

[Ned Bass (Inactive)]

The getcwd() system call returns ENOENT.

[Zhenyu Xu]

What system call returns what error in this case? I see a file ioctrl cmd (0x5401 TCGETS) upon miranda.log failure in the log.

[Ned Bass (Inactive)]

No Lustre messages appear in dmesg at the time.

[Ned Bass (Inactive)]

I attached a -1 debug log from the client when it hit this bug.

[Zhenyu Xu]

Would you please grab some debug logs and dmesg when you rehit it?