[LU-3177] mpi process crash on accessing file/directory

Tue, 11 Mar 2014 01:26:45 +0000

We're running WRF (3.4.1) using 300 to 500 processes. There are situations that some runs is stuck. We found that in those cases, one of the processes crashes at the very beginning. In WRF's rsl.error.* file, there are such an error message:

forrtl: severe (121): Cannot access current working directory for unit 27, file "Unknown"

About 5% of the runs encounter this problem. In searching Lustre bug report, we guess it might be related to this one:

https://projectlava.xyratex.com/show_bug.cgi?id=23978

We then follow the suggested workaround by using a getcwd wrapper (without changing the WRF source code), the problem seems to be gone in the subsequent 300 runs. And in the log message generated by the wrapper, we found the following:

rsl.out.0095: getcwdfixwrap: host/pid: node23/87536 time: 1366042583 problem null, retryctr: 0 errno: 2 errstr: "No such file or directory"
rsl.out.0095: getcwdfixwrap: host/pid: node23/87536 problem buf non null, value:

Seems that the workaround bails out the getcwd problem.

There seems to a fix in 1.8.x version. We wonder if such fix will be applied to the 2.1.4 version.

Whamcloud Community JIRA

[LU-3177] mpi process crash on accessing file/directory