https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4308 Lustre <p>When a MPI job is run, we see many of these messages "binary x changed while waiting for the page fault lock." Is this normal behavior or not? It was also reported here.</p> <p><a href="https://lists.01.org/pipermail/hpdd-discuss/2013-October/000560.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.01.org/pipermail/hpdd-discuss/2013-October/000560.html</a></p> <p>Nov 25 13:46:50 rhea25 kernel: Lustre: 105703:0:vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20000f81c:0x18:0x0&#93;</span> changed while waiting for the page fault lock<br/> Nov 25 13:46:53 rhea25 kernel: Lustre: 105751:0:(vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20000f81c:0x19:0x0&#93;</span> changed while waiting for the page fault lock<br/> Nov 25 13:46:57 rhea25 kernel: Lustre: 105803:0:(vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20000f81c:0x1a:0x0&#93;</span> changed while waiting for the page fault lock<br/> Nov 25 13:46:57 rhea25 kernel: Lustre: 105803:0:(vvp_io.c:699:vvp_io_fault_start()) Skipped 1 previous similar message<br/> Nov 25 13:47:00 rhea25 kernel: Lustre: 105846:0:(vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20000f81c:0x1b:0x0&#93;</span> changed while waiting for the page fault lock<br/> Nov 25 13:47:00 rhea25 kernel: Lustre: 105846:0:(vvp_io.c:699:vvp_io_fault_start()) Skipped 2 previous similar messages<br/> Nov 25 13:47:07 rhea25 kernel: Lustre: 105942:0:(vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20000f81c:0x1d:0x0&#93;</span> changed while waiting for the page fault lock</p> RHEL 6.4/MLNX OFED 2.0.2.6.8.10 LU-4308

MPI job causes errors "binary changed while waiting for the page fault lock"

Bug Minor Resolved Fixed Zhenyu Xu Blake Caldwell Mon, 25 Nov 2013 20:16:17 +0000 Thu, 28 Jan 2016 14:12:19 +0000 Thu, 14 Aug 2014 16:27:26 +0000 Lustre 2.4.1 Lustre 2.5.3 5 23 <p>Hi Blake</p> <p>I will get an engineer to comment asap, but just to clarify - is this really an S1 situation (i.e. a production cluster is completely inoperational)?</p> <p>Peter</p> <p>It should be severity 4</p> <p>Thanks for clarifying Blake.</p> <p>Alex</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>Blake,</p> <p>This message means that a mmapped executable file is under read while other threads may change its contents. Is this what the MPI job meant to do, ie. generating executable files while other jobs reading it?</p> <p>As per ORNL ok to close</p> <p>Hate to open something we said to close last week, but we have another occurrence of this issue on the same cluster. We are consulting with the sysadmin for that cluster, and likely with the user to discuss the question from Zhenyu. </p> <p>Mar 6 16:49:55 rhea101 kernel: Lustre: 24431:0:(vvp_io.c:699:vvp_io_fault_start()) binary <span class="error">&#91;0x20006c6eb:0xc:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>&#8211;<br/> -Jason</p> <p>We are observing a similar issue and sometimes see the same error. It only happens when we run specific jobs that use MPI with OpenMP threads. Those same jobs complete when using pure MPI.</p> <p>Symptoms:</p> <ul class="alternate" type="square"> <li>The file in question cannot be read on some specific clients (but can on most other clients)</li> <li>On those nodes where it cannot be read, processes that need to access /proc/PID/cmdline for the process executing that file block (w, ps, etc).</li> <li>File system can no longer unmount.</li> <li>sometimes clients will have open FIDs that cannot resolve to paths when we get the list of FIDs held open by the client on the MDS.</li> </ul> <p>Effectively, we have to power cycle or reboot -fn to get the nodes in question back up.</p> <p>The file in question can be read by other lustre clients. We tested running lustre client and server 2.4.2 and 2.5.1. Same results for both. To reduce the chance this is an MPI bug, we also tested on two versions of Intel MPI (4.1.0.024 and 4.1.3.049) and Intel compilers 13.1 and 14.0-1. To reduce the risk of race conditions, we pin MPI threads to a single socket.</p> <p>Here is a bit of strace for cp:</p> <p>===============<br/> stat("/scratch/short/jimj/s4_1534/stmp_2013062412_gdas_fcst1/global_fcst", </p> {st_mode=S_IFREG|0755, st_size=24534300, ...}) = 0<br/> stat("./global_fcst", 0x7fffb2d95eb0) = -1 ENOENT (No such file or directory)<br/> open("/scratch/short/jimj/s4_1534/stmp_2013062412_gdas_fcst1/global_fcst", O_RDONLY) = 3<br/> fstat(3, {st_mode=S_IFREG|0755, st_size=24534300, ...} <p>) = 0<br/> open("./global_fcst", O_WRONLY|O_CREAT|O_EXCL, 0755) = 4<br/> fstat(4, </p> {st_mode=S_IFREG|0755, st_size=0, ...} <p>) = 0<br/> mmap(NULL, 4202496, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b5b721c5000<br/> read(3,<br/> ===============</p> <p>At that point, it hangs indefinitely. </p> <p>For us, this may be a blocking issue. We're not sure where exactly the blame lies, but it does seem suspect that the systems cannot flush bad pages or read files other lustre clients can read. We can't go into production with this hardware until we resolve this.</p> <p>We left the clients in their broken state over night. Same deal. Then, we tested deleting the file from a bad actor and observed this:</p> <p>April 29 @ 17:43<br/> 172.17.1.251 - <span class="error">&#91;0x200001435:0xad:0x0&#93;</span> - /scratch/short/jimj/s4_1534/stmp_2013062412_gdas_fcst1/global_fcst<br/> 172.17.1.251 - <span class="error">&#91;0x200001435:0x12e:0x0&#93;</span> - <br/> /scratch/short/jimj/s4_1534/stmp_2013062412_gdas_fcst1/gfs_namelist</p> <p>(rm the file)</p> <p>April 30 @ 13:35<br/> 172.17.1.251 - <span class="error">&#91;0x200001435:0xad:0x0&#93;</span> - No file or directory</p> <p>The file handle is still stuck open on all nodes where it was unreadable. The stuck nodes cannot read the file, but they can delete it. However, even deleting it doesn't release their file handle.</p> <p>I'm not confident that our issue, which I suspect is related to a bug in the lustre client, has anything to do with the error 'binary changed...'.</p> <p>We did, however, revert our clients to 2.1.6 (still running server 2.5.1) and we no longer observe the same hanging file / hanging mount problem.</p> <p>Is it easy to reproduce? Can you collect -1 logs with as simple as possible reproduce procedure and upload the logs? Thank you.</p> <p>Also would you please trying this patch <a href="http://review.whamcloud.com/10483" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10483</a> ?</p> <p>Is there a patch for client 2.5.1?<br/> We are observing the same issues in Cyfronet with 2.5.1 client and mpi jobs</p> <p>Jul 14 19:24:56 n1043-amd kernel: Lustre: 21837:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock<br/> Jul 14 19:24:56 n1043-amd kernel: Lustre: 21837:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 54 previous similar messages<br/> Jul 14 19:34:59 n1043-amd kernel: Lustre: 21813:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock<br/> Jul 14 19:34:59 n1043-amd kernel: Lustre: 21812:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock<br/> Jul 14 19:34:59 n1043-amd kernel: Lustre: 21812:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 75 previous similar messages<br/> Jul 14 19:34:59 n1043-amd kernel: Lustre: 21813:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 75 previous similar messages<br/> Jul 14 19:36:47 n1043-amd kernel: LustreError: 11-0: scratch-MDT0000-mdc-ffff882834414c00: Communicating with 172.16.193.1@o2ib, operation mds_get_info failed with -1119251304.<br/> Jul 14 19:36:47 n1043-amd kernel: LustreError: Skipped 4 previous similar messages<br/> Jul 14 19:45:00 n1043-amd kernel: Lustre: 21825:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock<br/> Jul 14 19:45:00 n1043-amd kernel: Lustre: 21825:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 71 previous similar messages<br/> Jul 14 19:55:02 n1043-amd kernel: Lustre: 21828:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock<br/> Jul 14 19:55:02 n1043-amd kernel: Lustre: 21828:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 375 previous similar messages</p> <p>@Zhenyu Xu: logs related to the following object have been uploaded to ftp:<br/> Jul 14 19:55:02 n1043-amd kernel: Lustre: 21828:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x20d545086:0x191e4:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>Please find the logs here: <a href="ftp://ftp.whamcloud.com/uploads/lu-4308.cyfronet.log.gz" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/lu-4308.cyfronet.log.gz</a> </p> <p>patch for b2_5 branch <a href="http://review.whamcloud.com/11098" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11098</a></p> <p>We've been running 2.5.1 client packages with this patch included in two separate environments for the past month. It has eliminated these error messages from occurring.</p> <p>I applied patch to 2.5.2 client but problem persists. It was VASP MPI jobs which triggered this error.</p> <p>Lustre: 62291:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x201c53daa:0x1d27:0x0&#93;</span> changed while waiting for the page fault lock<br/> Lustre: 62291:0:(vvp_io.c:692:vvp_io_fault_start()) Skipped 1 previous similar message<br/> Lustre: 62618:0:(vvp_io.c:692:vvp_io_fault_start()) binary <span class="error">&#91;0x201c53daa:0x1d27:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>The patch that has been reportedly successful at a couple of sites has been landed for 2.5.3. It is not believed that this same issue exists on 2.6 or newer releases so equivalent changes are not needed on master. If there are still residual issues affecting 2.5.x releases then please open a new ticket to track those - thanks!</p> <p>I wanted to report back on our issue because it may be related.</p> <p>Our original observation was that some of our Lustre clients would deadlock when running MPI+OpenMP executables. The executable on those clients could only be partially read, and would hang indefinitely on copy or access to /proc/&lt;pid&gt;/exe or /proc/&lt;pid&gt;/cmdline.</p> <p>We upgraded to 2.5.2 and applied the aforementioned patch and observed no change.</p> <p>We did test the following workarounds some of which were entirely successful:</p> <p>(1) enabling 'localflock' for those clients as a mount flag was completely successful.</p> <p>(2) hosting the executable on an NFS mount was completely successful.</p> <p>(3) upgrading to lustre 2.6.0 was completely successful.</p> <p>(4) running lustre 2.1.6 with khugepaged disabled mitigated the issue to a large extent, with rare observed deadlocks.</p> <p>(5) we tried running on another lustre file system (lustre 2.4.2 servers running ZFS instead of lustre 2.5.1 running ldiskfs) but did not notice any improvement to the client deadlock issue.</p> <p>We settled on running lustre 2.6.0 on the clients because we also observed a performance increase when using it.</p> <p>NOTE: this issue may have been related to an irregularity we observed in slurmd and have found a work around for as well.</p> <p>We are seeing this issue on clients running 2.5.3</p> <p>vvp_io.c:694:vvp_io_fault_start()) binary <span class="error">&#91;0x20000aaa6:0x1d668:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>It manifested while the file system was under high load.</p> Related LU-7198 Development Rank 1|hzwa1z: Rank (Obsolete) 11800 Severity