https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10132 Lustre <p>Our users are seeing consistent IO errors at very early attempts to scale jobs that access executables on lustre volumes. The errors hang the entire job. These instances are 2.10 client and EE 3.1.1 server scenarios. I've been able to replicate the errors on several different filesystems. The errors do not occur when loading the EE 3.1.1/2.7 client stack to run the same jobs. Is there any expectation of 2.10 client compatibility with EE 3.1.1 servers?</p> <p>Example job:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ mpirun -perhost 1 -np 12 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello Hello world from processor ekf087, rank 2 out of 12 processors Hello world from processor ekf067, rank 11 out of 12 processors Hello world from processor ekf082, rank 1 out of 12 processors Hello world from processor ekf195, rank 5 out of 12 processors Hello world from processor ekf095, rank 3 out of 12 processors Hello world from processor ekf194, rank 4 out of 12 processors Hello world from processor ekf355, rank 6 out of 12 processors Hello world from processor ekf358, rank 7 out of 12 processors Hello world from processor ekf364, rank 10 out of 12 processors Hello world from processor ekf361, rank 9 out of 12 processors Hello world from processor ekf359, rank 8 out of 12 processors Hello world from processor ekf067, rank 0 out of 12 processors </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$mpirun -perhost 2 -np 24 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello [proxy:0:5@ekf195] HYDU_create_process (../../utils/launch/launch.c:825): execvp error on file /lfs/lfs11/tgeerdes/hello (Input/output error) </pre> </div></div> LU-10132

IO execvp errors 2.10 client/EE3.1.1 server

Bug Minor Resolved Fixed John Hammond Trent Geerdes Fri, 13 Oct 2017 17:38:59 +0000 Wed, 15 Nov 2017 12:25:02 +0000 Wed, 1 Nov 2017 08:32:23 +0000 Lustre 2.11.0 Lustre 2.10.2 0 8 <p>Hi Trent</p> <p>We would expect this combination to interoperate and it is included in our regular release tesitng. When you say 2.10 do you mean 2.10.0 or 2.10.1 clients?</p> <p>Peter</p> <p>Hi Peter,<br/> 2.10.0 clients and this is on Endeavour.<br/> Also the only FS I'm not able to replicate it on is our highest performing SSD based EE 3.1.1 FS. Smaller SSD based and large HDD based all exhibit the issue.</p> <p>Hi Trent, can you attach logs from the clients that experienced the error, the MDT(s), and any OSTs that contained stripes from the executable.</p> <p>Client, MDT, OST's don't log any messages related to the failures. Just the typical, unrelated client disconnects, reconnects.</p> <p>How easy is it to reproduce this?</p> <p>Pretty easy. Many of our customers have hit it and I didn't have trouble reproducing when trying. </p> <p>Could you run the following on each client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param debug_mb=256 lctl set_param debug="vfstrace rpctrace dlmtrace net neterror ha trace" lctl clear </pre> </div></div> <p>Then run your reproducer</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>date +%s mpirun -perhost 2 -np 24 -host ... </pre> </div></div> <p>Then run <tt>lctl dk &gt; <a href="https://jira.whamcloud.com/browse/LU-10132" title="IO execvp errors 2.10 client/EE3.1.1 server" class="issue-link" data-issue-key="LU-10132"><del>INTL-313</del></a>.log</tt> on one of the clients where execvp() fails and attach the log file here?</p> <p>You will probably want to restore the settings to <tt>debug</tt> and <tt>debug_mb</tt> afterwards.</p> <p>I've set the params and captured the log. Attached.</p> <p>Trent, do you mind if I move this to the LU project?</p> <p>Thanks to your logs I can reproduce it locally and now understand where this is coming from.</p> <p>Sure, go ahead. Thank you.</p> <p>This is from:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ll_xattr_cache_refill(struct inode *inode) { ... <span class="code-comment">/* Matched but no cache? Cancelled on error by a parallel refill. */</span> <span class="code-keyword">if</span> (unlikely(req == NULL)) { CDEBUG(D_CACHE, <span class="code-quote">"cancelled by a parallel getxattr\n"</span>); ll_intent_drop_lock(&amp;oit); GOTO(err_unlock, rc = -EIO); } ... } </pre> </div></div> <p>Looking at the code here, we should be returning <tt>-EAGAIN</tt> instead of <tt>-EIO</tt> so that <tt>ll_getxattr_common()</tt> will handle the race.</p> <p>This affects master as well.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29654" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29654</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10132" title="IO execvp errors 2.10 client/EE3.1.1 server" class="issue-link" data-issue-key="LU-10132"><del>LU-10132</del></a> llite: handle xattr cache refill race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9cc8c3a20c547ec75325dde3dd17f4b1dcc66348</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29795" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29795</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10132" title="IO execvp errors 2.10 client/EE3.1.1 server" class="issue-link" data-issue-key="LU-10132"><del>LU-10132</del></a> llite: handle xattr cache refill race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: f9a2f8bc817829416646dc7d3ea3add16055cefe</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29654/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29654/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10132" title="IO execvp errors 2.10 client/EE3.1.1 server" class="issue-link" data-issue-key="LU-10132"><del>LU-10132</del></a> llite: handle xattr cache refill race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3dcb7d098759614ae7deb532e1555bd82dac7936</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29795/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29795/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10132" title="IO execvp errors 2.10 client/EE3.1.1 server" class="issue-link" data-issue-key="LU-10132"><del>LU-10132</del></a> llite: handle xattr cache refill race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 78a5d681932e30797775aa10d22fc25b20aa58f7</p> Related Development Rank 1|hzzlvj: Rank (Obsolete) 9223372036854775807 Severity