https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3579 Lustre <p>After applying the patch for <a href="https://jira.whamcloud.com/browse/LU-1397" title="ENOENT on open()" class="issue-link" data-issue-key="LU-1397"><del>LU-1397</del></a>, Fujitsu has seen performance for their benchmark suite decrease by about 7%. This is pushing the system outside of the acceptance range, and so therefore a high priority for us. I am trying to get information about what exactly is going more slowly (what application/file operations), but would it be possible for someone to review the patch to see if there are any areas that could be improved?</p> <p>Thanks.</p> LU-3579

Performance regression after applying fix for LU-1397

Bug Major Resolved Cannot Reproduce Oleg Drokin Manish Patel Thu, 11 Jul 2013 18:09:21 +0000 Fri, 14 Nov 2014 14:48:45 +0000 Fri, 14 Nov 2014 14:48:45 +0000 Lustre 2.1.6 0 8 <p>Can we take a step back here? What motivated applying the patch for <a href="https://jira.whamcloud.com/browse/LU-1397" title="ENOENT on open()" class="issue-link" data-issue-key="LU-1397"><del>LU-1397</del></a> in the first place? Reading the notes relating to that, it was deemed unnecessary and abandoned. Are any other patches applied or is this otherwise a vanilla 2.1.6 deployment across the board?</p> <p>Well the patch we applied was actually part of <a href="https://jira.whamcloud.com/browse/LU-1234" title="Executing binary stored on Lustre results in &quot; (deleted)&quot; appended to /proc/self/exec" class="issue-link" data-issue-key="LU-1234"><del>LU-1234</del></a>. I'm rereading the bugs and perhaps I misread it originally. Basically the issue is that the applications are getting ENOENT errors on 75% of the runs. I thought that the patch in <a href="https://jira.whamcloud.com/browse/LU-1234" title="Executing binary stored on Lustre results in &quot; (deleted)&quot; appended to /proc/self/exec" class="issue-link" data-issue-key="LU-1234"><del>LU-1234</del></a> fixed the ENOENTs in 1397, but rereading it, it sounds like it was actually introduced by an earlier version of the <a href="https://jira.whamcloud.com/browse/LU-1234" title="Executing binary stored on Lustre results in &quot; (deleted)&quot; appended to /proc/self/exec" class="issue-link" data-issue-key="LU-1234"><del>LU-1234</del></a> patch? It's interesting to note however that we didn't run into the ENOENT issue after applying it, though it could have been luck.</p> <p>Oh I just realized I didn't say the version history in my initial post. They are running 2.1.3 servers and were running 2.1.3 clients. After running into the ENOENT issue, we upgraded them to 2.1.6 clients and that's when we first saw the regression. We then built a version of 2.1.3 only with:<br/> <a href="http://review.whamcloud.com/#/c/2400/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/2400/</a></p> <p>The regression was still in that version as well. We are rerunning the test with 2.1.3 clients just to confirm that nothing else has changed, and the regression is definitely in the <a href="https://jira.whamcloud.com/browse/LU-1234" title="Executing binary stored on Lustre results in &quot; (deleted)&quot; appended to /proc/self/exec" class="issue-link" data-issue-key="LU-1234"><del>LU-1234</del></a> patch. </p> <p>Going back to the original, ENOENT issue. It looks like files are not being created at all. Or if they are being created they are then completely disappearing. Here is the description from the customer:</p> <blockquote> <p>It appears data is going missing during a Quantum Espresso job run:</p> <p>An example of the output from a Quantum Espresso failure is attached.</p> <p>Whilst the QuantumEspresso application runs it opens a scratch file for each MPI process. Occasionally when it tries to reopen one of these files it’s not there and it crashes with an error message like:</p> <p> Error in routine seqopn (16):<br/> error opening ./ausurf.igk2</p> <p>All input, output files for the run are in:<br/> /scratch/nick.wilson/parallel_benchmarks/AUSURF112.19274.1</p> <p>We believe this to be a Lustre issue as when it’s run against local disk for scratch space there are no failures, only when we use Lustre for scratch space does this happen. </p></blockquote> <p>Do you have any opinions on what debug settings to run in order to get more information? I was thinking that rpctrace might be the best to start with. Any advice on it?</p> <p>Thanks.</p> <p>if the initial open-create succeeded, the file definitely was created and opened.<br/> Now if it's not there later, it might have been just deleted? Can you monitor delete stats on MDT to confirm this?</p> <p>You can also enable rpctrace, but that will only show you what sort of RPCs were sent around, not names of files or some such.<br/> On a client you need to also do vfstrace to see the names of files. On MDT you need to enable "inode" tracer to see those details, but that would slow it down quite a bit I imagine.</p> <p>Hi,</p> <p>This ticket can be closed, since we do not have any more information in this case.</p> <p>Thank You,<br/> Manish</p> <p>ok thanks Manish</p> Related LU-1397 Development Rank 1|hzvv6v: Rank (Obsolete) 9057 Severity