https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15819 Lustre <p>We received several reports about applications (IOR and other unrelated user-provided programs) started from Lustre receiving SIGBUS signals. </p> <p> </p> <p>We were able to reproduce the issue with IOR, RHEL7 kernel on the client.</p> <p> </p> <p>It seems that it is caused by <a href="https://jira.whamcloud.com/browse/LU-14541" title="Memory reclaim caused a stale data read" class="issue-link" data-issue-key="LU-14541"><del>LU-14541</del></a> and the mechanics is the following:</p> <p> </p> <p>1) a major fault in the IOR code happens</p> <p>2) ll_fault()<del>&gt;...</del>&gt;filemap_fault()</p> <p>3) ll_readpage() is issued from filemap_fault()</p> <p>4) wait_on_page_locked() is issued from filemap_fault()</p> <p>5) the uptodate check in filemap_fault() fails due to a parallel ClearPageUptodate() called from a blocking AST handler</p> <p>6) VM_FAULT_SIGBUS is returned</p> LU-15819

Executables run from Lustre may receive spurious SIGBUS signals

Bug Blocker Closed Duplicate Andrew Perepechko Andrew Perepechko Wed, 4 May 2022 16:21:55 +0000 Thu, 5 May 2022 17:17:27 +0000 Wed, 4 May 2022 17:23:41 +0000 Upstream 0 8 <p>Panda</p> <p>Isn't this a duplicate of <a href="https://jira.whamcloud.com/browse/LU-15815" title="fast_read/stale data/reclaim workround causes SIGBUS" class="issue-link" data-issue-key="LU-15815"><del>LU-15815</del></a>?</p> <p>Peter</p> <p>Peter, yes, it is. Sorry, I searched for similar bugs some time ago but forgot to do it right before opening the ticket. We can close this one.</p> <p>While this definitely seems like a duplicate of <a href="https://jira.whamcloud.com/browse/LU-15815" title="fast_read/stale data/reclaim workround causes SIGBUS" class="issue-link" data-issue-key="LU-15815"><del>LU-15815</del></a>, is anyone from HPE currently looking at this issue? There were comments in <a href="https://jira.whamcloud.com/browse/LU-14541" title="Memory reclaim caused a stale data read" class="issue-link" data-issue-key="LU-14541"><del>LU-14541</del></a> that Shadow was looking into this same issue:</p> <blockquote> <p>This patch is tested already - it solve a problem, but some problems exist.<br/> 1) page isn't freed and stay in memory for long time until page cache LRU will flush it.<br/> 2) page without uptodate flag may cause a EIO in some cases, a specially with splice. Don't sure - but possible.</p> <p>I have a different patch with change a cl_page states change to avoid own a CPS_FREED pages, but no resources to verify it.<br/> in our case, it reproduced with overstripe with 5000 stripes and sysctl -w vm.drop_caches=3 on client nodes in parallel to the IOR.</p></blockquote> <p>It looks like John Hammond has an MPI reproducer on <a href="https://jira.whamcloud.com/browse/LU-15815" title="fast_read/stale data/reclaim workround causes SIGBUS" class="issue-link" data-issue-key="LU-15815"><del>LU-15815</del></a>, and an even simpler reproducer on <a href="https://jira.whamcloud.com/browse/LU-14541" title="Memory reclaim caused a stale data read" class="issue-link" data-issue-key="LU-14541"><del>LU-14541</del></a> that can be run on a single client, so that should allow testing potential fixes much more easily.</p> <p>This issue is one of the few 2.15.0 blockers that does not have a fix yet.</p> <p>We discussed a few possible solutions specifically for this (<a href="https://jira.whamcloud.com/browse/LU-15815" title="fast_read/stale data/reclaim workround causes SIGBUS" class="issue-link" data-issue-key="LU-15815"><del>LU-15815</del></a>/<a href="https://jira.whamcloud.com/browse/LU-15819" title="Executables run from Lustre may receive spurious SIGBUS signals" class="issue-link" data-issue-key="LU-15819"><del>LU-15819</del></a>) issue.</p> <p>E.g. we could add the page-&gt;mapping == NULL check like the kernel does in do_generic_file_read() to work around invalidate_mapping_pages(). However, we want to avoid copying filemap_fault() implementations for every supported kernel.</p> <p> </p> <p>As for alternative fixes for <a href="https://jira.whamcloud.com/browse/LU-14541" title="Memory reclaim caused a stale data read" class="issue-link" data-issue-key="LU-14541"><del>LU-14541</del></a> such as mentioned by <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=alyashkov" class="user-hover" rel="alyashkov">alyashkov</a>, I believe, this kind of fix was too complicated and was abandoned in favour of ClearPageUptodate(). While there was understanding that it was not fully legitimate to call ClearPageUptodate(), the page fault path was not considered and properly tested. I asked him to put a comment as to why the patch was abandoned and whether it can be restored and finished.</p> <p>&gt; It looks like John Hammond has an MPI reproducer on <a href="https://jira.whamcloud.com/browse/LU-15815" title="fast_read/stale data/reclaim workround causes SIGBUS" class="issue-link" data-issue-key="LU-15815"><del>LU-15815</del></a>, and an even simpler reproducer on <a href="https://jira.whamcloud.com/browse/LU-14541" title="Memory reclaim caused a stale data read" class="issue-link" data-issue-key="LU-14541"><del>LU-14541</del></a> that can be run on a single client, so that should allow testing potential fixes much more easily.</p> <p>The MPI reproducer does not require multiple clients. It only requires 2 procs. You can use oversubscribe to place them both on the same node. It's unlikely that it has much to do with MPI it's just that the application generates the right memory map access pattern to reproduce the bug. I did try a bit to make something simpler but I not successful. It would be good to have a simplified reproducer.</p> Duplicate LU-15815 Related LU-14541 Development Rank 1|i02oyv: Rank (Obsolete) 9223372036854775807 Severity