https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13541 Lustre <p>While launching Slurm jobs on our cluster, some of jobs hung quite early, in a "dd" command where a dataset is copied to a local ext4 filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_file_read_iter+0xa1/0x290 [lustre] new_sync_read+0x122/0x1b0 __vfs_read+0x29/0x40 vfs_read+0x8e/0x130 ksys_read+0xa7/0xe0 </pre> </div></div> <p>It happened on multiple nodes on the cluster. But on some nodes, it works fine. It seems to be correlated to a kernel "divide error" (division by zero?) in the kernel log of those nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2171.682001] divide error: 0000 1 SMP NOPTI [ 2171.686888] CPU: 133 PID: 35015 Comm: python Tainted: P OE 5.3.0-24-generic #26~18.04.2-Ubuntu [ 2171.706858] RIP: 0010:ll_readpage+0x25d/0x730 [lustre] [ 2171.802801] Call Trace: [ 2171.805539] filemap_fault+0x9be/0x9f0 [ 2171.830810] ll_fault+0xdb/0x710 [lustre] [ 2171.839869] __do_fault+0x57/0x117 [ 2171.843668] __handle_mm_fault+0xda0/0x1230 [ 2171.848344] handle_mm_fault+0xcb/0x210 [ 2171.852634] __do_page_fault+0x2a1/0x4d0 [ 2171.857018] do_page_fault+0x2c/0xe0 [ 2171.861014] page_fault+0x34/0x40 </pre> </div></div> <p>It seems that some of these errors were caused by these jobs, according to the time. But some of them were not (probably by another unrelated job); but the bad state lingers and block anyone wanting to access this particular file. Other files seem fine, but this file is now poisoned.</p> LU-13541

Application hang and kernel "divide error" in ll_readpage

Bug Minor Resolved Fixed Wang Shilong Andreas Dilger Fri, 8 May 2020 21:48:43 +0000 Fri, 21 Aug 2020 20:40:14 +0000 Sat, 16 May 2020 13:55:45 +0000 Lustre 2.14.0 0 4 <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38545" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38545</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13541" title="Application hang and kernel &quot;divide error&quot; in ll_readpage" class="issue-link" data-issue-key="LU-13541"><del>LU-13541</del></a> llite: fix possible divide zero in ll_use_fast_io()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bb54a8ac674f2ed4de3cbc1f5183bab2b43d5dd6</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) x/i ll_readpage+0x25d 0x38a2d &lt;ll_readpage+605&gt;: idiv %rcx (gdb) disas /s ll_readpage 1524 skip_pages = (ras-&gt;ras_stride_length + 0x0000000000038a20 &lt;+592&gt;: mov 0x50(%r10),%rax 0x0000000000038a24 &lt;+596&gt;: add %rcx,%rax 1525 ras-&gt;ras_stride_bytes - 1) / ras-&gt;ras_stride_bytes; 0x0000000000038a27 &lt;+599&gt;: sub $0x1,%rax 1524 skip_pages = (ras-&gt;ras_stride_length + 0x0000000000038a2b &lt;+603&gt;: cqto 0x0000000000038a2d &lt;+605&gt;: idiv %rcx </pre> </div></div> <p>So, I guess "ll_use_fast_io" was inlined and ras-&gt;ras_stride_bytes == 0?</p></blockquote> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38545/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38545/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13541" title="Application hang and kernel &quot;divide error&quot; in ll_readpage" class="issue-link" data-issue-key="LU-13541"><del>LU-13541</del></a> llite: fix possible divide zero in ll_use_fast_io()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7cd0afe583211a11cfe3c1041e5b982e65769f37</p> <p>Landed for 2.14</p> Duplicate Related LU-12644 Development Rank 1|i00zzj: Rank (Obsolete) 9223372036854775807 Severity