https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11403 Lustre <p>REFORMAT=yes RACER_ENABLE_DOM=false MOUNT_2=yes SLOW=yes sh racer.sh</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> LustreError: 20867:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000402:0x116f5:0x0] BUG: unable to handle kernel NULL pointer dereference at (<span class="code-keyword">null</span>) IP: [&lt;ffffffffa0cc04e8&gt;] ll_fault+0xb8/0x610 [lustre] PGD 0 Oops: 0002 [#1] SMP Modules linked in: zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) lov(O) mdc(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) CPU: 0 PID: 20867 Comm: systemd-coredum Tainted: P O ------------ 3.10.0 #5 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 task: ffff8800aebaa0c0 ti: ffff88006485c000 task.ti: ffff88006485c000 RIP: 0010:[&lt;ffffffffa0cc04e8&gt;] [&lt;ffffffffa0cc04e8&gt;] ll_fault+0xb8/0x610 [lustre] RSP: 0018:ffff88006485fc70 EFLAGS: 00010246 RAX: 0000000000000001 RBX: 0000000000000100 RCX: 00000000c0000100 RDX: ffffffff81ab94c0 RSI: ffff8800aebaa0c0 RDI: ffff88011fc15b00 RBP: ffff88006485fcd8 R08: ffff88006485c000 R09: 00000000000004c1 R10: 000000009837f050 R11: ffff88011fc15b70 R12: 0000000000000000 R13: ffff8801157a4b60 R14: ffff88006485fce8 R15: ffff8800aebaa0c0 FS: 00007f4c1846c940(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000000689c0000 CR4: 00000000000007b0 Call Trace: [&lt;ffffffff811649f9&gt;] __do_fault.isra.12+0x79/0xd0 [&lt;ffffffff8116d563&gt;] ? __vma_link_file+0x43/0x80 [&lt;ffffffff81164a88&gt;] do_read_fault.isra.14+0x38/0x170 [&lt;ffffffff811703f7&gt;] ? mmap_region+0x207/0x740 [&lt;ffffffff8116a7ae&gt;] handle_mm_fault+0x77e/0x11a0 [&lt;ffffffff815b1078&gt;] __do_page_fault+0x1b8/0x4b0 [&lt;ffffffff815b13e3&gt;] trace_do_page_fault+0x43/0xd0 [&lt;ffffffff815b08ed&gt;] do_async_page_fault+0x5d/0xa0 </pre> </div></div> LU-11403

lov_io_init_empty() Page fault on a file without stripes

Bug Minor Resolved Fixed Patrick Farrell Alex Zhuravlev Wed, 19 Sep 2018 18:55:11 +0000 Sat, 8 Jun 2019 12:18:07 +0000 Sat, 4 May 2019 14:36:24 +0000 Upstream Lustre 2.13.0 Lustre 2.12.3 0 4 <p>Not sure why we haven't been hitting this forever, but I'm pretty sure this is the bug.</p> <p>When there's no stripe, we return EFAULT (lov_io_init_empty):</p> <p>Which is translated to VM_FAULT_NOPAGE (to_fault_error)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> case -EFAULT: result = VM_FAULT_NOPAGE; break;</pre> </div></div> <p> </p> <p>Then in ll_fault:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (!(result &amp; (VM_FAULT_RETRY | VM_FAULT_ERROR | VM_FAULT_LOCKED))) { struct page *vmpage = vmf-&gt;page; /* check if this page has been truncated */ lock_page(vmpage); </pre> </div></div> <p>But the page is <b>not</b> set in this case.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int __do_fault(struct vm_area_struct *vma, unsigned long address, pgoff_t pgoff, unsigned int flags, struct page *cow_page, struct page **page, void **entry, pmd_t *pmd, pte_t orig_pte) { struct vm_fault vmf; int ret; vmf.virtual_address = (void __user *)(address &amp; PAGE_MASK); vmf.pgoff = pgoff; vmf.flags = flags; vmf.page = NULL; &lt;---------------------- vmf.gfp_mask = __get_fault_gfp_mask(vma); vmf.cow_page = cow_page; vmf.orig_pte = orig_pte; vmf.pmd = pmd; vmf.vma = vma; ret = vma-&gt;vm_ops-&gt;fault(vma, &amp;vmf); &lt;------------- ll_fault </pre> </div></div> <p>Various errors can cause page to not get set.</p> <p>Fix seems clear - Check for the page.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34247" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34247</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> llite: Check if page is set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b0522f8f147f71a2bb150570769693de196df763</p> <p>attached a simpler reproducer for the ll_fault() crash <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32210/32210_m.c" title="m.c attached to LU-11403">m.c^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> (made by Andrew Perepechko).</p> <p><a href="https://review.whamcloud.com/34247" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34247</a> causes the reproducer to hang with repeatable messages in kernel log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 87.575164] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000401:0x1:0x0] [ 87.575546] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Skipped 251880 previous similar messages [ 89.577930] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000401:0x1:0x0] [ 89.578283] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Skipped 493482 previous similar messages [ 93.579391] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000401:0x1:0x0] [ 93.579750] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Skipped 946319 previous similar messages [ 101.581054] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000401:0x1:0x0] [ 101.581550] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Skipped 1945088 previous similar messages [ 117.585786] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Page fault on a file without stripes: [0x200000401:0x1:0x0] [ 117.586855] LustreError: 5061:0:(lov_io.c:1481:lov_io_init_empty()) Skipped 4004012 previous similar messages </pre> </div></div> <p>instead of crash. however , the hang is interruptible from the terminal.</p> <p>Oh, cool!  Thanks very much to Panda for the reproducer and Zam for the report.  I will take a look at the hang.  (Agreed it's better than panicking, but we should be able to do better.)</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34247/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34247/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> llite: ll_fault fixes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a8f4d1e5fd79e77f1347e983ec52f2ddc3e75ab9</p> <p>Landed for 2.13</p> <p>The sanity.sh test_61b added from this patch is failing regularly.</p> <p>My thinking is that it makes sense to instantiate the layout at the time that mmap() is called for the full range of the mapping, rather than waiting for the page fault. </p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34698" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34698</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> tests: Fix $tfile usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 37a5b05ac0fe80f551ec279f070726edb04d7b33</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34698/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34698/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> tests: Fix $tfile usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4191e0cdd0d96b848c1235471179d25d37a889dc</p> <p>Take two</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34914" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34914</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> tests: Fix $tfile usage<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 58572b0ba1be608226af3695c4f7d50399e2bded</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34935" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34935</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> llite: ll_fault fixes<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 9760d68dd6915492f4d2241d0009b86fba85e770</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34935/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34935/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> llite: ll_fault fixes<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: a1ffe58ab0c41009b77abe553850798a1030d653</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34914/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34914/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11403" title="lov_io_init_empty() Page fault on a file without stripes" class="issue-link" data-issue-key="LU-11403"><del>LU-11403</del></a> tests: Fix $tfile usage<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 5866a8975fe95cbd07b370ee4cf838ba223c13da</p> Related LU-12213 Development Rank 1|i002q7: Rank (Obsolete) 9223372036854775807 Severity