https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4451 Lustre <p>Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.631107&#93;</span> BUG: unable to handle kernel NULL pointer dereference at 0000000000000028<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.639140&#93;</span> IP: <span class="error">&#91;&lt;ffffffffa0b9c23d&gt;&#93;</span> ll_sai_unplug+0x1d/0x470 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.645869&#93;</span> PGD 0 <br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.647983&#93;</span> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP <br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.651330&#93;</span> Modules linked in: lmv(C) fld(C) mgc(C) lustre(C) lov(C) osc(C) mdc(C) fid(C) ptlrpc(C) obdclass(C) lvfs(C) zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) ksocklnd(C) ko2iblnd(C) lnet(C) sha512_ssse3 sha512_generic sha256_ssse3 sha256_generic crc32 crc32c_intel libcfs(C) sg ib_umad ib_ipoib xfs libcrc32c nfsd exportfs ch st hid_generic usbhid ehci_pci ehci_hcd psmouse uhci_hcd lpc_ich mfd_core aacraid usbcore usb_common i7core_edac igb edac_core i2c_algo_bit ib_qib ixgbe mdio rdma_ucm rdma_cm ib_cm acpi_cpufreq iw_cm ib_sa ib_mad ib_addr processor ipv6 ib_uverbs ib_core qla2xxx blcr(O) scsi_transport_fc blcr_imports(O) scsi_tgt dm_mod<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.712442&#93;</span> CPU: 2 PID: 15987 Comm: nfsd Tainted: P C O 3.12.5-ql-generic-15 #1<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.720706&#93;</span> Hardware name: Supermicro X8DTN/X8DTN, BIOS 080015 05/04/2009<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.727726&#93;</span> task: ffff8800bac99cc0 ti: ffff88019a610000 task.ti: ffff88019a610000<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.735354&#93;</span> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0b9c23d&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0b9c23d&gt;&#93;</span> ll_sai_unplug+0x1d/0x470 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.744483&#93;</span> RSP: 0018:ffff88019a611718 EFLAGS: 00010282<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.749945&#93;</span> RAX: 000000005a5a5a5a RBX: ffff8800802fa000 RCX: ffff8800802fa060<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.757177&#93;</span> RDX: ffff8800802fa060 RSI: ffff8800816ab580 RDI: ffff8800802fa000<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.764405&#93;</span> RBP: ffff88019a611798 R08: ffff88019a610000 R09: 0000000000000211<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.771737&#93;</span> R10: 0000000000000000 R11: 0140000000000000 R12: ffff8800816ab580<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.779009&#93;</span> R13: 0000000000000000 R14: ffff88019a611848 R15: ffff8800802fa058<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.786254&#93;</span> FS: 0000000000000000(0000) GS:ffff8801b9c80000(0000) knlGS:0000000000000000<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.794586&#93;</span> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.800429&#93;</span> CR2: 0000000000000028 CR3: 000000000169e000 CR4: 00000000000007e0<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.807677&#93;</span> Stack:<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.809763&#93;</span> 0000000000000000 dead000000200200 00000001002f9aab ffff8801b7134000<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.817515&#93;</span> ffffffff8104c950 ffff8800bac99cc0 ffff88019a611768 ffffffff8104dd0a<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.825162&#93;</span> 0000000000000000 0000000000000282 ffff88019a611798 ffff8800816ab580<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.832806&#93;</span> Call Trace:<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.835341&#93;</span> <span class="error">&#91;&lt;ffffffff8104c950&gt;&#93;</span> ? usleep_range+0x40/0x40<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.840959&#93;</span> <span class="error">&#91;&lt;ffffffff8104dd0a&gt;&#93;</span> ? recalc_sigpending+0x1a/0x50<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.846969&#93;</span> <span class="error">&#91;&lt;ffffffffa0b9ff03&gt;&#93;</span> do_statahead_enter+0x183/0x13a0 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.854006&#93;</span> <span class="error">&#91;&lt;ffffffffa093b8bd&gt;&#93;</span> ? ldlm_res_hop_get_locked+0xd/0x10 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.861226&#93;</span> <span class="error">&#91;&lt;ffffffff810aeaad&gt;&#93;</span> ? from_kgid+0xd/0x10<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.866452&#93;</span> <span class="error">&#91;&lt;ffffffffa0986495&gt;&#93;</span> ? get_my_ctx+0x55/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.872786&#93;</span> <span class="error">&#91;&lt;ffffffff8106c070&gt;&#93;</span> ? try_to_wake_up+0x290/0x290<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.878734&#93;</span> <span class="error">&#91;&lt;ffffffffa0b8c0d2&gt;&#93;</span> ll_lookup_it+0x552/0x970 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.885130&#93;</span> <span class="error">&#91;&lt;ffffffffa0b8acdb&gt;&#93;</span> ? ll_iget+0x13b/0x280 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.891316&#93;</span> <span class="error">&#91;&lt;ffffffffa0c21444&gt;&#93;</span> ? lmv_get_lustre_md+0xf4/0x290 <span class="error">&#91;lmv&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.897920&#93;</span> <span class="error">&#91;&lt;ffffffffa0c21787&gt;&#93;</span> ? lmv_free_lustre_md+0x1a7/0x4a0 <span class="error">&#91;lmv&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.904777&#93;</span> <span class="error">&#91;&lt;ffffffffa0b519f2&gt;&#93;</span> ? ll_dcompare+0x42/0x100 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.911148&#93;</span> <span class="error">&#91;&lt;ffffffffa0b8d21a&gt;&#93;</span> ll_lookup_nd+0x7a/0x170 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.917413&#93;</span> <span class="error">&#91;&lt;ffffffff81140fa8&gt;&#93;</span> lookup_real+0x18/0x50<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.922782&#93;</span> <span class="error">&#91;&lt;ffffffff81141bd3&gt;&#93;</span> __lookup_hash+0x33/0x40<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.928260&#93;</span> <span class="error">&#91;&lt;ffffffff81147246&gt;&#93;</span> lookup_one_len+0xc6/0x120<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.933928&#93;</span> <span class="error">&#91;&lt;ffffffffa03fe880&gt;&#93;</span> encode_entryplus_baggage+0x70/0x160 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.941157&#93;</span> <span class="error">&#91;&lt;ffffffffa03fecf5&gt;&#93;</span> encode_entry.isra.11+0x2c5/0x300 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.948028&#93;</span> <span class="error">&#91;&lt;ffffffffa04000e0&gt;&#93;</span> ? nfs3svc_encode_entry+0x10/0x10 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.954962&#93;</span> <span class="error">&#91;&lt;ffffffffa04000ef&gt;&#93;</span> nfs3svc_encode_entry_plus+0xf/0x20 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.962102&#93;</span> <span class="error">&#91;&lt;ffffffffa03f5a37&gt;&#93;</span> nfsd_readdir+0x177/0x270 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.968278&#93;</span> <span class="error">&#91;&lt;ffffffff8148d34c&gt;&#93;</span> ? cache_check+0x5c/0x330<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.973912&#93;</span> <span class="error">&#91;&lt;ffffffffa03f3550&gt;&#93;</span> ? _get_posix_acl+0x60/0x60 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.980260&#93;</span> <span class="error">&#91;&lt;ffffffffa03fcfac&gt;&#93;</span> nfsd3_proc_readdirplus+0xac/0x1b0 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.987298&#93;</span> <span class="error">&#91;&lt;ffffffffa03efca1&gt;&#93;</span> nfsd_dispatch+0xa1/0x1b0 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.993474&#93;</span> <span class="error">&#91;&lt;ffffffff81481d3f&gt;&#93;</span> svc_process_common+0x2ef/0x5a0<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31468.999555&#93;</span> <span class="error">&#91;&lt;ffffffff8148233f&gt;&#93;</span> svc_process+0xff/0x150<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.004949&#93;</span> <span class="error">&#91;&lt;ffffffffa03ef6ef&gt;&#93;</span> nfsd+0xbf/0x130 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.010341&#93;</span> <span class="error">&#91;&lt;ffffffffa03ef630&gt;&#93;</span> ? nfsd_destroy+0x80/0x80 <span class="error">&#91;nfsd&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.016520&#93;</span> <span class="error">&#91;&lt;ffffffff8106013b&gt;&#93;</span> kthread+0xbb/0xc0<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.021498&#93;</span> <span class="error">&#91;&lt;ffffffff81060080&gt;&#93;</span> ? kthread_freezable_should_stop+0x70/0x70<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.028541&#93;</span> <span class="error">&#91;&lt;ffffffff8150a8bc&gt;&#93;</span> ret_from_fork+0x7c/0xb0<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.034023&#93;</span> <span class="error">&#91;&lt;ffffffff81060080&gt;&#93;</span> ? kthread_freezable_should_stop+0x70/0x70<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.041063&#93;</span> Code: c7 00 7f bc a0 e8 84 f8 96 ff 0f 1f 40 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 f4 53 48 89 fb 48 83 ec 58 4c 8b 2f 48 85 f6 &lt;49&gt; 8b 45 28 48 8b 80 f8 02 00 00 4c 8b 78 18 0f 84 26 03 00 00 <br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.061700&#93;</span> RIP <span class="error">&#91;&lt;ffffffffa0b9c23d&gt;&#93;</span> ll_sai_unplug+0x1d/0x470 <span class="error">&#91;lustre&#93;</span><br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.068471&#93;</span> RSP &lt;ffff88019a611718&gt;<br/> Jan 7 18:44:47 fltpu-login kernel: <span class="error">&#91;31469.072091&#93;</span> CR2: 0000000000000028</p> - Client mainline kernel 3.12.5 with patches (mentioned in <a href="https://jira.whamcloud.com/browse/LU-4400" title="Another LBUG with NFS reexport mainline 3.12 client" class="issue-link" data-issue-key="LU-4400"><strike>LU-4400</strike></a>) / lustre-utils 2.4.2<br/> - Servers lustre 2.4.2/ZFS OSDs <br/> - ko2iblnd LU-4451

Kernel Oops with NFS reexport using mainline 3.12 client

Bug Major Resolved Fixed Lai Siyao Roland Fehrenbacher Tue, 7 Jan 2014 17:55:16 +0000 Mon, 19 Sep 2016 14:45:54 +0000 Mon, 19 Sep 2016 14:45:54 +0000 Lustre 2.4.2 0 7 <p>The Oops occurs after a couple of seconds when doing a "rm -r" on a large directory.</p> <p>Lai,<br/> Could you please have a look and comment on this one?<br/> Thank you!</p> <p><a href="http://review.whamcloud.com/#/c/6392/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6392/</a> should be able to fix this, could you apply this patch and verify?</p> <p>I can't find this commit in my git master clone? Where did you commit it?</p> <p>I did commit to master branch, and you should be able to use `git fetch <a href="http://review.whamcloud.com/fs/lustre-release" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/fs/lustre-release</a> refs/changes/92/6392/21 &amp;&amp; git cherry-pick FETCH_HEAD` to cherry-pick to your branch, eg. 2.4.2.</p> <p>Sorry for the late reply. Been busy with other stuff ...</p> <p>The patch in <a href="http://review.whamcloud.com/#/c/6392" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6392</a> fails miserably when trying to apply<br/> to the mainline 3.12 client code (see below). Do you have a patch that works?</p> <p>patch <del>l -p 1 &lt; ./0001</del><a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a><del>statahead-statahead-thread-wait-for-RPCs-to</del>.patch <br/> patching file lustre/include/obd.h<br/> Hunk #1 succeeded at 1098 (offset -40 lines).<br/> patching file lustre/llite/dcache.c<br/> Hunk #1 FAILED at 376.<br/> 1 out of 1 hunk FAILED &#8211; saving rejects to file lustre/llite/dcache.c.rej<br/> patching file lustre/llite/file.c<br/> Hunk #1 succeeded at 321 (offset -49 lines).<br/> Hunk #2 FAILED at 541.<br/> Hunk #3 succeeded at 664 (offset -47 lines).<br/> 1 out of 3 hunks FAILED &#8211; saving rejects to file lustre/llite/file.c.rej<br/> patching file lustre/llite/llite_internal.h<br/> Hunk #1 FAILED at 141.<br/> Hunk #2 FAILED at 183.<br/> Hunk #3 succeeded at 240 (offset -4 lines).<br/> Hunk #4 FAILED at 511.<br/> Hunk #5 FAILED at 1252.<br/> Hunk #6 FAILED at 1284.<br/> Hunk #7 succeeded at 1248 (offset -64 lines).<br/> Hunk #8 FAILED at 1319.<br/> 6 out of 8 hunks FAILED &#8211; saving rejects to file lustre/llite/llite_internal.h.rej<br/> patching file lustre/llite/llite_lib.c<br/> Hunk #1 FAILED at 137.<br/> Hunk #2 FAILED at 719.<br/> Hunk #3 FAILED at 740.<br/> Hunk #4 succeeded at 926 (offset -29 lines).<br/> 3 out of 4 hunks FAILED &#8211; saving rejects to file lustre/llite/llite_lib.c.rej<br/> patching file lustre/llite/statahead.c<br/> Hunk #1 FAILED at 64.<br/> Hunk #2 FAILED at 212.<br/> Hunk #3 succeeded at 244 with fuzz 2 (offset -1 lines).<br/> Hunk #4 succeeded at 303 (offset 23 lines).<br/> Hunk #5 FAILED at 299.<br/> Hunk #6 succeeded at 360 with fuzz 1 (offset 23 lines).<br/> Hunk #7 succeeded at 378 with fuzz 1 (offset 23 lines).<br/> Hunk #8 succeeded at 418 (offset 23 lines).<br/> Hunk #9 succeeded at 440 (offset 23 lines).<br/> Hunk #10 FAILED at 441.<br/> Hunk #11 FAILED at 476.<br/> Hunk #12 FAILED at 524.<br/> Hunk #13 FAILED at 599.<br/> Hunk #14 FAILED at 616.<br/> Hunk #15 FAILED at 678.<br/> Hunk #16 succeeded at 792 (offset 9 lines).<br/> Hunk #17 succeeded at 814 (offset 9 lines).<br/> Hunk #18 FAILED at 930.<br/> Hunk #19 FAILED at 1002.<br/> Hunk #20 FAILED at 1036.<br/> Hunk #21 succeeded at 1070 (offset 3 lines).<br/> Hunk #22 succeeded at 1142 (offset 3 lines).<br/> Hunk #23 FAILED at 1195.<br/> Hunk #24 succeeded at 1252 with fuzz 1 (offset 2 lines).<br/> Hunk #25 FAILED at 1487.<br/> 14 out of 25 hunks FAILED &#8211; saving rejects to file lustre/llite/statahead.c.rej</p> <p>Hmm, I'll rebase it to latest master code later.</p> <p>Please note, that we'd need a patch against the in-kernel code of vanilla 3.12.8, not against Lustre master.</p> <p>I don't have the test environment for 3.12.8, and lustre client support for vanilla kernel is done by Peng Tao. I've just updated the patch to latest master. </p> <p>I've ported the patch to the in-kernel client. Also needed to add ll_revalidate_dentry and change ll_revalidate_nd as in master (see patch 1) The problem is gone.<br/> Can someone review the patches and make sure they are included upstream.</p> <p>Patch 1 was merged upstream as commit : commit f236f69b48727d6459c02bfabcadb9bfaacbe504. The second patch has not been merged. Roland is this problem still present in the latest kernel tree?</p> <p>Yes. The second patch is absolutely needed. Our system runs fine for several months now with those two patches.</p> <p>Can some one link this to <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a></p> Related LU-3270 LU-4011 LU-6215 LU-4416 Development Rank 1|hzwcbz: Rank (Obsolete) 12201 Severity