Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-4731] sanity-lfsck test 1b: Oops: IP: [<ffffffff812948e7>] __list_add+0x17/0xa0 https://jira.whamcloud.com/browse/LU-4731 Lustre <p>While running sanity-lfsck test 1b, MDS hit the following Oops:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>03:15:32:Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled 03:15:32:Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug3 03:15:32:Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug3 03:15:32:BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 03:15:32:IP: [&lt;ffffffff812948e7&gt;] __list_add+0x17/0xa0 03:15:32:PGD 0 03:15:32:Oops: 0000 [#1] SMP 03:15:32:last sysfs file: /sys/devices/system/cpu/online 03:15:32:CPU 1 03:15:32:Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) jbd2 nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: llog_test] 03:15:32: 03:15:32:Pid: 30021, comm: ll_mgs_0001 Not tainted 2.6.32-431.5.1.el6_lustre.x86_64 #1 Red Hat KVM 03:15:32:RIP: 0010:[&lt;ffffffff812948e7&gt;] [&lt;ffffffff812948e7&gt;] __list_add+0x17/0xa0 03:15:32:RSP: 0018:ffff88006cb65cd0 EFLAGS: 00010282 03:15:32:RAX: ffff88007bf135c0 RBX: ffff88006d108450 RCX: 0000000000000002 03:15:32:RDX: 0000000000000000 RSI: ffffc90001768d20 RDI: ffff88006d108450 03:15:32:RBP: ffff88006cb65cf0 R08: ffff88006d108440 R09: 0000000000000000 03:15:32:R10: 000000000000000e R11: 20736e6172742029 R12: ffffffffffffffff 03:15:32:R13: 0000000000000001 R14: 0000000000000000 R15: ffff8800605b5540 03:15:32:FS: 0000000000000000(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 03:15:32:CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b 03:15:32:CR2: 0000000000000008 CR3: 00000000379c4000 CR4: 00000000000006e0 03:15:32:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 03:15:32:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 03:15:32:Process ll_mgs_0001 (pid: 30021, threadinfo ffff88006cb64000, task ffff88006d28a080) 03:15:32:Stack: 03:15:32: 0000000000000000 000000000000001a ffffffffffffffff 0000000000000001 03:15:32:&lt;d&gt; ffff88006cb65d00 ffffffffa053a93b ffff88006cb65d60 ffffffffa05425af 03:15:32:&lt;d&gt; ffff88006cb65d30 ffff88006d108440 ffffffffffffffff ffff8800ffffffff 03:15:32:Call Trace: 03:15:32: [&lt;ffffffffa053a93b&gt;] lnet_res_lh_initialize+0x4b/0x50 [lnet] 03:15:32: [&lt;ffffffffa05425af&gt;] LNetMEAttach+0xff/0x220 [lnet] 03:15:32: [&lt;ffffffffa0772f12&gt;] ptlrpc_register_rqbd+0x82/0x390 [ptlrpc] 03:15:32: [&lt;ffffffffa0780ec5&gt;] ptlrpc_server_post_idle_rqbds+0x75/0xe0 [ptlrpc] 03:15:32: [&lt;ffffffffa0789dd1&gt;] ptlrpc_main+0xb11/0x1740 [ptlrpc] 03:15:32: [&lt;ffffffffa07892c0&gt;] ? ptlrpc_main+0x0/0x1740 [ptlrpc] 03:15:32: [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 03:15:32: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 03:15:32: [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 03:15:32: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 03:15:32:Code: ff 48 8b 03 eb 92 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 &lt;4c&gt; 8b 42 08 49 89 f5 49 89 d4 49 39 f0 75 27 4d 8b 45 00 4d 39 03:15:32:RIP [&lt;ffffffff812948e7&gt;] __list_add+0x17/0xa0 03:15:32: RSP &lt;ffff88006cb65cd0&gt; 03:15:32:CR2: 0000000000000008 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/fef44202-a551-11e3-a61d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fef44202-a551-11e3-a61d-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1)<br/> Distro/Arch: RHEL6.5/x86_64<br/> MDSCOUNT=2<br/> LU-4731 sanity-lfsck test 1b: Oops: IP: [<ffffffff812948e7>] __list_add+0x17/0xa0 Bug Blocker Resolved Cannot Reproduce WC Triage Jian Yu Fri, 7 Mar 2014 08:20:13 +0000 Thu, 9 Jan 2020 07:31:31 +0000 Thu, 9 Jan 2020 07:31:31 +0000 Lustre 2.5.1 0 5 <p>This is a regression failure occurred on Lustre b2_5 branch. It did not occur before.</p> <p>Here is a patch trying to reproduce the failure: <a href="http://review.whamcloud.com/9548" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9548</a></p> <p>It looks like this is not directly related to LFSCK, since the crash is in LNET. However, it might be indirectly caused by LFSCK if that has a use-after-free or other bad memory access. It definitely looks like a corrupt list pointer (0x000000008) that would be caused by incorrect memory access.</p> <p>Let's see if this is hit again.</p> <p>Close old bug</p> Development Rank 1|hzwh2n: Rank (Obsolete) 13006 Severity