https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4410 Lustre <p>sanityn test 40a hung and hit the following failure on one client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>21:36:52:Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ================ 21:34:49 (1387604089) 21:36:53:BUG: soft lockup - CPU#0 stuck for 67s! [ptlrpcd_0:2892] 21:36:53:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode 8139too 8139cp mii virtio_balloon i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] 21:36:53:CPU 0 21:36:53:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) 21:36:53:BUG: soft lockup - CPU#1 stuck for 67s! [ll_sa_4070:4079] 21:36:53:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode 8139too 8139cp mii virtio_balloon i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] 21:36:53:CPU 1 21:36:53:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode 8139too 8139cp mii virtio_balloon i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] 21:36:53: 21:36:53:Pid: 4079, comm: ll_sa_4070 Not tainted 2.6.32-358.23.2.el6.x86_64 #1 Red Hat KVM 21:36:53:RIP: 0010:[&lt;ffffffff81510aae&gt;] [&lt;ffffffff81510aae&gt;] _spin_lock+0x1e/0x30 21:36:53:RSP: 0018:ffff88006c26bda0 EFLAGS: 00000206 21:36:53:RAX: 0000000000000002 RBX: ffff88006c26bda0 RCX: ffff88007cfd8800 21:36:54:RDX: 0000000000000000 RSI: ffff88006c25fec0 RDI: ffff88007a737ec0 21:36:54:RBP: ffffffff8100bb8e R08: ffff88007d860e68 R09: 00000000fffffffe 21:36:54:R10: 0000000000000000 R11: 0000000000000001 R12: ffff88006c26bd80 21:36:54:R13: ffff88006d6c9000 R14: 0000000000001000 R15: 0000000000000000 21:36:54:FS: 00007fb227702700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 21:36:54:CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 21:36:54:CR2: 00007f7bbff64000 CR3: 000000006c183000 CR4: 00000000000006e0 21:36:54:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 21:36:54:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 21:36:54:Process ll_sa_4070 (pid: 4079, threadinfo ffff88006c26a000, task ffff88006bd25500) 21:36:54:Stack: 21:36:54: ffff88006c26be10 ffffffffa0abb680 ffff88007a737bf8 ffff88006e9501c8 21:36:54:&lt;d&gt; 0000000000000000 ffff88007a737b00 ffff88007caa01c0 ffff88006bf57200 21:36:54:&lt;d&gt; ffff88006c26bdf0 ffff88007a7ba800 ffff88007a7ba970 ffff88007a737e80 21:36:54:Call Trace: 21:36:54: [&lt;ffffffffa0abb680&gt;] ? ll_post_statahead+0x50/0xa80 [lustre] 21:36:55: [&lt;ffffffffa0abf8c8&gt;] ? ll_statahead_thread+0x268/0xfa0 [lustre] 21:36:55: [&lt;ffffffff81063990&gt;] ? default_wake_function+0x0/0x20 21:36:55: [&lt;ffffffffa0abf660&gt;] ? ll_statahead_thread+0x0/0xfa0 [lustre] 21:36:55: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 21:36:55: [&lt;ffffffffa0abf660&gt;] ? ll_statahead_thread+0x0/0xfa0 [lustre] 21:36:55: [&lt;ffffffffa0abf660&gt;] ? ll_statahead_thread+0x0/0xfa0 [lustre] 21:36:55: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/7cca784a-6b4b-11e3-99ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7cca784a-6b4b-11e3-99ba-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs<br/> LU-4410

sanityn test 40a: BUG: soft lockup - CPU#0 stuck for 67s! [ptlrpcd_0:2892]

Bug Blocker Resolved Fixed WC Triage Jian Yu 22pl Mon, 23 Dec 2013 06:31:40 +0000 Sun, 10 Oct 2021 21:09:03 +0000 Sun, 10 Oct 2021 21:09:03 +0000 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.2 Lustre 2.5.3 0 9 <p>Here is the query result on Maloo for sanityn test 40a timeout failure on Lustre b2_4 branch:<br/> <a href="http://tinyurl.com/ovmwgad" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/ovmwgad</a></p> <p>The failure did not occur on previous Lustre b2_4 builds.</p> <p>Just ran the same test with FSTYPE=zfs on Lustre 2.4.2 RC2 for 10 times. All runs passed. So, it seems like this is an occasional failure.</p> <p>I hit this problem while testing a b2_5 patch and lustre-rsync_test test_6 timedout. Logs are at <a href="https://maloo.whamcloud.com/test_sets/0712c6e6-e762-11e3-b2f3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0712c6e6-e762-11e3-b2f3-52540035b04c</a></p> <p>Note: This was run on ldiskfs, not ZFS</p> <p>More instance on Lustre b2_5 branch while running lustre-rsync-test test 6 with FSTYPE=ldiskfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/b8b57494-ec5c-11e3-98b3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b8b57494-ec5c-11e3-98b3-52540035b04c</a></p> <p>Hi Nasf,</p> <p>It looks like the failure is related to statahead. It originally occurred on Lustre b2_4 branch with ZFS on sanityn test 40a, and now frequently occurs on Lustre b2_5 branch with ldiskfs on lustre-rsync-test test 6. Could you please help take a look if these two test failures have the same root cause? Thanks.</p> <p>It seems that some thread was blocked with the ll_inode_info::lli_sa_lock held. There is some known bug for it. Here is the patch:</p> <p><a href="http://review.whamcloud.com/#/c/9665/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9665/</a></p> <p>Would you please to try the patch? Thanks!</p> <blockquote><p>Would you please to try the patch?</p></blockquote> <p>Sure, I'll do this. Thank you!</p> <p>Here is the patch back-ported to Lustre b2_5 branch: <a href="http://review.whamcloud.com/10674" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10674</a></p> <p>Hmm, I saw similar problem when I run my patch <a href="http://review.whamcloud.com/#/c/10622/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10622/</a> on master. <a href="https://maloo.whamcloud.com/test_sets/196f5da8-f2d5-11e3-b88b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/196f5da8-f2d5-11e3-b88b-52540035b04c</a></p> <p>Is this also needed on master?</p> <blockquote><p>Here is the patch back-ported to Lustre b2_5 branch: <a href="http://review.whamcloud.com/10674" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10674</a></p></blockquote> <p>The patch was reverted from Lustre b2_5 branch because we need wait until master version is fully ready.</p> <p>Another sanityn test 40a failure instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/17d0d2b0-f5fa-11e3-9d30-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/17d0d2b0-f5fa-11e3-9d30-52540035b04c</a></p> <p>Another lustre-rsync-test test_6 on master branch review-dne-part-1:<br/> <a href="https://testing.hpdd.intel.com/test_sets/5f96c5ec-094a-11e4-b76d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5f96c5ec-094a-11e4-b76d-5254006e85c2</a></p> <p>lustre-rsync-test 6, master, zfs, single MDT: <a href="https://testing.hpdd.intel.com/test_sets/67240e86-1cf6-11e4-9a83-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/67240e86-1cf6-11e4-9a83-5254006e85c2</a></p> <p>Another failure instance:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/6d60501e-1dbb-11e4-8fe8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/6d60501e-1dbb-11e4-8fe8-5254006e85c2</a></p> <p>While verifying patch <a href="http://review.whamcloud.com/11615" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11615</a> with FSTYPE=zfs on Lustre b2_5 branch, lustre-rsync-test hit the same failure:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8cc59532-30ac-11e4-9e60-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8cc59532-30ac-11e4-9e60-5254006e85c2</a></p> <p>Here is the back-ported patch for Lustre b2_5 branch: <a href="http://review.whamcloud.com/13846" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13846</a></p> Duplicate LU-5605 LU-6270 Development Rank 1|hzwbrz: Rank (Obsolete) 12104 Severity