https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6439 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/96df30ce-dccc-11e4-a6e6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/96df30ce-dccc-11e4-a6e6-5254006e85c2</a>.</p> <p>from evidence in console log it looks like the client node panic'ed and even took a crash dump:</p> <p>17:25:50:Saving to remote location onyx-4.onyx.hpdd.intel.com:/export/scratch/dumps<br/> 17:25:50:Saving vmcore-dmesg.txt<br/> 17:25:50:Saved vmcore-dmesg.txt</p> <p>This seems like a one off and possibly a TEI issue. Other test runs of patches depending on the one in this test run completed without any problems.</p> <p>The sub-test test_120g failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity 120g</p> LU-6439

sanity test_120g: panic on client

Bug Major Resolved Fixed Lai Siyao Maloo Tue, 7 Apr 2015 15:30:36 +0000 Mon, 26 Mar 2018 18:16:48 +0000 Tue, 6 Feb 2018 04:59:02 +0000 0 3 <p>found the saved crash dump. in the vmcore-dmesg.txt I see the following:</p> <p>&lt;0&gt;BUG: soft lockup - CPU#0 stuck for 67s! <span class="error">&#91;ptlrpcd_0:2379&#93;</span><br/> &lt;4&gt;Modules linked in: ext2 lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: speedstep_lib&#93;</span><br/> &lt;4&gt;CPU 0 <br/> &lt;4&gt;Modules linked in: ext2 lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U)<br/> &lt;0&gt;BUG: soft lockup - CPU#1 stuck for 67s! <span class="error">&#91;ll_sa_28008:28009&#93;</span><br/> &lt;4&gt;Modules linked in: ext2 lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: speedstep_lib&#93;</span><br/> &lt;4&gt;CPU 1 <br/> &lt;4&gt;Modules linked in: ext2 lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: speedstep_lib&#93;</span><br/> &lt;4&gt;<br/> &lt;4&gt;Pid: 28009, comm: ll_sa_28008 Not tainted 2.6.32-431.29.2.el6.x86_64 #1 Red Hat KVM<br/> &lt;4&gt;RIP: 0010:<span class="error">&#91;&lt;ffffffff8152b84e&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff8152b84e&gt;&#93;</span> _spin_lock+0x1e/0x30<br/> &lt;4&gt;RSP: 0018:ffff88006f26fd30 EFLAGS: 00000206<br/> &lt;4&gt;RAX: 0000000000000001 RBX: ffff88006f26fd30 RCX: 0000000000000003<br/> &lt;4&gt;RDX: 0000000000000000 RSI: 000000001082ebea RDI: ffff88006a1e2440<br/> &lt;4&gt;RBP: ffffffff8100bb8e R08: 0000000031353433 R09: 0000000000000000<br/> &lt;4&gt;R10: ffff880067eb68c0 R11: 0000000000000080 R12: 0000000000000000<br/> &lt;4&gt;R13: 0000000000000eef R14: 0000000200001b71 R15: 0000000000000000<br/> &lt;4&gt;FS: 0000000000000000(0000) GS:ffff880002300000(0000) knlGS:0000000000000000<br/> &lt;4&gt;CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br/> &lt;4&gt;CR2: 0000000001bb80b8 CR3: 000000007d793000 CR4: 00000000000006e0<br/> &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> &lt;4&gt;Process ll_sa_28008 (pid: 28009, threadinfo ffff88006f26e000, task ffff88007abec080)<br/> &lt;4&gt;Stack:<br/> &lt;4&gt; ffff88006f26fdc0 ffffffffa0b6a275 0000000000000005 0000000000000080<br/> &lt;4&gt;&lt;d&gt; 0000000000001a77 ffff88006cb14080 ffff88006a1e2178 ffff88005fc241c8<br/> &lt;4&gt;&lt;d&gt; 0000000000000000 0000000000000000 ffff88007a38ab00 ffff88006b218200<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6a275&gt;&#93;</span> ? ll_statahead_one+0x295/0xdc0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6b11b&gt;&#93;</span> ? ll_statahead_thread+0x37b/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6ada0&gt;&#93;</span> ? ll_statahead_thread+0x0/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109abf6&gt;&#93;</span> ? kthread+0x96/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109ab60&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> &lt;4&gt;Code: 00 00 00 01 74 05 e8 92 3a d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 &lt;0f&gt; 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 <br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6a275&gt;&#93;</span> ? ll_statahead_one+0x295/0xdc0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6b11b&gt;&#93;</span> ? ll_statahead_thread+0x37b/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6ada0&gt;&#93;</span> ? ll_statahead_thread+0x0/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109abf6&gt;&#93;</span> ? kthread+0x96/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109ab60&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> &lt;0&gt;Kernel panic - not syncing: softlockup: hung tasks<br/> &lt;4&gt;Pid: 28009, comm: ll_sa_28008 Not tainted 2.6.32-431.29.2.el6.x86_64 #1<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; &lt;IRQ&gt; <span class="error">&#91;&lt;ffffffff8152873c&gt;&#93;</span> ? panic+0xa7/0x16f<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810e6200&gt;&#93;</span> ? watchdog_timer_fn+0x0/0x1e0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810e63ca&gt;&#93;</span> ? watchdog_timer_fn+0x1ca/0x1e0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109f6be&gt;&#93;</span> ? __run_hrtimer+0x8e/0x1a0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a6a9f&gt;&#93;</span> ? ktime_get_update_offsets+0x4f/0xd0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109fa26&gt;&#93;</span> ? hrtimer_interrupt+0xe6/0x260<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81031f1d&gt;&#93;</span> ? local_apic_timer_interrupt+0x3d/0x70<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff815325e5&gt;&#93;</span> ? smp_apic_timer_interrupt+0x45/0x60<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100bb93&gt;&#93;</span> ? apic_timer_interrupt+0x13/0x20<br/> &lt;4&gt; &lt;EOI&gt; <span class="error">&#91;&lt;ffffffff8152b84e&gt;&#93;</span> ? _spin_lock+0x1e/0x30<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6a275&gt;&#93;</span> ? ll_statahead_one+0x295/0xdc0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6b11b&gt;&#93;</span> ? ll_statahead_thread+0x37b/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0b6ada0&gt;&#93;</span> ? ll_statahead_thread+0x0/0xfb0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109abf6&gt;&#93;</span> ? kthread+0x96/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8109ab60&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>so it looks like a statahead issue.</p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-4410" title="sanityn test 40a: BUG: soft lockup - CPU#0 stuck for 67s! [ptlrpcd_0:2892]" class="issue-link" data-issue-key="LU-4410"><del>LU-4410</del></a>?</p> <p>this looks to be a dup of <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a>, there is a backport patch for 2.5 <a href="http://review.whamcloud.com/#/c/12901/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12901/</a>, which should be able to fix this.</p> <p>Landed for 2.11</p> Duplicate LU-3270 Development Rank 1|hzxaaf: Rank (Obsolete) 9223372036854775807 Severity