https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2505 Lustre <p>lfsck test hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; creating /mnt/lustre/d0.lfsck/testfile.30 creating files in /mnt/lustre/d0.lfsck/d9 creating files in /mnt/lustre/d0.lfsck/d10 creating files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 linking files in /mnt/lustre/d0.lfsck/d11 CMD: client-26vm8 lctl get_param -n obdfilter.*.uuid CMD: client-26vm8 lctl get_param -n obdfilter.lustre-OST0000.mntdev Stopping clients: client-26vm1,client-26vm2.lab.whamcloud.com /mnt/lustre (opts:-f) </pre> </div></div> <p>Console log on client-26vm2 showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 4267:0:(llite_lib.c:2290:ll_dirty_page_discard_warn()) dirty page discard: 10.10.4.154@tcp:/lustre/fid: [0x200002b13:0x1224:0x0]/ may get corrupted (rc -108) Lustre: 4267:0:(llite_lib.c:2290:ll_dirty_page_discard_warn()) dirty page discard: 10.10.4.154@tcp:/lustre/fid: [0x200002b13:0x1237:0x0]/ may get corrupted (rc -108) Lustre: 22194:0:(cl_lock.c:2026:cl_lock_page_out()) Writing 1 pages error: -108 BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[&lt;ffffffff8150098e&gt;] [&lt;ffffffff8150098e&gt;] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 &lt;d&gt; ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 &lt;d&gt; ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [&lt;ffffffffa08ee29c&gt;] ? osc_io_submit+0x31c/0x540 [osc] [&lt;ffffffffa081c148&gt;] ? cl_io_submit_rw+0x78/0x180 [obdclass] [&lt;ffffffffa0958ad5&gt;] ? lov_io_submit+0x355/0xc30 [lov] [&lt;ffffffffa0813c65&gt;] ? cl_page_put+0x1e5/0x580 [obdclass] [&lt;ffffffffa081680a&gt;] ? pgoff_at_lock+0x1a/0x70 [obdclass] [&lt;ffffffffa081c148&gt;] ? cl_io_submit_rw+0x78/0x180 [obdclass] [&lt;ffffffffa081f29f&gt;] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [&lt;ffffffffa08166c7&gt;] ? cl_lock_page_out+0x207/0x330 [obdclass] [&lt;ffffffffa0816860&gt;] ? pageout_cb+0x0/0x100 [obdclass] [&lt;ffffffffa08ea99f&gt;] ? osc_lock_flush+0x4f/0x90 [osc] [&lt;ffffffffa08eaaaf&gt;] ? osc_lock_cancel+0xcf/0x1b0 [osc] [&lt;ffffffffa08152a5&gt;] ? cl_lock_cancel0+0x75/0x160 [obdclass] [&lt;ffffffffa0815f8b&gt;] ? cl_lock_cancel+0x13b/0x140 [obdclass] [&lt;ffffffffa08ebb8a&gt;] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [&lt;ffffffffa0420cc0&gt;] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [&lt;ffffffffa043b14b&gt;] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [&lt;ffffffffa043efd8&gt;] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [&lt;ffffffffa0428db8&gt;] ? cleanup_resource+0x168/0x300 [ptlrpc] [&lt;ffffffffa06baefe&gt;] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [&lt;ffffffffa0428f7a&gt;] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [&lt;ffffffffa06bb18f&gt;] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [&lt;ffffffffa0428f50&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa0428f50&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa06bcbaf&gt;] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [&lt;ffffffffa0425a49&gt;] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [&lt;ffffffffa08e2122&gt;] ? osc_import_event+0x3d2/0x1470 [osc] [&lt;ffffffffa0487c59&gt;] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [&lt;ffffffffa08e283c&gt;] ? osc_import_event+0xaec/0x1470 [osc] [&lt;ffffffffa045f53d&gt;] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [&lt;ffffffffa08dd070&gt;] ? osc_iocontrol+0xc80/0x11b0 [osc] [&lt;ffffffff8127cf26&gt;] ? vsnprintf+0x2b6/0x5f0 [&lt;ffffffffa092f7ff&gt;] ? lov_iocontrol+0x27f/0x1e90 [lov] [&lt;ffffffffa064eadf&gt;] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [&lt;ffffffff81188961&gt;] ? path_put+0x31/0x40 [&lt;ffffffff81188325&gt;] ? putname+0x35/0x50 [&lt;ffffffffa09dd640&gt;] ? ll_umount_begin+0x2c0/0xc30 [lustre] [&lt;ffffffff8119b52a&gt;] ? sys_umount+0x11a/0x3a0 [&lt;ffffffff810d6d42&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff81503ade&gt;] ? do_page_fault+0x3e/0xa0 [&lt;ffffffff8100b0f2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>The full console log is attached.</p> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/679bdff0-4828-11e2-a406-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/679bdff0-4828-11e2-a406-52540035b04c</a></p> Lustre Tag: v2_1_4_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/159/">http://build.whamcloud.com/job/lustre-b2_1/159/</a><br/> Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32_279.14.1.el6)<br/> LU-2505

lfsck: BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194]

Bug Blocker Resolved Fixed Jinshan Xiong Jian Yu Tue, 18 Dec 2012 02:00:06 +0000 Sat, 22 Dec 2012 13:41:38 +0000 Sat, 22 Dec 2012 13:41:38 +0000 Lustre 2.4.0 Lustre 2.1.4 Lustre 2.4.0 Lustre 2.1.4 0 5 <p>RHEL6.3/x86_64 (2.3.0 Server + 2.1.4 RC1 Client):<br/> <a href="https://maloo.whamcloud.com/test_sets/29296da0-486e-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/29296da0-486e-11e2-8cdc-52540035b04c</a></p> <p>Bob is looking into this</p> <p>I reproduced this issue manually on Toro client-12vm<span class="error">&#91;1,3,4&#93;</span> against Lustre v2_1_4_RC1 and got the kernel dump file of client-12vm1.</p> <p>Client: client-12vm1<br/> MGS/MDS: client-12vm3<br/> OSS: client-12vm4</p> <p>The dump file is /scratch/logs/2.1.4/<a href="https://jira.whamcloud.com/browse/LU-2505" title="lfsck: BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194]" class="issue-link" data-issue-key="LU-2505"><del>LU-2505</del></a>/client-12vm1.dump on brent node.</p> <p>I also put the kernel-debuginfo-* packages into that directory.</p> <p>The console log client-12vm1 is attached.</p> <p>This should be the same problem of <a href="https://jira.whamcloud.com/browse/LU-2451" title="recovery-small test_24b: BUG: soft lockup - CPU#0 stuck for 67s! [ll_imp_inval:4791]" class="issue-link" data-issue-key="LU-2451"><del>LU-2451</del></a> because it also spin on client obd list lock. The patch is at: <a href="http://review.whamcloud.com/4806" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4806</a></p> <p>yujian: what's the build # you're running?</p> <blockquote><p>yujian: what's the build # you're running?</p></blockquote> <p>Lustre v2_1_4_RC1: <a href="http://build.whamcloud.com/job/lustre-b2_1/159/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/159/</a></p> <p>I'm re-provisioning the test nodes to verify <a href="http://review.whamcloud.com/4806" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4806</a>.</p> <p>I took a look at the kernel dump - thank you for the dump file, yujian.</p> <p>Apparently the problem is in ll_dirty_page_discard_warn() where it allocates a page with GFP_KERNEL. And this function can be called inside client obd list lock as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 10150 TASK: ffff88007d11f540 CPU: 0 COMMAND: <span class="code-quote">"ptlrpcd-brw"</span> #0 [ffff88007d075840] schedule at ffffffff814fe052 #1 [ffff88007d075908] __cond_resched at ffffffff8106244a #2 [ffff88007d075928] _cond_resched at ffffffff814fea60 #3 [ffff88007d075938] __alloc_pages_nodemask at ffffffff811272e5 #4 [ffff88007d075a58] alloc_pages_current at ffffffff8115c7ea #5 [ffff88007d075a88] __get_free_pages at ffffffff8112572e #6 [ffff88007d075a98] ll_dirty_page_discard_warn at ffffffffa0a3d252 [lustre] #7 [ffff88007d075b08] vvp_page_completion_common at ffffffffa0a7adfc [lustre] #8 [ffff88007d075b38] vvp_page_completion_write_common at ffffffffa0a7ae6b [lustre] #9 [ffff88007d075b68] vvp_page_completion_write at ffffffffa0a7b83e [lustre] #10 [ffff88007d075ba8] cl_page_completion at ffffffffa05eed8f [obdclass] #11 [ffff88007d075c48] osc_completion at ffffffffa0880812 [osc] #12 [ffff88007d075c98] osc_ap_completion at ffffffffa086a544 [osc] #13 [ffff88007d075cf8] brw_interpret at ffffffffa0876d69 [osc] #14 [ffff88007d075d98] ptlrpc_check_set at ffffffffa0736d5b [ptlrpc] #15 [ffff88007d075e38] ptlrpcd_check at ffffffffa0767fe0 [ptlrpc] #16 [ffff88007d075e68] ptlrpcd at ffffffffa076827e [ptlrpc] #17 [ffff88007d075f48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>The fix is pretty simple, use GFP_ATOMIC instead.</p> <p>patch is at: <a href="http://review.whamcloud.com/4866" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4866</a></p> <p>Jinshan</p> <p>Does this issue also affect master?</p> <p>Peter</p> <p>This is caused by <a href="http://review.whamcloud.com/1908" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1908</a> from <a href="https://jira.whamcloud.com/browse/LU-921" title="generate warnings in case of discarding dirty pages" class="issue-link" data-issue-key="LU-921"><del>LU-921</del></a>.</p> <p>patch for master is at: <a href="http://review.whamcloud.com/4870" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4870</a></p> <p>Jinshan has a fix for this</p> <p>Landed for 2.1.4 and 2.4</p> Duplicate LU-2451 Related LU-921 Development Rank 1|hzvdyv: Rank (Obsolete) 5875 Severity