https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1736 Lustre <p>simul, running test #2 - file stat, shared mode<br/> Clients having repeated soft CPU lockup. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>Aug 11 10:45:27 ehyperion686 kernel: BUG: soft lockup - CPU#0 stuck for 67s! <span class="error">&#91;simul:24453&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: BUG: soft lockup - CPU#0 stuck for 67s! <span class="error">&#91;simul:24453&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: Modules linked in: lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core e1000e <span class="error">&#91;last unloaded: cpufreq_ondemand&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: CPU 0<br/> Aug 11 10:45:27 ehyperion686 kernel: Modules linked in: lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm dcdbas i2c_i801 i2c_core ahci iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core e1000e <span class="error">&#91;last unloaded: cpufreq_ondemand&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel:<br/> Aug 11 10:45:27 ehyperion686 kernel: Pid: 24453, comm: simul Not tainted 2.6.32-279.2.1.el6.x86_64 #1 Dell XS23-TY /XS23-TY<br/> Aug 11 10:45:27 ehyperion686 kernel: RIP: 0010:<span class="error">&#91;&lt;ffffffff8150024e&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff8150024e&gt;&#93;</span> _spin_lock+0x1e/0x30<br/> Aug 11 10:45:27 ehyperion686 kernel: RSP: 0018:ffff880028203d90 EFLAGS: 00000297<br/> Aug 11 10:45:27 ehyperion686 kernel: RAX: 000000000000ad35 RBX: ffff880028203d90 RCX: 0000000000001fa4<br/> Aug 11 10:45:27 ehyperion686 kernel: RDX: 000000000000ad34 RSI: 0000000000000000 RDI: ffff8801a60486e8<br/> Aug 11 10:45:27 ehyperion686 kernel: RBP: ffffffff8100bc13 R08: 8000000000000000 R09: 00004f07ef0d6d80<br/> Aug 11 10:45:27 ehyperion686 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880028203d10<br/> Aug 11 10:45:27 ehyperion686 kernel: R13: ffffffff81aa5700 R14: ffff8801a60486e0 R15: ffffffff81505d0b<br/> Aug 11 10:45:27 ehyperion686 kernel: FS: 00002aaaafc522c0(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<br/> Aug 11 10:45:27 ehyperion686 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> Aug 11 10:45:27 ehyperion686 kernel: CR2: 00002aaab10fe044 CR3: 0000000179036000 CR4: 00000000000006f0<br/> Aug 11 10:45:27 ehyperion686 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Aug 11 10:45:27 ehyperion686 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Aug 11 10:45:27 ehyperion686 kernel: Process simul (pid: 24453, threadinfo ffff88033ac38000, task ffff88032e585500)<br/> Aug 11 10:45:27 ehyperion686 kernel: Stack:<br/> Aug 11 10:45:27 ehyperion686 kernel: ffff880028203dc0 ffffffffa026d120 0000000000000230 0000000000000000<br/> Aug 11 10:45:27 ehyperion686 kernel: &lt;d&gt; ffffffff81aa5780 ffff88032dcfb480 ffff880028203e10 ffffffffa05860ca<br/> Aug 11 10:45:27 ehyperion686 kernel: &lt;d&gt; ffff880028203e30 ffffffff810e0d44 0000000000000100 0000000000000008<br/> Aug 11 10:45:27 ehyperion686 kernel: Call Trace:<br/> Aug 11 10:45:27 ehyperion686 kernel: &lt;IRQ&gt;</p> <p>11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa026d120&gt;&#93;</span> ? lprocfs_counter_sub+0xc0/0x100 <span class="error">&#91;lvfs&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05860ca&gt;&#93;</span> ? ldlm_lock_free+0x3a/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff810e0d44&gt;&#93;</span> ? rcu_process_dyntick+0x114/0x120<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04b9e7f&gt;&#93;</span> ? class_handle_free_cb+0x2f/0x110 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff810e0dce&gt;&#93;</span> ? force_quiescent_state+0x7e/0x1a0<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff810e12fd&gt;&#93;</span> ? __rcu_process_callbacks+0x10d/0x330<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff810e154b&gt;&#93;</span> ? rcu_process_callbacks+0x2b/0x50<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81073ec1&gt;&#93;</span> ? __do_softirq+0xc1/0x1e0<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81096c50&gt;&#93;</span> ? hrtimer_interrupt+0x140/0x250<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8100c24c&gt;&#93;</span> ? call_softirq+0x1c/0x30<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8100de85&gt;&#93;</span> ? do_softirq+0x65/0xa0<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81073ca5&gt;&#93;</span> ? irq_exit+0x85/0x90<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81505d10&gt;&#93;</span> ? smp_apic_timer_interrupt+0x70/0x9b<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8100bc13&gt;&#93;</span> ? apic_timer_interrupt+0x13/0x20<br/> Aug 11 10:45:27 ehyperion686 kernel: &lt;EOI&gt;<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04eec60&gt;&#93;</span> ? check_and_discard_cb+0x0/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa026d140&gt;&#93;</span> ? lprocfs_counter_sub+0xe0/0x100 <span class="error">&#91;lvfs&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa026d120&gt;&#93;</span> ? lprocfs_counter_sub+0xc0/0x100 <span class="error">&#91;lvfs&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07dcc81&gt;&#93;</span> ? lov_io_fini+0x211/0x3d0 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04f0548&gt;&#93;</span> ? cl_io_init0+0xa8/0x190 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04f35a6&gt;&#93;</span> ? cl_io_fini+0x76/0x160 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04efdd4&gt;&#93;</span> ? cl_2queue_init+0x34/0xe0 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04eabeb&gt;&#93;</span> ? cl_lock_page_out+0xeb/0x3b0 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04e3686&gt;&#93;</span> ? cl_env_peek+0x36/0x110 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa040a0b7&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x37/0x120 <span class="error">&#91;libcfs&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04eec60&gt;&#93;</span> ? check_and_discard_cb+0x0/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa0779f0f&gt;&#93;</span> ? osc_lock_flush+0x4f/0x90 <span class="error">&#91;osc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa0779faa&gt;&#93;</span> ? osc_lock_cancel+0x5a/0x170 <span class="error">&#91;osc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04e9a65&gt;&#93;</span> ? cl_lock_cancel0+0x75/0x130 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa04ea5ab&gt;&#93;</span> ? cl_lock_cancel+0x11b/0x120 <span class="error">&#91;obdclass&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa077b538&gt;&#93;</span> ? osc_ldlm_blocking_ast+0x148/0x300 <span class="error">&#91;osc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa0588be3&gt;&#93;</span> ? ldlm_cancel_callback+0x63/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05a3421&gt;&#93;</span> ? ldlm_cli_cancel_local+0x71/0x350 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05a66af&gt;&#93;</span> ? ldlm_cli_cancel_list_local+0xef/0x230 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa058db72&gt;&#93;</span> ? ldlm_res_hop_get_locked+0x12/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05a6958&gt;&#93;</span> ? ldlm_cancel_resource_local+0x168/0x280 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa0590044&gt;&#93;</span> ? ldlm_resource_get+0xa4/0x6e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07655ff&gt;&#93;</span> ? osc_destroy+0x11f/0x8f0 <span class="error">&#91;osc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07baa5d&gt;&#93;</span> ? lov_set_add_req+0x2d/0x50 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07bf9c2&gt;&#93;</span> ? lov_prep_destroy_set+0x372/0xa40 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07aac37&gt;&#93;</span> ? lov_destroy+0x357/0xcd0 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05d0465&gt;&#93;</span> ? lustre_msg_buf+0x85/0x90 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07d00c3&gt;&#93;</span> ? lsm_unpackmd_v1+0xf3/0x200 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa07b5474&gt;&#93;</span> ? lov_alloc_memmd+0x54/0x360 <span class="error">&#91;lov&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05fc611&gt;&#93;</span> ? req_capsule_set_size+0x71/0x1b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa085af51&gt;&#93;</span> ? ll_objects_destroy+0x571/0x1a90 <span class="error">&#91;lustre&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05d0465&gt;&#93;</span> ? lustre_msg_buf+0x85/0x90 <span class="error">&#91;ptlrpc&#93;</span></p> <p>Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa05fcc4b&gt;&#93;</span> ? __req_capsule_get+0x14b/0x6b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa0855546&gt;&#93;</span> ? ll_update_times+0x26/0x110 <span class="error">&#91;lustre&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffffa085cf7f&gt;&#93;</span> ? ll_unlink+0x35f/0x780 <span class="error">&#91;lustre&#93;</span><br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81188d3f&gt;&#93;</span> ? vfs_unlink+0x9f/0xe0<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81187a8a&gt;&#93;</span> ? lookup_hash+0x3a/0x50<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8118b273&gt;&#93;</span> ? do_unlinkat+0x183/0x1c0<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff81180786&gt;&#93;</span> ? sys_newstat+0x36/0x50<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8100c5b5&gt;&#93;</span> ? math_state_restore+0x45/0x60<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8118b2c6&gt;&#93;</span> ? sys_unlink+0x16/0x20<br/> Aug 11 10:45:27 ehyperion686 kernel: <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> Aug 11 10:45:27 ehyperion686 kernel: Code: 00 00 00 01 74 05 e8 e2 e3 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 &lt;0f&gt; b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89</p> Hyperion RHEL6 servers/clients LU-1736

simul - clients soft CPU lockup

Bug Blocker Resolved Duplicate Zhenyu Xu Cliff White Sat, 11 Aug 2012 14:15:09 +0000 Mon, 20 Aug 2012 16:32:35 +0000 Sun, 19 Aug 2012 11:22:53 +0000 Lustre 2.1.3 0 4 <p>Oleg</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Bobijam</p> <p>Oleg thought that this issue was in an area of code that you have been working on recently. Can you please treat this as your top priority and look into what has changed between 2.1.2 and 2.1.3 to cause this issue?</p> <p>Thanks</p> <p>Peter</p> <p>we need port <a href="http://review.whamcloud.com/2451" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2451</a>, <a href="http://review.whamcloud.com/3026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3026</a> and <a href="http://review.whamcloud.com/3240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3240</a> to b2_1, I'll push a combined patch.</p> <p>Combined patch <a href="http://review.whamcloud.com/#change,3607" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3607</a></p> <p>This is a bit confusing.</p> <p>What does static allocation of stats structures at mount time have to do with soft-lockups when running simul? If there is a connection, we need a clear explanation.</p> <p>Why are these commits are being squashed into one, which makes comparison of the two branches much more difficult in the future? I am really opposed to this practice. </p> <p>I do see that one of the three commits contains a revert and additional code in a single commit. I would really like to see that avoided in the future as well. Can we have rules to that effect added to the wiki somewhere?</p> <p>In b2_1 branch when we disable percpu stats, lprocfs_counter_sub() uses cfs_spin_lock to protect the stat but an interrupt could happen during it and the interrupt handler could possible call lprocfs_counter_sub() again, causing lockup.</p> <p>there is a glitch in patch #2451 (ls_biggest_alloc_num was not protected) and #3026 (clear stat is not multithread safe), and #3240 (fix the glith and revert the stat clear code), so the 3 patches combined as one is a wholesome one, that's the reason I squashed them into one for b2_1 branch.</p> <p>Closing as duplicate of <a href="https://jira.whamcloud.com/browse/LU-1282" title="Lustre 2.1 client memory usage at mount is excessive" class="issue-link" data-issue-key="LU-1282"><del>LU-1282</del></a></p> <p>Actually I sort of think when we port stuff from current branches back into stable branches, it's better to merge whatever fixes were made on top of needed patches into one. Makes for a cleaner changelog and we pull in known good (combined) patches instead of the whole experimental stream of "try X, X did not work, revert - try Y, Y did not work, try Z, ok it works, here are some final adjustments".<br/> Code ends up to be the same as in more new branches, though.</p> <p>But it makes it much harder to compare the commits on multiple branches. I want to be able to look at two branches with "git log --oneline" and have some hope of figuring out if a change is in both branches. If we're always squashing things together it makes for a not-very-consistent log comparison.</p> Development Rank 1|hzvgof: Rank (Obsolete) 6351 Severity