https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1138 Lustre <p>Customer reports that a few compute nodes have been panic'ing. They have seen the behavior on 7 nodes. Each node has seen the problem numerous times. It looks like it may be similar to <a href="https://jira.whamcloud.com/browse/LU-93" title="Kernel panic caused by Lustre 1.8.5/1.8.6 on RH6 (patchless client)" class="issue-link" data-issue-key="LU-93"><del>LU-93</del></a>. I'd like to get Whamcloud to weigh in on whether you think it is related or if it is a known issue. The trace backs and console messages are attached.</p> RHEL 6 2.6.32-71.el6.x86_64 kernel LU-1138

Client Panic on Lustre 1.8.6 and RHEL 6

Bug Major Resolved Cannot Reproduce Zhenyu Xu Dennis Nelson Fri, 24 Feb 2012 15:15:17 +0000 Mon, 22 Jan 2024 23:01:19 +0000 Tue, 24 Apr 2012 19:19:50 +0000 Lustre 1.8.6 0 2 <p>Customer just asked me to bump up the priority on this one. They just reported that this issue has caused hundreds of nodes to become unresponsive on their system.</p> <p>This doesn't appear to be the same as <a href="https://jira.whamcloud.com/browse/LU-93" title="Kernel panic caused by Lustre 1.8.5/1.8.6 on RH6 (patchless client)" class="issue-link" data-issue-key="LU-93"><del>LU-93</del></a>, which was causing the client to crash.</p> <p>In this case, it looks like all of the threads are stuck in ll_teardown_mmaps-&gt;unmap_mapping_range() because the node is trying to free memory under memory pressure.</p> <p>This is a somewhat unusual workload for Lustre, because while mmap IO is functional, it is quite inefficient (single page RPCs) and rarely used.</p> <p>Has this application been running in the past on Lustre? Are there any changes in the environment that might have caused the application to start failing (e.g. kernel, Lustre, or application upgrade)?</p> <p>I received the following from the customer today:</p> <p>Please ask WC to stand down on it being P1. We found in a sample that there was<br/> lustre so we went with that. Once we started looking at all of the traces lustre is present it SOME of<br/> the stack traces but it is not in the most common. I would appreciate if Andreas<br/> can have a look at some more strack traces to see if there is anything<br/> he's seen before though.</p> <p><a href="ftp://shell.sgi.com/collect/jhanson/nodeswithsoftlockupconsoles.tar.bz2" class="external-link" target="_blank" rel="nofollow noopener">ftp://shell.sgi.com/collect/jhanson/nodeswithsoftlockupconsoles.tar.bz2</a></p> <p>What I've found by looking at these</p> <p>Once there is a BUG: soft lockup the next lines are like this (example chosen at random)</p> <p>BUG: soft lockup - CPU#0 stuck for 61s! <span class="error">&#91;global_fcst:30024&#93;</span><br/> Modules linked in: acpi_cpufreq freq_table mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad mlx4_ib iw_cxgb3 ko2iblnd(U) rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr obdclass(U) lnet(U) lvfs(U) libcfs(U) xpmem(U) xp gru xvma(U) numatools(U) microcode serio_raw i2c_i801 i2c_core iTCO_wdt<br/> iTCO_vendor_support ioatdma ahci mlx4_en mlx4_core igb dca dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache nfs_acl auth_rpcgss sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi qla4xxx scsi_transport_iscsi <span class="error">&#91;last unloaded: ipmi_msghandler&#93;</span><br/> CPU 0:<br/> Modules linked in: acpi_cpufreq freq_table mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad mlx4_ib iw_cxgb3 ko2iblnd(U) rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr obdclass(U) lnet(U) lvfs(U) libcfs(U) xpmem(U) xp gru xvma(U) numatools(U) microcode serio_raw i2c_i801 i2c_core iTCO_wdt<br/> iTCO_vendor_support ioatdma ahci mlx4_en mlx4_core igb dca dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache nfs_acl auth_rpcgss sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi qla4xxx scsi_transport_iscsi <span class="error">&#91;last unloaded: ipmi_msghandler&#93;</span><br/> Pid: 30024, comm: global_fcst Tainted: G W ---------------- 2.6.32-71.el6.x86_64 #1 AltixICE8400IP105<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffff814caa3e&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff814caa3e&gt;&#93;</span> _spin_lock+0x1e/0x30<br/> RSP: 0018:ffff8802e9b3fc38 EFLAGS: 00000297<br/> RAX: 000000000000e364 RBX: ffff8802e9b3fc38 RCX: ffff8804b764de80<br/> RDX: 0000000000000000 RSI: ffff88033d53d208 RDI: ffff880637837268<br/> RBP: ffffffff81013c8e R08: ffff8802e9b3fe10 R09: 0000000000100000<br/> R10: 00007fffffff2dc0 R11: 0000000000000213 R12: ffff88033b712100<br/> R13: ffffffff817300c0 R14: ffff88033b7126b8 R15: 0000000000010518<br/> FS: 00002aaaaf3e0800(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 00002aaaae8f0840 CR3: 000000033ca90000 CR4: 00000000000006f0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0304fb1&gt;&#93;</span> ? xpmem_tg_ref_by_tgid+0x41/0xe0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81068598&gt;&#93;</span> ? get_task_mm+0x28/0x70<br/> <span class="error">&#91;&lt;ffffffffa030073a&gt;&#93;</span> ? xpmem_make+0x9a/0x360 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8110c037&gt;&#93;</span> ? __lock_page+0x67/0x70<br/> <span class="error">&#91;&lt;ffffffffa02ff19d&gt;&#93;</span> ? xpmem_ioctl+0xdd/0x3f0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8110dade&gt;&#93;</span> ? filemap_fault+0xbe/0x510<br/> <span class="error">&#91;&lt;ffffffff8110c177&gt;&#93;</span> ? unlock_page+0x27/0x30<br/> <span class="error">&#91;&lt;ffffffff81135837&gt;&#93;</span> ? handle_pte_fault+0xf7/0xad0<br/> <span class="error">&#91;&lt;ffffffff811502a7&gt;&#93;</span> ? alloc_pages_current+0x87/0xd0<br/> <span class="error">&#91;&lt;ffffffff8117f182&gt;&#93;</span> ? vfs_ioctl+0x22/0xa0<br/> <span class="error">&#91;&lt;ffffffff81258ae5&gt;&#93;</span> ? _atomic_dec_and_lock+0x55/0x80<br/> <span class="error">&#91;&lt;ffffffff81013c8e&gt;&#93;</span> ? apic_timer_interrupt+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff8117f324&gt;&#93;</span> ? do_vfs_ioctl+0x84/0x580<br/> <span class="error">&#91;&lt;ffffffff811363fd&gt;&#93;</span> ? handle_mm_fault+0x1ed/0x2b0<br/> <span class="error">&#91;&lt;ffffffff8117f8a1&gt;&#93;</span> ? sys_ioctl+0x81/0xa0<br/> <span class="error">&#91;&lt;ffffffff81013172&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> <p>So I went to look for commonality after Call Trace: and found little with lots<br/> of possible places to check.</p> <p>guest@globe:/cores/people/jhanson/noaa/softlockup/nodeswithsoftlockupconsoles&gt; grep --binary-files=text -h -A1 "Call Trace" r* | sort | uniq<br/> &#8211;<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff810117bc&gt;&#93;</span> ? __switch_to+0x1ac/0x320<br/> <span class="error">&#91;&lt;ffffffff81013ace&gt;&#93;</span> ? common_interrupt+0xe/0x13<br/> <span class="error">&#91;&lt;ffffffff81013b76&gt;&#93;</span> retint_careful+0x14/0x32<br/> <span class="error">&#91;&lt;ffffffff81013c8e&gt;&#93;</span> ? apic_timer_interrupt+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff81013cee&gt;&#93;</span> ? invalidate_interrupt1+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff81013d4e&gt;&#93;</span> ? invalidate_interrupt4+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff81013d6e&gt;&#93;</span> ? invalidate_interrupt5+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff81014162&gt;&#93;</span> ? kernel_thread+0x82/0xe0<br/> <span class="error">&#91;&lt;ffffffff81014645&gt;&#93;</span> ? math_state_restore+0x45/0x60<br/> <span class="error">&#91;&lt;ffffffff8101660f&gt;&#93;</span> ? dump_trace+0x1af/0x3a0<br/> <span class="error">&#91;&lt;ffffffff8101a4f9&gt;&#93;</span> ? read_tsc+0x9/0x20<br/> <span class="error">&#91;&lt;ffffffff8104f61c&gt;&#93;</span> ? enqueue_task+0x5c/0x70<br/> <span class="error">&#91;&lt;ffffffff8104fff9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> <span class="error">&#91;&lt;ffffffff810507f8&gt;&#93;</span> ? resched_task+0x68/0x80<br/> <span class="error">&#91;&lt;ffffffff810508a5&gt;&#93;</span> ? check_preempt_curr_idle+0x15/0x20<br/> <span class="error">&#91;&lt;ffffffff81056303&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff81056630&gt;&#93;</span> ? __dequeue_entity+0x30/0x50<br/> <span class="error">&#91;&lt;ffffffff81059d12&gt;&#93;</span> ? finish_task_switch+0x42/0xd0<br/> <span class="error">&#91;&lt;ffffffff8105a808&gt;&#93;</span> ? pull_task+0x58/0x70<br/> <span class="error">&#91;&lt;ffffffff8105c490&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffff8105c4a2&gt;&#93;</span> ? default_wake_function+0x12/0x20<br/> <span class="error">&#91;&lt;ffffffff8105c4e5&gt;&#93;</span> ? wake_up_process+0x15/0x20<br/> <span class="error">&#91;&lt;ffffffff8105c756&gt;&#93;</span> ? update_curr+0xe6/0x1e0<br/> <span class="error">&#91;&lt;ffffffff8105fa72&gt;&#93;</span> ? enqueue_entity+0x122/0x320<br/> <span class="error">&#91;&lt;ffffffff8105fcb3&gt;&#93;</span> ? enqueue_task_fair+0x43/0x90<br/> <span class="error">&#91;&lt;ffffffff81061b71&gt;&#93;</span> ? dequeue_entity+0x1a1/0x1e0<br/> <span class="error">&#91;&lt;ffffffff81062b84&gt;&#93;</span> ? find_busiest_group+0x254/0xb40<br/> <span class="error">&#91;&lt;ffffffff8106329a&gt;&#93;</span> ? find_busiest_group+0x96a/0xb40<br/> <span class="error">&#91;&lt;ffffffff81066d6e&gt;&#93;</span> ? select_task_rq_fair+0x9ee/0xab0<br/> <span class="error">&#91;&lt;ffffffff810670c1&gt;&#93;</span> ? check_preempt_wakeup+0x41/0x3c0<br/> <span class="error">&#91;&lt;ffffffff81067244&gt;&#93;</span> ? check_preempt_wakeup+0x1c4/0x3c0<br/> <span class="error">&#91;&lt;ffffffff81067732&gt;&#93;</span> migration_thread+0x1d2/0x310<br/> <span class="error">&#91;&lt;ffffffff81069207&gt;&#93;</span> ? dup_mm+0x2a7/0x520<br/> <span class="error">&#91;&lt;ffffffff8106b857&gt;&#93;</span> warn_slowpath_common+0x87/0xc0<br/> <span class="error">&#91;&lt;ffffffff8106b9f5&gt;&#93;</span> ? __call_console_drivers+0x75/0x90<br/> <span class="error">&#91;&lt;ffffffff8106d0a1&gt;&#93;</span> do_syslog+0x461/0x4c0<br/> <span class="error">&#91;&lt;ffffffff8106f805&gt;&#93;</span> do_wait+0x1c5/0x250<br/> <span class="error">&#91;&lt;ffffffff8107064f&gt;&#93;</span> do_exit+0x56f/0x820<br/> <span class="error">&#91;&lt;ffffffff810737a5&gt;&#93;</span> ksoftirqd+0xd5/0x110<br/> <span class="error">&#91;&lt;ffffffff8107d5ac&gt;&#93;</span> ? lock_timer_base+0x3c/0x70<br/> <span class="error">&#91;&lt;ffffffff8107e616&gt;&#93;</span> ? mod_timer+0x146/0x230<br/> <span class="error">&#91;&lt;ffffffff8107e718&gt;&#93;</span> ? add_timer+0x18/0x30<br/> <span class="error">&#91;&lt;ffffffff8108ac20&gt;&#93;</span> ? __call_usermodehelper+0x0/0xa0<br/> <span class="error">&#91;&lt;ffffffff8108c4a0&gt;&#93;</span> ? worker_thread+0x0/0x2a0<br/> <span class="error">&#91;&lt;ffffffff8108cc82&gt;&#93;</span> ? queue_work_on+0x42/0x60<br/> <span class="error">&#91;&lt;ffffffff81091cb6&gt;&#93;</span> ? autoremove_wake_function+0x16/0x40<br/> <span class="error">&#91;&lt;ffffffff81091eae&gt;&#93;</span> ? prepare_to_wait_exclusive+0x4e/0x80<br/> <span class="error">&#91;&lt;ffffffff81091f8e&gt;&#93;</span> ? prepare_to_wait+0x4e/0x80<br/> <span class="error">&#91;&lt;ffffffff81095da3&gt;&#93;</span> ? __hrtimer_start_range_ns+0x1a3/0x430<br/> <span class="error">&#91;&lt;ffffffff8109638a&gt;&#93;</span> ? down_read_trylock+0x1a/0x30<br/> <span class="error">&#91;&lt;ffffffff81096bff&gt;&#93;</span> ? up+0x2f/0x50<br/> <span class="error">&#91;&lt;ffffffff81098f05&gt;&#93;</span> async_manager_thread+0xc5/0x100<br/> <span class="error">&#91;&lt;ffffffff8109b9a9&gt;&#93;</span> ? ktime_get_ts+0xa9/0xe0<br/> <span class="error">&#91;&lt;ffffffff810a25a9&gt;&#93;</span> futex_wait_queue_me+0xb9/0xf0<br/> <span class="error">&#91;&lt;ffffffff810a666b&gt;&#93;</span> ? rt_mutex_adjust_pi+0x7b/0x90<br/> <span class="error">&#91;&lt;ffffffff810c2b01&gt;&#93;</span> ? cpuset_print_task_mems_allowed+0x91/0xb0<br/> <span class="error">&#91;&lt;ffffffff810c2b01&gt;&#93;</span> ? cpuset_print_task_mems_allowed+0x91/0xb0<br/> <span class="error">&#91;&lt;ffffffff810ca7b6&gt;&#93;</span> ? audit_hold_skb+0x26/0x50<br/> <span class="error">&#91;&lt;ffffffff810cab7b&gt;&#93;</span> ? kauditd_send_skb+0x3b/0x90<br/> <span class="error">&#91;&lt;ffffffff810d3d4b&gt;&#93;</span> ? audit_syscall_exit+0x25b/0x290<br/> <span class="error">&#91;&lt;ffffffff8110351b&gt;&#93;</span> slow_work_thread+0x32b/0x3a0<br/> <span class="error">&#91;&lt;ffffffff81108047&gt;&#93;</span> ? perf_event_exit_task+0x37/0x160<br/> <span class="error">&#91;&lt;ffffffff8110b832&gt;&#93;</span> ? iov_iter_copy_from_user_atomic+0x92/0x130<br/> <span class="error">&#91;&lt;ffffffff8110bb70&gt;&#93;</span> ? find_get_pages_tag+0x40/0x120<br/> <span class="error">&#91;&lt;ffffffff8110c060&gt;&#93;</span> ? sync_page+0x0/0x50<br/> <span class="error">&#91;&lt;ffffffff8110c0b0&gt;&#93;</span> ? sync_page_killable+0x0/0x40<br/> <span class="error">&#91;&lt;ffffffff8110eecb&gt;&#93;</span> oom_kill_process+0xcb/0x2e0<br/> <span class="error">&#91;&lt;ffffffff8111b3a5&gt;&#93;</span> ? __rmqueue+0xc5/0x490<br/> <span class="error">&#91;&lt;ffffffff8111bd57&gt;&#93;</span> bad_page+0x107/0x160<br/> <span class="error">&#91;&lt;ffffffff8111cf91&gt;&#93;</span> ? get_page_from_freelist+0x3d1/0x820<br/> <span class="error">&#91;&lt;ffffffff8111e1c6&gt;&#93;</span> ? __alloc_pages_nodemask+0xf6/0x810<br/> <span class="error">&#91;&lt;ffffffff8111e48d&gt;&#93;</span> ? __alloc_pages_nodemask+0x3bd/0x810<br/> <span class="error">&#91;&lt;ffffffff8111e745&gt;&#93;</span> __alloc_pages_nodemask+0x675/0x810<br/> <span class="error">&#91;&lt;ffffffff8111f78a&gt;&#93;</span> ? determine_dirtyable_memory+0x1a/0x30<br/> <span class="error">&#91;&lt;ffffffff81120951&gt;&#93;</span> ? do_writepages+0x21/0x40<br/> <span class="error">&#91;&lt;ffffffff8112bc27&gt;&#93;</span> ? vma_prio_tree_next+0x47/0x70<br/> <span class="error">&#91;&lt;ffffffff8112d14d&gt;&#93;</span> ? zone_statistics+0x7d/0xa0<br/> <span class="error">&#91;&lt;ffffffff8112d980&gt;&#93;</span> ? vmstat_update+0x0/0x40<br/> <span class="error">&#91;&lt;ffffffff8112de70&gt;&#93;</span> ? bdi_sync_supers+0x0/0x60<br/> <span class="error">&#91;&lt;ffffffff811336b5&gt;&#93;</span> ? unmap_vmas+0xa85/0xc00<br/> <span class="error">&#91;&lt;ffffffff811345a2&gt;&#93;</span> ? unmap_mapping_range+0x72/0x150<br/> <span class="error">&#91;&lt;ffffffff81135a85&gt;&#93;</span> ? handle_pte_fault+0x345/0xad0<br/> <span class="error">&#91;&lt;ffffffff81136455&gt;&#93;</span> ? handle_mm_fault+0x245/0x2b0<br/> <span class="error">&#91;&lt;ffffffff81139582&gt;&#93;</span> ? unlink_file_vma+0x42/0x70<br/> <span class="error">&#91;&lt;ffffffff8113e59d&gt;&#93;</span> ? rmap_walk+0x7d/0x1c0<br/> <span class="error">&#91;&lt;ffffffff8113f2de&gt;&#93;</span> ? page_referenced+0x9e/0x2f0<br/> <span class="error">&#91;&lt;ffffffff8113fb72&gt;&#93;</span> ? try_to_unmap_file+0x42/0x750<br/> <span class="error">&#91;&lt;ffffffff81156007&gt;&#93;</span> ? cache_grow+0x217/0x320<br/> <span class="error">&#91;&lt;ffffffff811560bf&gt;&#93;</span> ? cache_grow+0x2cf/0x320<br/> <span class="error">&#91;&lt;ffffffff81157e51&gt;&#93;</span> ? drain_array+0xe1/0x100<br/> <span class="error">&#91;&lt;ffffffff81158d38&gt;&#93;</span> ? drain_freelist+0x78/0xc0<br/> <span class="error">&#91;&lt;ffffffff81158d80&gt;&#93;</span> ? cache_reap+0x0/0x260<br/> <span class="error">&#91;&lt;ffffffff8115fe28&gt;&#93;</span> ? __mem_cgroup_uncharge_common+0x78/0x260<br/> <span class="error">&#91;&lt;ffffffff81161c89&gt;&#93;</span> ? mem_cgroup_charge_common+0x99/0xc0<br/> <span class="error">&#91;&lt;ffffffff81165218&gt;&#93;</span> khugepaged+0x958/0x1190<br/> <span class="error">&#91;&lt;ffffffff8116c65a&gt;&#93;</span> ? do_sync_read+0xfa/0x140<br/> <span class="error">&#91;&lt;ffffffff81175fdb&gt;&#93;</span> pipe_wait+0x5b/0x80<br/> <span class="error">&#91;&lt;ffffffff81258839&gt;&#93;</span> ? cpumask_next_and+0x29/0x50<br/> <span class="error">&#91;&lt;ffffffff81262a54&gt;&#93;</span> ? vsnprintf+0x484/0x5f0<br/> <span class="error">&#91;&lt;ffffffff81264025&gt;&#93;</span> ? memmove+0x45/0x50<br/> <span class="error">&#91;&lt;ffffffff812fcaa0&gt;&#93;</span> ? flush_to_ldisc+0x0/0x1b0<br/> <span class="error">&#91;&lt;ffffffff812fee81&gt;&#93;</span> vt_event_wait+0xa1/0x100<br/> <span class="error">&#91;&lt;ffffffff8137fe39&gt;&#93;</span> hub_thread+0x369/0x17f0<br/> <span class="error">&#91;&lt;ffffffff8138a164&gt;&#93;</span> ? usb_suspend_both+0x1a4/0x320<br/> <span class="error">&#91;&lt;ffffffff814277d0&gt;&#93;</span> ? eth_type_trans+0x40/0x140<br/> <span class="error">&#91;&lt;ffffffff81445e95&gt;&#93;</span> ? ip_local_out+0x25/0x30<br/> <span class="error">&#91;&lt;ffffffff8144e7e6&gt;&#93;</span> ? tcp_sendmsg+0x756/0xa30<br/> <span class="error">&#91;&lt;ffffffff8149b2d6&gt;&#93;</span> ? unix_stream_sendmsg+0x3c6/0x3e0<br/> <span class="error">&#91;&lt;ffffffff814c7b23&gt;&#93;</span> panic+0x78/0x137<br/> <span class="error">&#91;&lt;ffffffff814c8286&gt;&#93;</span> ? thread_return+0x4e/0x778<br/> <span class="error">&#91;&lt;ffffffff814c8b00&gt;&#93;</span> ? _cond_resched+0x30/0x40<br/> <span class="error">&#91;&lt;ffffffff814c8c5c&gt;&#93;</span> ? wait_for_common+0x14c/0x180<br/> <span class="error">&#91;&lt;ffffffff814c8d4d&gt;&#93;</span> ? wait_for_completion+0x1d/0x20<br/> <span class="error">&#91;&lt;ffffffff814c8f34&gt;&#93;</span> schedule_timeout+0x194/0x2f0<br/> <span class="error">&#91;&lt;ffffffff814c8f3c&gt;&#93;</span> ? schedule_timeout+0x19c/0x2f0<br/> <span class="error">&#91;&lt;ffffffff814c8fc5&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> <span class="error">&#91;&lt;ffffffff814c96e0&gt;&#93;</span> ? __mutex_lock_slowpath+0x70/0x180<br/> <span class="error">&#91;&lt;ffffffff814c97ae&gt;&#93;</span> __mutex_lock_slowpath+0x13e/0x180<br/> <span class="error">&#91;&lt;ffffffff814c9ad8&gt;&#93;</span> schedule_hrtimeout_range+0xc8/0x160<br/> <span class="error">&#91;&lt;ffffffff814c9b4d&gt;&#93;</span> schedule_hrtimeout_range+0x13d/0x160<br/> <span class="error">&#91;&lt;ffffffff814c9c1b&gt;&#93;</span> do_nanosleep+0x8b/0xc0<br/> <span class="error">&#91;&lt;ffffffff814ca6b5&gt;&#93;</span> rwsem_down_failed_common+0x95/0x1d0<br/> <span class="error">&#91;&lt;ffffffff814cac1b&gt;&#93;</span> ? _spin_unlock_bh+0x1b/0x20<br/> <span class="error">&#91;&lt;ffffffff814cd766&gt;&#93;</span> ? notifier_call_chain+0x16/0x80<br/> <span class="error">&#91;&lt;ffffffffa00a78be&gt;&#93;</span> ? __put_nfs_open_context+0x3e/0xc0 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa00a9e10&gt;&#93;</span> ? fib6_clean_node+0x0/0xd0 <span class="error">&#91;ipv6&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa00b0540&gt;&#93;</span> ? nfs_wait_bit_uninterruptible+0x0/0x20 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01407fd&gt;&#93;</span> ? call_transmit_status+0x4d/0xe0 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01433e9&gt;&#93;</span> ? xprt_release_xprt+0x89/0x90 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01435bf&gt;&#93;</span> ? xprt_reserve+0x1cf/0x1f0 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01444a0&gt;&#93;</span> ? xprt_autoclose+0x0/0x70 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0146210&gt;&#93;</span> ? xs_tcp_connect_worker4+0x0/0x30 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01488a0&gt;&#93;</span> ? rpc_async_release+0x0/0x20 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0148d00&gt;&#93;</span> ? rpc_wait_bit_killable+0x0/0x40 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0149760&gt;&#93;</span> ? rpc_async_schedule+0x0/0x20 <span class="error">&#91;sunrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01e68be&gt;&#93;</span> ? __put_nfs_open_context+0x3e/0xc0 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01e7560&gt;&#93;</span> ? nfs_wait_bit_killable+0x0/0x40 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01ef540&gt;&#93;</span> ? nfs_wait_bit_uninterruptible+0x0/0x20 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01f40cd&gt;&#93;</span> ? nfs_commit_free+0x3d/0x50 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01f4688&gt;&#93;</span> ? nfs_writeback_release_full+0x128/0x1b0 <span class="error">&#91;nfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01fe3a5&gt;&#93;</span> xpmem_clear_PFNtable+0x185/0x340 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02467b0&gt;&#93;</span> ? process_req+0x0/0x1a0 <span class="error">&#91;ib_addr&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02745ae&gt;&#93;</span> ? mlx4_ib_post_send+0x4be/0xf10 <span class="error">&#91;mlx4_ib&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02a80cd&gt;&#93;</span> ? mcast_work_handler+0xed/0x830 <span class="error">&#91;ib_sa&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa030073a&gt;&#93;</span> xpmem_make+0x9a/0x360 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0304fb1&gt;&#93;</span> ? xpmem_tg_ref_by_tgid+0x41/0xe0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03054f1&gt;&#93;</span> ? xpmem_PFNs_exist_in_range_l3+0x51/0xa0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0308445&gt;&#93;</span> xpmem_clear_PFNtable+0x185/0x340 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0309ec8&gt;&#93;</span> ? xpmem_recall_PFNs_of_tg+0xf8/0x2d0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa030a40b&gt;&#93;</span> xpmem_pgcl_thread+0x1db/0x220 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0320ab2&gt;&#93;</span> lcw_dispatch_main+0xd2/0x400 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0353b8b&gt;&#93;</span> ? mlx4_ib_poll_cq+0x2ab/0x780 <span class="error">&#91;mlx4_ib&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0379c9d&gt;&#93;</span> ? LNetMDAttach+0x35d/0x4c0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03dbc5a&gt;&#93;</span> obd_zombie_impexp_thread+0x15a/0x2b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa046a330&gt;&#93;</span> ? ipoib_reap_ah+0x0/0x50 <span class="error">&#91;ib_ipoib&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04e6c3a&gt;&#93;</span> ? kiblnd_queue_tx+0x4a/0x60 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04f3eb6&gt;&#93;</span> ? loi_list_maint+0xa6/0x130 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa050fb64&gt;&#93;</span> ? cache_add_extent+0x134/0x640 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa056efd0&gt;&#93;</span> ? ib_mad_completion_handler+0x0/0x810 <span class="error">&#91;ib_mad&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057a492&gt;&#93;</span> ? cm_process_work+0x32/0x110 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057bcff&gt;&#93;</span> ? cm_rep_handler+0x31f/0x590 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057bf70&gt;&#93;</span> ? cm_work_handler+0x0/0x11d6 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0584330&gt;&#93;</span> ? cma_work_handler+0x0/0xb0 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa059fc81&gt;&#93;</span> ? kiblnd_init_tx_msg+0x91/0x200 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05a4465&gt;&#93;</span> kiblnd_scheduler+0x325/0x760 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05bafed&gt;&#93;</span> ? ldlm_lock_put+0x19d/0x450 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05bffb1&gt;&#93;</span> ? ldlm_lock_decref+0x41/0xb0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05c0af3&gt;&#93;</span> ? ldlm_resource_putref_internal+0xb3/0x4c0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e3397&gt;&#93;</span> ? ldlm_callback_handler+0xa57/0x1e10 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05e6140&gt;&#93;</span> ldlm_bl_thread_main+0x3f0/0x440 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa060d1d0&gt;&#93;</span> ptlrpc_wait_event+0x3b0/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa060e6a7&gt;&#93;</span> ? lov_merge_lvb+0xb7/0x240 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0684ac2&gt;&#93;</span> ? ll_removepage+0x352/0x8d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0695c9c&gt;&#93;</span> ? ll_file_mmap+0x12c/0x180 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06ef6a7&gt;&#93;</span> ? lov_merge_lvb+0xb7/0x240 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f20f5&gt;&#93;</span> ? lov_finish_set+0x435/0x710 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07056a7&gt;&#93;</span> ? lov_merge_lvb+0xb7/0x240 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073f1a4&gt;&#93;</span> ll_close_thread+0x124/0x260 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa075aac2&gt;&#93;</span> ? ll_removepage+0x352/0x8d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09d7c9c&gt;&#93;</span> ? ll_file_mmap+0x12c/0x180 <span class="error">&#91;lustre&#93;</span><br/> &lt;IRQ&gt;<br/> &lt;IRQ&gt; <span class="error">&#91;&lt;ffffffff8106b857&gt;&#93;</span> warn_slowpath_common+0x87/0xc0<br/> &lt;IRQ&gt; <span class="error">&#91;&lt;ffffffff810d8740&gt;&#93;</span> ? handle_IRQ_event+0x60/0x170<br/> &lt;IRQ&gt; <span class="error">&#91;&lt;ffffffff814c7b23&gt;&#93;</span> panic+0x78/0x137</p> <p>It is probably not unexpected there are many places because<br/> guest@globe:/cores/people/jhanson/noaa/softlockup/nodeswithsoftlockupconsoles&gt; grep --binary-files=text -h -A1 "Call Trace" r* | wc -l<br/> 372554</p> <p>In the history if this cluster (as reflected in the console logs) we have had BUG: soft lockup 119496 times.</p> <p>There are a wide variety of places where the back trace starts but the two most dominant are</p> <p>grep -<del>binary-files=text -h -A1 "Call Trace" r* | grep -v "Call Trace" | grep -v ^</del>- | grep unmap_mapping_range | wc -l<br/> 49558<br/> grep -<del>binary-files=text -h -A1 "Call Trace" r* | grep -v "Call Trace" | grep -v ^</del>- | grep xpmem_tg_ref_by_tgid | wc -l<br/> 30446</p> <p>After the first function in the dominant ones it starts to diverge for unmap_mapping_range<br/> grep --binary-files=text -h -A1 "unmap_mapping_range" r* | sort | uniq<br/> &#8211;<br/> <span class="error">&#91;&lt;ffffffff81013cce&gt;&#93;</span> ? invalidate_interrupt0+0xe/0x20<br/> <span class="error">&#91;&lt;ffffffff810ddc95&gt;&#93;</span> ? call_rcu_sched+0x15/0x20<br/> <span class="error">&#91;&lt;ffffffff811343b4&gt;&#93;</span> unmap_mapping_range_vma+0x64/0xf0<br/> <span class="error">&#91;&lt;ffffffff811343ea&gt;&#93;</span> ? unmap_mapping_range_vma+0x9a/0xf0<br/> <span class="error">&#91;&lt;ffffffff811344d7&gt;&#93;</span> ? unmap_mapping_range_tree+0x97/0xf0<br/> <span class="error">&#91;&lt;ffffffff811344d7&gt;&#93;</span> unmap_mapping_range_tree+0x97/0xf0<br/> <span class="error">&#91;&lt;ffffffff811345a2&gt;&#93;</span> ? unmap_mapping_range+0x72/0x150<br/> <span class="error">&#91;&lt;ffffffff811345a2&gt;&#93;</span> unmap_mapping_range+0x72/0x150<br/> <span class="error">&#91;&lt;ffffffff81134661&gt;&#93;</span> ? unmap_mapping_range+0x131/0x150<br/> <span class="error">&#91;&lt;ffffffff81134661&gt;&#93;</span> unmap_mapping_range+0x131/0x150<br/> <span class="error">&#91;&lt;ffffffff814caa3e&gt;&#93;</span> ? _spin_lock+0x1e/0x30<br/> <span class="error">&#91;&lt;ffffffff814caa41&gt;&#93;</span> ? _spin_lock+0x21/0x30<br/> <span class="error">&#91;&lt;ffffffffa01fb4f1&gt;&#93;</span> ? xpmem_PFNs_exist_in_range_l3+0x51/0xa0 <span class="error">&#91;xpmem&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa042231c&gt;&#93;</span> ? ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa042231c&gt;&#93;</span> ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa069631c&gt;&#93;</span> ? ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa069631c&gt;&#93;</span> ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c31c&gt;&#93;</span> ? ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c31c&gt;&#93;</span> ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09d831c&gt;&#93;</span> ? ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09d831c&gt;&#93;</span> ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ac631c&gt;&#93;</span> ? ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ac631c&gt;&#93;</span> ll_teardown_mmaps+0x6c/0x1c0 <span class="error">&#91;lustre&#93;</span></p> <p>For xpmem_tg_ref_by_tgid it is only get_task_mm</p> <p>Bobi</p> <p>Andreas is rather busy at the moment so could you please review and comment on this latest information from our customer?</p> <p>Thanks</p> <p>Peter</p> <p>When did this situation happen? Did it happen after switched to RHEL6? Upgrading from older Lustre 1.8.6, or since starting to use specific kernel version or other software?</p> <p>Just looking at open cases. Customer found this was not a Lustre issue after all. I believe that they upgraded the kernel to fix the issue. Please close this.</p> <p>ok thanks for the update Dennis</p> Development Rank 1|hzvh9r: Rank (Obsolete) 6447 Severity