https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3007 Lustre <p>After finishing a test run with the 2.4 file system when we went to umount the entire file system one of the OSS expected a oops when it went to umount.<br/> The oops is:</p> <p>Mar 9 20:11:12 widow-oss11c2 kernel: <span class="error">&#91;112998.231220&#93;</span> Lustre: server umount routed1-OST00b1 complete<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.326796&#93;</span> LustreError: 20552:0:(obd_mount.c:2115:server_put_super()) routed1-OST01d1: failed to disconnect lwp. (rc=-110)<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.359171&#93;</span> LustreError: 20552:0:(obd_mount.c:2115:server_put_super()) Skipped 1 previous similar message<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.370669&#93;</span> Lustre: server umount routed1-OST0171 complete<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.370672&#93;</span> Lustre: Skipped 1 previous similar message<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.429394&#93;</span> Lustre: Failing over routed1-OST01d1<br/> Mar 9 20:11:13 widow-oss11c2 kernel: <span class="error">&#91;112999.448337&#93;</span> Lustre: Skipped 1 previous similar message<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.073001&#93;</span> BUG: soft lockup - CPU#0 stuck for 67s! <span class="error">&#91;obd_zombid:5676&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.073001&#93;</span> Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ext4 mbcache jbd2 sha512_generic sha256_generic ipmi_devintf ipmi_si ipmi_msghandler autofs4 dm_round_robin scsi_wait_scan sd_mod crc_t10dif ib_srp scsi_transport_srp scsi_tgt nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg sr_mod cdrom radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core ib_mthca ib_mad ib_core shpchp microcode dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support i5000_edac edac_core i5k_amb ioatdma dca ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc bnx2 <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Pid: 5676, comm: obd_zombid Not tainted 2.6.32-279.22.1.el6.head.x86_64 #1 Dell Inc. PowerEdge 1950/0TT740<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> RIP: 0010:<span class="error">&#91;&lt;ffffffff81147e76&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff81147e76&gt;&#93;</span> remove_vm_area+0x86/0xa0<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> RSP: 0018:ffff880419427cb0 EFLAGS: 00000287<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> RAX: ffff8800a66fb540 RBX: ffff880419427cd0 RCX: ffff88011e073440<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> RDX: ffff88011e0733c0 RSI: 0000000000000001 RDI: ffffffff81abea30<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> RBP: ffffffff8100bb8e R08: 0000000000000000 R09: ffffc900cc6ee000<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> R10: ffff880028402500 R11: ffff88008acf3800 R12: ffff880419427c80<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> R13: 0000000000000297 R14: ffff8800000206c0 R15: ffffea0002410270<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> FS: 00007f4834066700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> CR2: 00000000004050b9 CR3: 0000000001a85000 CR4: 00000000000006f0<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Process obd_zombid (pid: 5676, threadinfo ffff880419426000, task ffff8803e2329540)<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Stack:<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> ffff88001b6df840 ffff8800a66fb540 ffff88008ad17850 ffffc900cc6ee000<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> &lt;d&gt; ffff880419427d00 ffffffff81147f4e 000000000000005a ffff88006f537800<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> &lt;d&gt; ffff8803d5e93400 ffff8803d5e93430 ffff880419427d10 ffffffff811480ba<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Call Trace:<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff81147f4e&gt;&#93;</span> ? __vunmap+0x2e/0x120<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff811480ba&gt;&#93;</span> ? vfree+0x2a/0x40<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa06e3b0e&gt;&#93;</span> ? cfs_free_large+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa0e307c2&gt;&#93;</span> ? ptlrpc_free_rqbd+0xc2/0x3b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa0e33c45&gt;&#93;</span> ? ptlrpc_unregister_service+0x5d5/0x10b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff81052223&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa1253292&gt;&#93;</span> ? ost_cleanup+0x52/0x1a0 <span class="error">&#91;ost&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07f1d92&gt;&#93;</span> ? class_decref+0x212/0x580 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa06f32e1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf549&gt;&#93;</span> ? obd_zombie_impexp_cull+0x309/0x5d0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf8d5&gt;&#93;</span> ? obd_zombie_impexp_thread+0xc5/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Code: 30 ea ab 81 48 89 45 e8 e8 88 4d 3a 00 48 8b 15 19 73 e6 00 48 c7 c1 78 f1 fa 81 48 8b 45 e8 48 39 d3 74 0c 90 48 89 d1 48 8b 12 &lt;48&gt; 39 d3 75 f5 48 8b 13 48 89 11 f0 81 05 a4 6b 97 00 00 00 00<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> Call Trace:<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff81147f4e&gt;&#93;</span> ? __vunmap+0x2e/0x120<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff811480ba&gt;&#93;</span> ? vfree+0x2a/0x40<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa06e3b0e&gt;&#93;</span> ? cfs_free_large+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa0e307c2&gt;&#93;</span> ? ptlrpc_free_rqbd+0xc2/0x3b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa0e33c45&gt;&#93;</span> ? ptlrpc_unregister_service+0x5d5/0x10b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff81052223&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa1253292&gt;&#93;</span> ? ost_cleanup+0x52/0x1a0 <span class="error">&#91;ost&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07f1d92&gt;&#93;</span> ? class_decref+0x212/0x580 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa06f32e1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf549&gt;&#93;</span> ? obd_zombie_impexp_cull+0x309/0x5d0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf8d5&gt;&#93;</span> ? obd_zombie_impexp_thread+0xc5/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20<br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffffa07cf810&gt;&#93;</span> ? obd_zombie_impexp_thread+0x0/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> Mar 9 20:12:30 widow-oss11c2 kernel: <span class="error">&#91;113076.102203&#93;</span> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Mar 9 20:13:35 widow-oss11c2 kernel: <span class="error">&#91;113140.803317&#93;</span> Lustre: server umount routed1-OST0231 complete<br/> Mar 9 20:13:35 widow-oss11c2 kernel: <span class="error">&#91;113140.819063&#93;</span> Lustre: Skipped 1 previous similar message</p> OSS server runing Lustre 2.3.61 LU-3007

Umount OST caused a oops in ptlrpc_free_rqbd

Bug Major Resolved Cannot Reproduce Bruno Faccini James A Simmons Thu, 21 Mar 2013 14:28:40 +0000 Wed, 17 Jul 2013 14:48:08 +0000 Wed, 17 Jul 2013 14:48:08 +0000 Lustre 2.4.0 0 5 <p>James,<br/> It is only a soft-lockup trace/stack, was it a one-shoot or were there multiple ??<br/> How many OSTs to be stopped ??<br/> A full log may be welcome.</p> <p>so it seems the unmount actually succeeded from further messages.<br/> the only issue is we are probably taking too long to unregister some buffers when there are many of them, so should not be holding any locks over that?</p> <p>Bruno the soft-lockup only happened once on single OSS when unmounting this particular OST. The total OSTs were 376 if I remember correctly. The other were fine.</p> <p>Humm, Having a look to some assembly of the remove_vm_area() routine it seems that the soft-lockup occured during the for loop to find the matching vm_area :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>struct vm_struct *remove_vm_area(const void *addr) { struct vmap_area *va; va = find_vmap_area((unsigned long)addr); if (va &amp;&amp; va-&gt;flags &amp; VM_VM_AREA) { struct vm_struct *vm = va-&gt;private; if (!(vm-&gt;flags &amp; VM_UNLIST)) { struct vm_struct *tmp, **p; /* * remove from list and disallow access to * this vm_struct before unmap. (address range * confliction is maintained by vmap.) */ write_lock(&amp;vmlist_lock); for (p = &amp;vmlist; (tmp = *p) != vm; p = &amp;tmp-&gt;next) &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; ; *p = tmp-&gt;next; write_unlock(&amp;vmlist_lock); } vmap_debug_free_range(va-&gt;va_start, va-&gt;va_end); free_unmap_vmap_area(va); vm-&gt;size -= PAGE_SIZE; return vm; } return NULL; } </pre> </div></div> <p>Thus it is kind of strange that this thread was the only "impacted"!!...<br/> Also is it reasonable to think that it spent more than 67s in this loop and finally exited from it ?<br/> May be it spent most of this time trying to write_lock(&amp;vmlist_lock) before ?</p> <p>I found out from our admin it was more than OST umounting that had a problem. The logs you saw was for one that completed. The rest never completed and after 30 minutes the admin rebooted the machine instead.</p> <p>can we get logs from one of the nodes that did not complete unmounting, please?</p> <p>Here is the entire log of the day for this OSS that crashed when unmounting.</p> <p>Thank's for the log, but next time this kind of hung situation requires a hard reboot/reset, is it possible that you take a crash-dump instead ??</p> <p>James,<br/> There is nothing more of interest in the log, thus we need at least a full back trace (Alt+SysRq+T, or echo t &gt; /proc/sysrq-trigger) to be taken at the time the soft-lockup is reported, and a forced crash-dump would be a must.</p> <p>We have changed are policy now for test shots. For the last test shot we did not encounter this problem. If we don't encounter this at the next test shot I say we can close the ticket.</p> <p>Hello James, <br/> How went the further test shots finally ??<br/> If ok, can we close this ticket? If not, have you been able to take a crash-dump upon a new occurrence?</p> <p>Yes you can close this ticket. If we encounter it we can reopen but we haven't seen this problem lately.</p> <p>Ok thank's James.</p> Development Rank 1|hzvlv3: Rank (Obsolete) 7320 Severity