https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3614 Lustre <p>Hello,</p> <p>Our OST-servers periodically crashes with kernel panic error:</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 9<br/> Modules linked in: lmv(U) obdfilter(U) fsfilt_ldiskfs(U) exportfs ost(U) mgc(U) ldiskfs(U) jbd2 lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ses enclosure sg microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core ib_mthca ib_mad ib_core igb dca ext3 jbd mbcache sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 26235, comm: ldlm_bl_04 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP: 0018:ffff88016223bd40 EFLAGS: 00010206<br/> RAX: ffffffffa0993100 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000000<br/> RDX: 000000000000e4f9 RSI: ffff8801f08ead78 RDI: ffffffffa0993100<br/> RBP: ffff88016223bd70 R08: 0000000000000000 R09: ffff88014f9a2400<br/> R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: ffff8801f08ead78<br/> R13: ffff8801f41a9b58 R14: ffff8801f41a9b58 R15: ffff88020cf45240<br/> FS: 00007f01e69b9700(0000) GS:ffff8800282a0000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 000000000042b650 CR3: 0000000001a85000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process ldlm_bl_04 (pid: 26235, threadinfo ffff88016223a000, task ffff8801ad57e080)<br/> Stack:<br/> ffff88016223bd40 ffff8801f08ead78 0000000000000000 ffff88022ce47c18<br/> &lt;d&gt; ffff8801f41a9b58 ffff88020cf45240 ffff88016223bdc0 ffffffffa0976ab4<br/> &lt;d&gt; ffffffffa04f944d 0000000000000000 ffff88016223bda0 ffff8801f41a9b58<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0976ab4&gt;&#93;</span> osc_lock_cancel+0xa4/0x1b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04f944d&gt;&#93;</span> ? cl_env_nested_get+0x5d/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ff225&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04fff0b&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0977bba&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a123&gt;&#93;</span> ldlm_handle_bl_callback+0x123/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a561&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: fd 48 c7 c7 00 31 99 a0 e8 bd 60 b7 e0 49 8b 5c 24 28 48 85 db 0f 84 7f 00 00 00 49 c7 44 24 28 00 00 00 00 48 c7 c0 00 31 99 a0 &lt;48&gt; c7 83 70 01 00 00 00 00 00 00 49 c7 44 24 60 00 00 00 00 66<br/> RIP <span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP &lt;ffff88016223bd40&gt;</p> Linux 2.6.32-279.19.1.el6_lustre.x86_64 #1 SMP LU-3614

Kernel Panic "osc_lock_detach"

Bug Critical Resolved Cannot Reproduce Bruno Faccini Rustem Bikboulatov Mon, 22 Jul 2013 13:49:04 +0000 Sat, 9 Oct 2021 06:00:23 +0000 Sat, 9 Oct 2021 06:00:23 +0000 Lustre 2.1.5 1 9 <p>Hello, </p> <p>Today, we are faced with yet another case of "kernel panic" on another OSS server. Here is the output log. We have a crash dump file, and if you need to send it for analysis, we are ready to do it. These problems concern us, since we used Lustre to build news television production, and it is associated with risk of problems with the air. Please help in diagnosing and fixing the problem. Thanks.</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 6<br/> Modules linked in: lmv(U) obdfilter(U) fsfilt_ldiskfs(U) exportfs ost(U) mgc(U) ldiskfs(U) jbd2 lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ses enclosure microcode serio_raw sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support i7core_edac edac_core ioatdma ib_mthca ib_mad ib_core igb dca ext3 jbd mbcache sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 8815, comm: ldlm_bl_28 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0978751&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0978751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP: 0018:ffff8802d8525d40 EFLAGS: 00010206<br/> RAX: ffffffffa0995100 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000000<br/> RDX: 00000000000069ea RSI: ffff8801a4fd7ee8 RDI: ffffffffa0995100<br/> RBP: ffff8802d8525d70 R08: 0000000000000000 R09: ffff8802f4493000<br/> R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: ffff8801a4fd7ee8<br/> R13: ffff8802e39ecaa0 R14: ffff8802e39ecaa0 R15: ffff880118e55240<br/> FS: 00007fcdae134700(0000) GS:ffff8801c5840000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 0000000001ed4d38 CR3: 0000000001a85000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process ldlm_bl_28 (pid: 8815, threadinfo ffff8802d8524000, task ffff88003ee8e080)<br/> Stack:<br/> ffff8802d8525d40 ffff8801a4fd7ee8 0000000000000000 ffff88011c005dd8<br/> &lt;d&gt; ffff8802e39ecaa0 ffff880118e55240 ffff8802d8525dc0 ffffffffa0978ab4<br/> &lt;d&gt; ffffffffa04fb44d 0000000000000000 ffff8802d8525da0 ffff8802e39ecaa0<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0978ab4&gt;&#93;</span> osc_lock_cancel+0xa4/0x1b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04fb44d&gt;&#93;</span> ? cl_env_nested_get+0x5d/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0501225&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0501f0b&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0979bba&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062c123&gt;&#93;</span> ldlm_handle_bl_callback+0x123/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062c561&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa062c2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa062c2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062c2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: fd 48 c7 c7 00 51 99 a0 e8 bd 40 b7 e0 49 8b 5c 24 28 48 85 db 0f 84 7f 00 00 00 49 c7 44 24 28 00 00 00 00 48 c7 c0 00 51 99 a0 &lt;48&gt; c7 83 70 01 00 00 00 00 00 00 49 c7 44 24 60 00 00 00 00 66<br/> RIP <span class="error">&#91;&lt;ffffffffa0978751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP &lt;ffff8802d8525d40&gt;</p> <p>It happened again on another server.</p> <p><span class="error">&#91;root@n23 lustre_2.1.5&#93;</span># crash /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6_lustre.x86_64/vmlinux /var/crash/127.0.0.1-2013-08-07-17\:00\:44/vmcore</p> <p>crash 6.0.4-2.el6<br/> Copyright (C) 2002-2012 Red Hat, Inc.<br/> Copyright (C) 2004, 2005, 2006 IBM Corporation<br/> Copyright (C) 1999-2006 Hewlett-Packard Co<br/> Copyright (C) 2005, 2006 Fujitsu Limited<br/> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.<br/> Copyright (C) 2005 NEC Corporation<br/> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.<br/> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.<br/> This program is free software, covered by the GNU General Public License,<br/> and you are welcome to change it and/or distribute copies of it under<br/> certain conditions. Enter "help copying" to see the conditions.<br/> This program has absolutely no warranty. Enter "help warranty" for details.</p> <p>GNU gdb (GDB) 7.3.1<br/> Copyright (C) 2011 Free Software Foundation, Inc.<br/> License GPLv3+: GNU GPL version 3 or later &lt;<a href="http://gnu.org/licenses/gpl.html" class="external-link" target="_blank" rel="nofollow noopener">http://gnu.org/licenses/gpl.html</a>&gt;<br/> This is free software: you are free to change and redistribute it.<br/> There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br/> and "show warranty" for details.<br/> This GDB was configured as "x86_64-unknown-linux-gnu"...</p> <p> KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6_lustre.x86_64/vmlinux<br/> DUMPFILE: /var/crash/127.0.0.1-2013-08-07-17:00:44/vmcore <span class="error">&#91;PARTIAL DUMP&#93;</span><br/> CPUS: 16<br/> DATE: Wed Aug 7 16:59:40 2013<br/> UPTIME: 28 days, 17:16:23<br/> LOAD AVERAGE: 0.13, 0.32, 0.33<br/> TASKS: 848<br/> NODENAME: n23<br/> RELEASE: 2.6.32-279.19.1.el6_lustre.x86_64<br/> VERSION: #1 SMP Wed Mar 20 16:37:18 PDT 2013<br/> MACHINE: x86_64 (2400 Mhz)<br/> MEMORY: 12 GB<br/> PANIC: ""<br/> PID: 26404<br/> COMMAND: "ldlm_bl_10"<br/> TASK: ffff880287ea6080 <span class="error">&#91;THREAD_INFO: ffff880287ea0000&#93;</span><br/> CPU: 0<br/> STATE: TASK_RUNNING (PANIC)</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 0<br/> Modules linked in: lmv(U) obdfilter(U) fsfilt_ldiskfs(U) exportfs ost(U) mgc(U) ldiskfs(U) jbd2 lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx ses enclosure microcode sg serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core ib_mthca ib_mad ib_core igb dca ext3 jbd mbcache sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 26404, comm: ldlm_bl_10 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP: 0018:ffff880287ea1d40 EFLAGS: 00010206<br/> RAX: ffffffffa0993100 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000000<br/> RDX: 0000000000006415 RSI: ffff880193938e30 RDI: ffffffffa0993100<br/> RBP: ffff880287ea1d70 R08: 0000000000000000 R09: ffff8801bafa1200<br/> R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: ffff880193938e30<br/> R13: ffff8801842aec10 R14: ffff8801842aec10 R15: ffff88016101bd80<br/> FS: 00007fd3193ef700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 00000000024585e0 CR3: 0000000001a85000 CR4: 00000000000006f0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process ldlm_bl_10 (pid: 26404, threadinfo ffff880287ea0000, task ffff880287ea6080)<br/> Stack:<br/> ffff880287ea1d40 ffff880193938e30 0000000000000000 ffff880179cb26d8<br/> &lt;d&gt; ffff8801842aec10 ffff88016101bd80 ffff880287ea1dc0 ffffffffa0976ab4<br/> &lt;d&gt; ffffffffa04f944d 0000000000000000 ffff880287ea1da0 ffff8801842aec10<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0976ab4&gt;&#93;</span> osc_lock_cancel+0xa4/0x1b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04f944d&gt;&#93;</span> ? cl_env_nested_get+0x5d/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ff225&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04fff0b&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0977bba&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a123&gt;&#93;</span> ldlm_handle_bl_callback+0x123/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a561&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062a2e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: fd 48 c7 c7 00 31 99 a0 e8 bd 60 b7 e0 49 8b 5c 24 28 48 85 db 0f 84 7f 00 00 00 49 c7 44 24 28 00 00 00 00 48 c7 c0 00 31 99 a0 &lt;48&gt; c7 83 70 01 00 00 00 00 00 00 49 c7 44 24 60 00 00 00 00 66<br/> RIP <span class="error">&#91;&lt;ffffffffa0976751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP &lt;ffff880287ea1d40&gt;</p> <p>crash&gt; kmem -i<br/> PAGES TOTAL PERCENTAGE<br/> TOTAL MEM 3014316 11.5 GB ----<br/> FREE 596526 2.3 GB 19% of TOTAL MEM<br/> USED 2417790 9.2 GB 80% of TOTAL MEM<br/> SHARED 803219 3.1 GB 26% of TOTAL MEM<br/> BUFFERS 3691 14.4 MB 0% of TOTAL MEM<br/> CACHED 1671331 6.4 GB 55% of TOTAL MEM<br/> SLAB 473084 1.8 GB 15% of TOTAL MEM</p> <p>TOTAL SWAP 524286 2 GB ----<br/> SWAP USED 11 44 KB 0% of TOTAL SWAP<br/> SWAP FREE 524275 2 GB 99% of TOTAL SWAP</p> <p>Hello, anybody there?</p> <p>Hello again!</p> <p>We install a lustre client on a separate server from the OSS. This server only worked with lustre client and application software. A day later we got the same symptom - server "kernel panic". The problem is most likely related only to lustre client.</p> <p><span class="error">&#91;root@r01 ~&#93;</span># crash /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6_lustre.x86_64/vmlinux /var/crash/127.0.0.1-2013-08-08-20\:42\:27/vmcore</p> <p>crash 6.0.4-2.el6<br/> Copyright (C) 2002-2012 Red Hat, Inc.<br/> Copyright (C) 2004, 2005, 2006 IBM Corporation<br/> Copyright (C) 1999-2006 Hewlett-Packard Co<br/> Copyright (C) 2005, 2006 Fujitsu Limited<br/> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.<br/> Copyright (C) 2005 NEC Corporation<br/> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.<br/> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.<br/> This program is free software, covered by the GNU General Public License,<br/> and you are welcome to change it and/or distribute copies of it under<br/> certain conditions. Enter "help copying" to see the conditions.<br/> This program has absolutely no warranty. Enter "help warranty" for details.</p> <p>GNU gdb (GDB) 7.3.1<br/> Copyright (C) 2011 Free Software Foundation, Inc.<br/> License GPLv3+: GNU GPL version 3 or later &lt;<a href="http://gnu.org/licenses/gpl.html" class="external-link" target="_blank" rel="nofollow noopener">http://gnu.org/licenses/gpl.html</a>&gt;<br/> This is free software: you are free to change and redistribute it.<br/> There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br/> and "show warranty" for details.<br/> This GDB was configured as "x86_64-unknown-linux-gnu"...</p> <p> KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6_lustre.x86_64/vmlinux<br/> DUMPFILE: /var/crash/127.0.0.1-2013-08-08-20:42:27/vmcore <span class="error">&#91;PARTIAL DUMP&#93;</span><br/> CPUS: 16<br/> DATE: Thu Aug 8 20:41:22 2013<br/> UPTIME: 04:19:08<br/> LOAD AVERAGE: 0.04, 0.95, 1.89<br/> TASKS: 568<br/> NODENAME: r01<br/> RELEASE: 2.6.32-279.19.1.el6_lustre.x86_64<br/> VERSION: #1 SMP Wed Mar 20 16:37:18 PDT 2013<br/> MACHINE: x86_64 (2400 Mhz)<br/> MEMORY: 12 GB<br/> PANIC: ""<br/> PID: 3170<br/> COMMAND: "ldlm_bl_00"<br/> TASK: ffff88033482d540 <span class="error">&#91;THREAD_INFO: ffff880338c30000&#93;</span><br/> CPU: 5<br/> STATE: TASK_RUNNING (PANIC)</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 5<br/> Modules linked in: lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) vfat fat usb_storage autofs4 cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode serio_raw ses enclosure sg i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core ib_mthca ib_mad ib_core igb dca ext3 jbd mbcache sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 3170, comm: ldlm_bl_00 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0953751&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0953751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP: 0018:ffff880338c31d40 EFLAGS: 00010206<br/> RAX: ffffffffa0970100 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000000<br/> RDX: 000000000000e841 RSI: ffff88010049dcc0 RDI: ffffffffa0970100<br/> RBP: ffff880338c31d70 R08: 0000000000000000 R09: ffff8801d7171400<br/> R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: ffff88010049dcc0<br/> R13: ffff8801b858eaa0 R14: ffff8801b858eaa0 R15: ffff8801b6309900<br/> FS: 00007f8ad6ef7700(0000) GS:ffff8801c5820000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 00007f8d0a82d000 CR3: 0000000001a85000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process ldlm_bl_00 (pid: 3170, threadinfo ffff880338c30000, task ffff88033482d540)<br/> Stack:<br/> ffff880338c31d40 ffff88010049dcc0 0000000000000000 ffff880186358698<br/> &lt;d&gt; ffff8801b858eaa0 ffff8801b6309900 ffff880338c31dc0 ffffffffa0953ab4<br/> &lt;d&gt; ffffffffa04d644d 0000000000000000 ffff880338c31da0 ffff8801b858eaa0<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0953ab4&gt;&#93;</span> osc_lock_cancel+0xa4/0x1b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04d644d&gt;&#93;</span> ? cl_env_nested_get+0x5d/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dc225&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dcf0b&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0954bba&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0607123&gt;&#93;</span> ldlm_handle_bl_callback+0x123/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0607561&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa06072e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa06072e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06072e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: fd 48 c7 c7 00 01 97 a0 e8 bd 90 b9 e0 49 8b 5c 24 28 48 85 db 0f 84 7f 00 00 00 49 c7 44 24 28 00 00 00 00 48 c7 c0 00 01 97 a0 &lt;48&gt; c7 83 70 01 00 00 00 00 00 00 49 c7 44 24 60 00 00 00 00 66<br/> RIP <span class="error">&#91;&lt;ffffffffa0953751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP &lt;ffff880338c31d40&gt;</p> <p>crash&gt; kmem -i<br/> PAGES TOTAL PERCENTAGE<br/> TOTAL MEM 3014310 11.5 GB ----<br/> FREE 511917 2 GB 16% of TOTAL MEM<br/> USED 2502393 9.5 GB 83% of TOTAL MEM<br/> SHARED 1791768 6.8 GB 59% of TOTAL MEM<br/> BUFFERS 357 1.4 MB 0% of TOTAL MEM<br/> CACHED 1794256 6.8 GB 59% of TOTAL MEM<br/> SLAB 599425 2.3 GB 19% of TOTAL MEM</p> <p>TOTAL SWAP 524286 2 GB ----<br/> SWAP USED 0 0 0% of TOTAL SWAP<br/> SWAP FREE 524286 2 GB 100% of TOTAL SWAP</p> <p>Running with latest b2_1 build #220 I got this same crash with reproducer of <a href="https://jira.whamcloud.com/browse/LU-4112" title="Random eviction of clients on lock callback timeout" class="issue-link" data-issue-key="LU-4112"><del>LU-4112</del></a>.<br/> Will submit a patch soon.</p> <p>Bruno, that's the good news. We will wait for the patch.<br/> I also want to pay attention on bug <a href="http://jira.hpdd.intel.com/browse/LU-3766" class="external-link" target="_blank" rel="nofollow noopener">LU-3766</a>. Perhaps these two bugs are related, as they are appear in the case of active use the file locking feature by applications (flock mount option).</p> <p>Bruno - Any word on this one?</p> <p>For what it's worth, on the Cray side, we think this may be a cl_lock bug related issue, similar to <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>.</p> <p>Rustem - Can you share details of what you are running and what the setup is when you hit this bug? We'd like to try to get debugging information, but do not have a reliable reproducer.</p> <p>Lustre Cluster Diagram</p> <p>Patrick, here the cluster configuration:</p> <p>Lustre Server MGS/MDS - mmp-2<br/> Lustre Servers OSS - n11, n12, n13, n14, n15, n21, n22, n23, n24, n25<br/> Lustre Clients - r01, r02, r03, r04, mmp-1, vn-1, cln01, cln02, cln03, cln04</p> <p>(refer to the diagram "20140113 - Hardware Diagram v0.1_R3.gif" in attachment)</p> <p>Environment:<br/> Linux 2.6.32-279.19.1.el6_lustre.x86_64 #1 SMP</p> <p>Mount points:</p> <p>OSS:<br/> /dev/md11 on /lustre/ost type lustre (rw,noauto,_netdev,abort_recov)</p> <p>MGS/MDS:</p> <p>/dev/lustre_mgs on /lustre/mgs type lustre (rw,noauto,_netdev,abort_recov)<br/> /dev/lustre_mdt1 on /lustre/mdt1 type lustre (rw,noauto,_netdev,abort_recov)</p> <p>Clients:<br/> mmp-2@tcp:mmp-1@tcp:/lustre1 on /array1 type lustre (rw,noauto,_netdev,flock,abort_recov,lazystatfs)</p> <p>Stripe config:</p> <p><span class="error">&#91;root@mmp-1 ~&#93;</span># lfs getstripe /array1/.<br/> /array1/.<br/> stripe_count: 1 stripe_size: 1048576 stripe_offset: -1</p> <p>kdump config:</p> <p>core_collector makedumpfile -c --message-level 1 -d 31</p> <p>Application Software Description (LRVfarm):</p> <p>LRVfarm - it's the software that processes media files (video+audio) and creates a proxy video ( low resolution video). Each running task contains a small task file, which is located on Lustre file system. Software LRVfarm runs several threads (processes) - 8 processes on each client (r01, r02, r03, r04). For a total of 32 LRVfarm processes running in cluster. Each process uses the file locking feature when performing tasks. LRVfarm process locks a task file and perform the task. Other LRVfarm processes (on local client and remote clients) also attempt to lock that task files, but it is not possible in the case when the task already running (for the reason that the file is already locked by another LRVfarm process). Periodically, all clients are crashes (kernel panic) with different errors. Here the crash statistics for the last time:</p> <p>Client r01<br/> ==================</p> <p>2013-12-30-13:27:09 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-12-18-17:53:58 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-11-27-11:33:23 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-11-22-19:13:07 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-29-18:38:51 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-06-13:47:27 osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> 2013-09-29-07:10:08 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-08-29-16:34:36 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-08-08-20:42:27 osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span></p> <p>Client r02<br/> ==================</p> <p>2014-01-13-16:50:05 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2014-01-07-23:05:13 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-12-25-10:01:04 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-12-16-00:34:43 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-11-25-12:54:33 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-11-14-21:59:24 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-09-11-15:57:46 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:</p> <p>Client r03<br/> ==================</p> <p>2013-11-06-20:25:26 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-16-19:04:01 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-14-18:38:01 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-09-30-19:26:50 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-09-12-19:55:17 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-09-10-11:22:25 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-08-19-01:19:55 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-08-13-10:15:56 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:</p> <p>Client r04<br/> ==================</p> <p>2013-12-08-06:19:11 cl_lock_mutex_get+0x2e/0xe0 <span class="error">&#91;obdclass&#93;</span><br/> 2013-12-03-13:32:04 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-11-16-08:49:25 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-21-22:14:46 ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed:<br/> 2013-10-13-18:48:27 osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> 2013-09-01-14:26:14 osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span></p> <p>The last crash with "osc_lock_detach" error occurred on October 13, 2013. At the time of creating of this ticket error "osc_lock_detach" was the most common one. Perhaps at that time was different load conditions on Lustre. But now the most frequently occurring errors - "ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed" (<a href="https://jira.whamcloud.com/browse/LU-3766" title="ASSERTION( stripe &lt; lio-&gt;lis_stripe_count )" class="issue-link" data-issue-key="LU-3766"><del>LU-3766</del></a>). To reproduce the "osc_lock_detach" error we have to wait a few months, and it does not guarantee that the error be reproduced at all. Nevertheless, I can send the crush dump (vmcore) of "osc_lock_detach" error from October 2013 in Kdump format (kdump config see above).</p> <p>Today there was a kernel panic "osc_lock_detach" again:</p> <p>==================================================================</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 4<br/> Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ses enclosure ib_mthca ib_mad ib_core microcode igb sg serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext3 jbd mbcache mpt2sas(U) scsi_transport_sas raid_class sd_mod crc_t10dif ahci dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 23021, comm: ldlm_bl_37 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0952751&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0952751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP: 0018:ffff8801dacafd40 EFLAGS: 00010206<br/> RAX: ffffffffa096f100 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000000<br/> RDX: 000000000000677e RSI: ffff880190cf8e30 RDI: ffffffffa096f100<br/> RBP: ffff8801dacafd70 R08: 0000000000000000 R09: ffff88026637d200<br/> R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: ffff880190cf8e30<br/> R13: ffff880129b02ef0 R14: ffff880129b02ef0 R15: ffff8801bca1d6c0<br/> FS: 00007f89a61e3700(0000) GS:ffff880145800000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 00007fc7675a9000 CR3: 000000011d86a000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process ldlm_bl_37 (pid: 23021, threadinfo ffff8801dacae000, task ffff880292206ae0)<br/> Stack:<br/> ffff8801dacafd40 ffff880190cf8e30 0000000000000000 ffff8801d5c20518<br/> &lt;d&gt; ffff880129b02ef0 ffff8801bca1d6c0 ffff8801dacafdc0 ffffffffa0952ab4<br/> &lt;d&gt; ffffffffa04d544d 0000000000000000 ffff8801dacafda0 ffff880129b02ef0<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0952ab4&gt;&#93;</span> osc_lock_cancel+0xa4/0x1b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04d544d&gt;&#93;</span> ? cl_env_nested_get+0x5d/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04db225&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dbf0b&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0953bba&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0606123&gt;&#93;</span> ldlm_handle_bl_callback+0x123/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0606561&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa06062e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa06062e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06062e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: fd 48 c7 c7 00 f1 96 a0 e8 bd a0 b9 e0 49 8b 5c 24 28 48 85 db 0f 84 7f 00 00 00 49 c7 44 24 28 00 00 00 00 48 c7 c0 00 f1 96 a0 &lt;48&gt; c7 83 70 01 00 00 00 00 00 00 49 c7 44 24 60 00 00 00 00 66<br/> RIP <span class="error">&#91;&lt;ffffffffa0952751&gt;&#93;</span> osc_lock_detach+0x51/0x1b0 <span class="error">&#91;osc&#93;</span><br/> RSP &lt;ffff8801dacafd40&gt;</p> <p>==========================================================================</p> <p>Prior to this, we have changed the mount options for lustre clients on all servers. We excluded mode "flock", as it was suspected that a crash associated with it. Now mount options are:</p> <p>Clients (r01, r02, r03, r04, mmp-1, vn-1):<br/> mmp-2@tcp:mmp-1@tcp:/lustre1 on /array1 type lustre (rw,noauto,_netdev,noflock,abort_recov,lazystatfs)</p> <p>Clients (cln01, cln02, cln03, cln04):<br/> mmp-2@tcp:mmp-1@tcp:/lustre1 on /array1 type lustre (rw,noauto,_netdev,localflock,abort_recov,lazystatfs)</p> <p>I also want to note that the issue <a href="https://jira.whamcloud.com/browse/LU-4886" title="Kernel Panic &quot;cl_lock_put&quot;" class="issue-link" data-issue-key="LU-4886"><del>LU-4886</del></a> (also my issues) is associated with this issue. These two errors are very similar trace log, and it happens on the same servers (r01, r02, r03, r04) with the lustre client 2.1.5.</p> Development Rank 1|hzvvsn: Rank (Obsolete) 9290 Severity