https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7619 Lustre <p>Some quite old patches have introduced several REFASSERT()s, to strengthen lu_ref feature controls, but not under required lu_ref::lf_guard spin-lock protection.<br/> This can cause unexpected "self" dead-lock situations (vs LBUG/ASSERT!!) upon lu_ref failure detection.<br/> This has been unveiled during my own usage of the feature to track and debug some unreferencing bug/lack.</p> LU-7619

USE_LU_REF/lu_ref feature broken after some REFASSERT()s have been added without lu_ref::lf_guard protection

Bug Minor Resolved Fixed Bruno Faccini Bruno Faccini Wed, 30 Dec 2015 16:20:09 +0000 Wed, 13 Jul 2016 18:31:43 +0000 Wed, 13 Jul 2016 18:31:43 +0000 Lustre 2.9.0 0 3 <p>For info, the specific dead-lock situation I have encountered when using lu_ref failure and caused by this bug had the following signature/stack :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;1&gt;LustreError: dumping log to /tmp/lustre-log.1451225788.113113 &lt;0&gt;BUG: soft lockup - CPU#1 stuck for 67s! [ldlm_bl_03:113432] &lt;4&gt;Modules linked in: lustre(U) ofd(U) osp(U) lod(U) ost(U) mdt(U) mdd(U) mgs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) lfsck(U) jbd2 obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) ksocklnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm sg joydev microcode iTCO_wdt iTCO_vendor_support igb i2c_algo_bit sb_edac edac_core i2c_i801 i2c_core lpc_ich mfd_core ioatdma dca shpchp ext3 jbd mbcache sd_mod crc_t10dif isci libsas scsi_transport_sas mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] &lt;4&gt;CPU 1 &lt;4&gt;Modules linked in: lustre(U) ofd(U) osp(U) lod(U) ost(U) mdt(U) mdd(U) mgs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) lfsck(U) jbd2 obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) ksocklnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm sg joydev microcode iTCO_wdt iTCO_vendor_support igb i2c_algo_bit sb_edac edac_core i2c_i801 i2c_core lpc_ich mfd_core ioatdma dca shpchp ext3 jbd mbcache sd_mod crc_t10dif isci libsas scsi_transport_sas mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] &lt;4&gt; &lt;4&gt;Pid: 113432, comm: ldlm_bl_03 Not tainted 2.6.32.573.8.1.el6_lustre #1 Intel Corporation S2600GZ/S2600GZ &lt;4&gt;RIP: 0010:[&lt;ffffffff8153cdfc&gt;] [&lt;ffffffff8153cdfc&gt;] _spin_lock+0x1c/0x30 &lt;4&gt;RSP: 0018:ffff88030307ba30 EFLAGS: 00000297 &lt;4&gt;RAX: 000000000000059c RBX: ffff88030307ba30 RCX: 0000000000000000 &lt;4&gt;RDX: 000000000000059b RSI: 0000000000000000 RDI: ffffffffa071db68 &lt;4&gt;RBP: ffffffff8100bc0e R08: 00000000fffffffe R09: 0000000000000000 &lt;4&gt;R10: 000000000000000f R11: 000000000000000f R12: ffff8802fac8f970 &lt;4&gt;R13: ffff8803030df060 R14: 0000000000000097 R15: 00000000fffffffc &lt;4&gt;FS: 0000000000000000(0000) GS:ffff880038620000(0000) knlGS:0000000000000000 &lt;4&gt;CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b &lt;4&gt;CR2: 0000000002527c1c CR3: 0000000001a8d000 CR4: 00000000000407e0 &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;Process ldlm_bl_03 (pid: 113432, threadinfo ffff880303078000, task ffff8803f1eb4040) &lt;4&gt;Stack: &lt;4&gt; ffff88030307ba50 ffffffffa05fb78a ffff8803030df060 ffff8803030df060 &lt;4&gt;&lt;d&gt; ffff88030307ba70 ffffffffa05fb9f2 ffff88030307ba70 ffff8803030df000 &lt;4&gt;&lt;d&gt; ffff88030307bab0 ffffffffa05f3955 ffff8802fae12888 ffff8803030df000 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa05fb78a&gt;] ? lu_ref_print_all+0x1a/0x80 [obdclass] &lt;4&gt; [&lt;ffffffffa05fb9f2&gt;] ? lu_ref_fini+0x82/0x170 [obdclass] &lt;4&gt; [&lt;ffffffffa05f3955&gt;] ? cl_page_free+0xe5/0x540 [obdclass] &lt;4&gt; [&lt;ffffffffa05f3f5c&gt;] ? cl_page_put+0x1ac/0x3e0 [obdclass] &lt;4&gt; [&lt;ffffffffa05fc179&gt;] ? lu_ref_del+0x109/0x2c0 [obdclass] &lt;4&gt; [&lt;ffffffffa0aa709c&gt;] ? osc_page_gang_lookup+0x1dc/0x380 [osc] &lt;4&gt; [&lt;ffffffffa0aa6b80&gt;] ? discard_cb+0x0/0x190 [osc] &lt;4&gt; [&lt;ffffffffa0aa7384&gt;] ? osc_lock_discard_pages+0x144/0x240 [osc] &lt;4&gt; [&lt;ffffffffa0a9d8b5&gt;] ? osc_lock_flush+0x55/0x260 [osc] &lt;4&gt; [&lt;ffffffffa0aa6b80&gt;] ? discard_cb+0x0/0x190 [osc] &lt;4&gt; [&lt;ffffffffa0a9d8eb&gt;] ? osc_lock_flush+0x8b/0x260 [osc] &lt;4&gt; [&lt;ffffffffa0a9dd68&gt;] ? osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa07c34bc&gt;] ? ldlm_cancel_callback+0x6c/0x1a0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa07de40a&gt;] ? ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa07e2fac&gt;] ? ldlm_cli_cancel+0x7c/0x380 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0a9db9b&gt;] ? osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa04889f1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] &lt;4&gt; [&lt;ffffffffa07e6ea0&gt;] ? ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] &lt;4&gt; [&lt;ffffffffa07e81a1&gt;] ? ldlm_bl_thread_main+0x271/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff810672b0&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt; [&lt;ffffffffa07e7f30&gt;] ? ldlm_bl_thread_main+0x0/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff810a0fce&gt;] ? kthread+0x9e/0xc0 &lt;4&gt; [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff810a0f30&gt;] ? kthread+0x0/0xc0 &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17756" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17756</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7619" title="USE_LU_REF/lu_ref feature broken after some REFASSERT()s have been added without lu_ref::lf_guard protection" class="issue-link" data-issue-key="LU-7619"><del>LU-7619</del></a> obdclass: protect REFASSERT() with lu_ref::lf_guard<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 71ab9f599d2035aac6d906be3402e7b0f8aa9044</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17756/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17756/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7619" title="USE_LU_REF/lu_ref feature broken after some REFASSERT()s have been added without lu_ref::lf_guard protection" class="issue-link" data-issue-key="LU-7619"><del>LU-7619</del></a> obdclass: protect REFASSERT() with lu_ref::lf_guard<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 838514369ac245d2dcfdcda7715a2798fe9ee755</p> <p>Patch landed to master for 2.9.0</p> Development Rank 1|hzxx1z: Rank (Obsolete) 9223372036854775807 Severity