https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1217 Lustre <p>system crashed with following assertion </p> <p>LustreError: 35225:0:(osc_page.c:293:osc_page_putref_lock()) ASSERTION(cfs_atomic_read(&amp;lock-&gt;cll_ref) &gt; 1 || olock-&gt;ols_hold == 0) failed<br/> LustreError: 35225:0:(osc_page.c:293:osc_page_putref_lock()) LBUG<br/> Pid: 35225, comm: ptlrpcd_6</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0369855&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0369e95&gt;&#93;</span> lbug_with_loc+0x75/0xe0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0375fe6&gt;&#93;</span> libcfs_assertion_failed+0x66/0x70 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa064ca06&gt;&#93;</span> osc_page_putref_lock+0xb6/0xc0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa064ca28&gt;&#93;</span> osc_page_completion_read+0x18/0x20 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04555af&gt;&#93;</span> cl_page_completion+0x21f/0x550 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa045610b&gt;&#93;</span> ? cl_page_put+0x1ab/0x440 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa053b604&gt;&#93;</span> ? lustre_msg_get_opc+0x94/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa045f1e8&gt;&#93;</span> ? cl_req_page_done+0x38/0x180 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa064d145&gt;&#93;</span> osc_completion+0x165/0x380 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa055d734&gt;&#93;</span> ? at_measured+0x114/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa054099e&gt;&#93;</span> ? ptlrpc_master_callback+0x3e/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0634f48&gt;&#93;</span> osc_ap_completion+0x128/0x300 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0641cb9&gt;&#93;</span> brw_interpret+0x4a9/0x710 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa052c066&gt;&#93;</span> ? after_reply+0x7b6/0xe30 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810017ac&gt;&#93;</span> ? __switch_to+0x1ac/0x320<br/> <span class="error">&#91;&lt;ffffffff81041760&gt;&#93;</span> ? fair___dequeue_entity+0x30/0x50<br/> <span class="error">&#91;&lt;ffffffffa0530c0a&gt;&#93;</span> ptlrpc_check_set+0x48a/0x1ba0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810657ec&gt;&#93;</span> ? lock_timer_base+0x3c/0x70<br/> <span class="error">&#91;&lt;ffffffffa0565147&gt;&#93;</span> ptlrpcd_check+0x547/0x560 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa056543c&gt;&#93;</span> ptlrpcd+0x2dc/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8104c780&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa0565160&gt;&#93;</span> ? ptlrpcd+0x0/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810041aa&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa0565160&gt;&#93;</span> ? ptlrpcd+0x0/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0565160&gt;&#93;</span> ? ptlrpcd+0x0/0x460 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810041a0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Unless I did something really wrong, <br/> cfs_atomic_read(&amp;lock-&gt;cll_ref) = cfs_atomic_read(0xffff881dea5c9e78) = 258</p> <p>At time of crash, the system was higly loaded with a lot of interactive commands and did suffer of disconnection from some OST &amp; MDT.</p> LU-1217

ASSERTION(cfs_atomic_read(&lock->cll_ref) > 1 || olock->ols_hold == 0) failed

Bug Minor Resolved Fixed Jinshan Xiong Alexandre Louvet Wed, 14 Mar 2012 15:55:49 +0000 Mon, 4 Jun 2012 04:43:29 +0000 Mon, 4 Jun 2012 04:43:29 +0000 Lustre 2.1.0 Lustre 2.3.0 Lustre 2.1.2 0 8 <p>Bobi</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>it's impossible for this atomic count to be 258 if it hit this assertion. I don't know why it's `cfs_atomic_read(&amp;lock-&gt;cll_ref) &gt; 1' instead of `&gt; 0'.</p> <p>Alexandre</p> <p>You mentioned on yesterday's call that it would be possible to supply a crash dump for this ticket. That would definitely help expedite us understanding this scenario.</p> <p>Thanks</p> <p>Peter</p> <p>Would you mind uploading crash image/kernel image/symbol files to our ftp site?</p> <p>We hit this today as well on one of our classified 2.1 clients. We have a crash dump, but can't upload it. I'm happy to do any analysis of the dump at your direction.</p> <p>first, we need to know the troublesome cl_lock's content.</p> <blockquote><p>first, we need to know the troublesome cl_lock's content.</p></blockquote> <p>I've only been able to find one address on the stack that looks like a cl_lock. But, with lock-&gt;cll_ref = 99 it shouldn't have hit the assertion. Maybe something touched that memory after the atomic read?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>struct cl_lock { cll_ref = { counter = 99 }, cll_layers = { next = 0xffff88... prev = 0xffff88... }, cll_linkage = { next = 0xffff88... prev = 0xffff88... }, cll_descr = { cld_obj = 0xffff88... cld_start = 0, cld_end = 18446744073709551615, cld_gid = 0, cld_mode = CLM_READ, cld_enq_flags = 0 }, cll_state = CLS_HELD, cll_wq = { lock = { raw_lock = { slock = 169675293 } }, task_list = { next = 0xffff88... prev = 0xffff88... } }, cll_guard = { count = { counter = 1 }, wait_lock = { raw_lock = { slock = 1179666 } }, wait_list = { next = 0xffff88... prev = 0xffff88... }, owner = 0x0 }, cll_guarder = 0x0, cll_depth = 0, cll_intransit_owner = 0x0, cll_error = 0, cll_holds = 1, cll_users = 1, cll_flags = 0, cll_inclosure = { next = 0xffff88... prev = 0xffff88... }, cll_conflict = 0x0, cll_reference = {&lt;No data fields&gt;}, cll_holders = {&lt;No data fields&gt;}, cll_obj_ref = 0x0 } </pre> </div></div> <p>let's try to print the lock state precisely with patch: <a href="http://review.whamcloud.com/#change,2383" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2383</a></p> <p>OK, but it will be several weeks before we can update this system, and even then the bug may be rare to reproduce.</p> <p>from the attached crash dump, the lock looks okay. I don't know why.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; p *(struct osc_lock *)0xffff881fc381a140 $28 = { ols_cl = { cls_lock = 0xffff881dea5c9e78, cls_obj = 0xffff881f30d9e4e0, cls_ops = 0xffffffffa06537e0, cls_linkage = { next = 0xffff881dea5c9e80, prev = 0xffff881e8c1e8938 } }, ols_lock = 0xffff881e974df240, ols_lvb = { lvb_size = 1443889168, lvb_mtime = 1329614555, lvb_atime = 1329614548, lvb_ctime = 1329614555, lvb_blocks = 2820112 }, ols_flags = 1052673, ols_handle = { cookie = 8399946539139540594 }, ols_einfo = { ei_type = 11, ei_mode = 4, ei_cb_bl = 0xffffffffa064ee10, ei_cb_cp = 0xffffffffa064fcb0, ei_cb_gl = 0xffffffffa064ea50, ei_cb_wg = 0xffffffffa064e7a0, ei_cbdata = 0xffff881fc381a140, ei_async = 0 }, ols_state = OLS_GRANTED, ols_pageref = { counter = 256 }, ols_hold = 1, ols_has_ref = 1, ols_locklessable = 0, ols_ast_wait = 0, ols_flush = 0, ols_glimpse = 0, ols_owner = 0xffff881aac5471d0 } crash&gt; p *(struct cl_lock *)0xffff881dea5c9e78 $29 = { cll_ref = { counter = 258 }, cll_layers = { next = 0xffff881e8c1e8938, prev = 0xffff881fc381a158 }, cll_linkage = { next = 0xffff8819d4c0df38, prev = 0xffff8819d4c0df38 }, cll_descr = { cld_obj = 0xffff8819d4c0df58, cld_start = 0, cld_end = 18446744073709551615, cld_gid = 0, cld_mode = CLM_READ, cld_enq_flags = 0 }, cll_state = CLS_HELD, cll_wq = { lock = { raw_lock = { slock = 76350605 } }, task_list = { next = 0xffff881dea5c9ed8, prev = 0xffff881dea5c9ed8 } }, cll_guard = { count = { counter = 1 }, wait_lock = { raw_lock = { slock = 0 } }, wait_list = { next = 0xffff881dea5c9ef0, prev = 0xffff881dea5c9ef0 }, owner = 0x0 }, cll_guarder = 0x0, cll_depth = 0, cll_intransit_owner = 0x0, cll_error = 0, cll_holds = 1, cll_users = 1, cll_flags = 0, cll_inclosure = { next = 0xffff881dea5c9f38, prev = 0xffff881dea5c9f38 }, cll_conflict = 0x0, cll_reference = {&lt;No data fields&gt;}, cll_holders = {&lt;No data fields&gt;}, cll_obj_ref = 0x0 } </pre> </div></div> <p>Ned, do you also apply multi-threaded ptlrpcd patch on top of your 2.1 tree?</p> <p>No, we do not have multi-threaded pltrpcd.</p> <p><a href="https://github.com/chaos/lustre/tree/2.1.0-llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/tree/2.1.0-llnl</a></p> <p>I have no idea what happened. Can you please apply the patch I mentioned so that we can see what was exactly the lock state?</p> <p>Hi Jinshan, we have applied the patch to our tree and it will by rolled out in the coming weeks.</p> <p>A sysadmin got a hit on a client on our secure network with your debugging patch applied. He could not cut-and-paste. What I know so far is just that your patch reported that the olock pointer is non-null and hold is 1. If there is something specific that you want from CL_LOCK_DEBUG, let me know and I'll either have the admin transcribe it or a developer can do it when we return from LUG.</p> <p>I will need the detailed info of the lock to address the problem.</p> <p>Ah, yes, I see all values that cl_lock_print includes. We'll need Ned to copy those into this ticket when he gets back from LUG.</p> <p>Yes, I knew this from your previous comment either. Let's do it after you guys come back from LUG.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lock@ffff8803948bc978 [2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100cf000:0x3e138b:0x0] { lovsub@ffff8804340bd9a0 [0 ffff8804340db9a0: [ 0 ffff8801679d2d60 R(1):[256, 18446744073709551615]@[0x3f5c87bae:0x6a:0x0]] osc@ffff88041a6671f8: ffff880429e7eb40 00101001 0xa29f718631a3c48d 3 ffff8803b773f2b8 size: 367001600 mtime: 1335282921 atime: 1335282807 ctime: 1335282921 blocks: 716808 } lock@ffff8803948bc978 olock: ffff88041a6671f8, hold: 1. PID: 14280 COMMAND: "ptlrpcd-brw" machine_kexec crash_kexec panic lbug_with_loc libcfs_assertion_failed osc_page_putref_lock osc_page_completion_read cl_page_completion osc_completion osc_ap_completion brw_interpret ptlrpc_check_set ptlrpcd_check ptlrpcd kernel_thread </pre> </div></div> <p>Ned, thank you for the information.</p> <p>I realize this problem is a race between lock use and assertion check as following:<br/> 1. this page held the last refcount of the lock;<br/> 2. in osc_page_putref_lock(), cfs_atomic_read(&amp;lock-&gt;cll_ref) == 1 was true, but it was delayed for a while before checking ols_hold due to interrupt...<br/> 3. another proc calls cl_use() to bring this lock into CLS_HELD state, so ols_hold is set;<br/> 4. osc_page_putref_lock() continued to check ols_hold, definitely it was set and of course lock's refcount should be greater 1 for sure at this moment.</p> <p>Fix this problem is easy - just to remove the assertion.</p> <p>Please apply patch: <a href="http://review.whamcloud.com/2604" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2604</a>.</p> <p>Will do.</p> <p>Landed for 2.1.2 and 2.3</p> Development Rank 1|hzv6mv: Rank (Obsolete) 4609 Severity