https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2056 Lustre <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&lt;ConMan&gt; Console [hyperion-rst6] log at 2012-09-30 16:00:00 PDT. 2012-09-30 16:20:23 [A[B[1;2B[1;2B[1;2B[1;2B[1;2B[B[B[B[B[B[BLustre: 4162:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1349048656/real 1349048656] req@ffff880176df8c00 x1414547178955781/t0(0) o400-&gt;MGC192.168.127.6@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1349048762 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1 2012-09-30 16:46:02 LustreError: 166-1: MGC192.168.127.6@o2ib: Connection to MGS (at 0@lo) was lost; in progress operations using <span class="code-keyword">this</span> service will fail 2012-09-30 16:46:02 Lustre: MGS: Client b825c79a-32bd-3ab8-c5fa-b66aeb9bb741 (at 0@lo) reconnecting 2012-09-30 16:46:02 LustreError: 4242:0:(obd_class.h:527:obd_set_info_async()) obd_set_info_async: dev 0 no operation 2012-09-30 16:47:09 Lustre: MGC192.168.127.6@o2ib: Reactivating <span class="code-keyword">import</span> 2012-09-30 16:47:09 Lustre: MGC192.168.127.6@o2ib: Connection restored to MGS (at 0@lo) &lt;ConMan&gt; Console [hyperion-rst6] log at 2012-09-30 17:00:00 PDT. 2012-09-30 17:02:38 LustreError: 6069:0:(osd_handler.c:837:osd_trans_start()) ASSERTION( oti-&gt;oti_w_locks == 0 ) failed: 2012-09-30 17:02:38 LustreError: 6069:0:(osd_handler.c:837:osd_trans_start()) LBUG 2012-09-30 17:02:38 Pid: 6069, comm: mdt_rdpg03_002 2012-09-30 17:02:38 </pre> </div></div> <p>For some reason, stack did not dump, but vmcore obtained</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; bt PID: 6069 TASK: ffff880141836aa0 CPU: 7 COMMAND: <span class="code-quote">"mdt_rdpg03_002"</span> #0 [ffff880156e759c8] machine_kexec at ffffffff8103281b #1 [ffff880156e75a28] crash_kexec at ffffffff810ba792 #2 [ffff880156e75af8] panic at ffffffff814fd591 #3 [ffff880156e75b78] lbug_with_loc at ffffffffa0393f6b [libcfs] #4 [ffff880156e75b98] osd_trans_start at ffffffffa0a8d2bc [osd_ldiskfs] #5 [ffff880156e75bd8] mdd_trans_start at ffffffffa0f043a4 [mdd] #6 [ffff880156e75be8] mdd_close at ffffffffa0edf4f6 [mdd] #7 [ffff880156e75c58] cml_close at ffffffffa06baef6 [cmm] #8 [ffff880156e75c88] mdt_mfd_close at ffffffffa0f8a18e [mdt] #9 [ffff880156e75ce8] mdt_close at ffffffffa0f8ae0a [mdt] #10 [ffff880156e75d38] mdt_handle_common at ffffffffa0f66802 [mdt] #11 [ffff880156e75d88] mdt_readpage_handle at ffffffffa0f676d5 [mdt] #12 [ffff880156e75d98] ptlrpc_server_handle_request at ffffffffa0966b3c [ptlrpc] #13 [ffff880156e75e98] ptlrpc_main at ffffffffa0968111 [ptlrpc] #14 [ffff880156e75f48] kernel_thread at ffffffff8100c14a </pre> </div></div> LLNL/Hyperion lustre-reviews 9573 LU-2056

SWL - osd_trans_start()) ASSERTION( oti->oti_w_locks == 0 ) failed:

Bug Blocker Resolved Fixed Hongchao Zhang Cliff White HB Sun, 30 Sep 2012 21:30:33 +0000 Fri, 19 Apr 2013 14:20:21 +0000 Mon, 14 Jan 2013 15:21:43 +0000 Lustre 2.4.0 Lustre 2.4.0 0 5 <p>VMcore is at ~cliffw/lu2056/vmcore-rst6.01Oct on brent</p> <p>vmcore is at ~cliffw/lu2056/vmcore-rst6.01Oct on brent</p> <p>comment in osd_trans_start()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* * XXX: current rule is that we first start tx, * then lock object(s), but we can't use * this rule for data (due to locking specifics * in ldiskfs). also in long-term we'd like to * use usually-used (locks;tx) ordering. so, * UGLY thing is that we'll use one ordering for * data (ofd) and reverse ordering for metadata * (mdd). then at some point we'll fix the latter */ if (dev-&gt;od_is_md) { LASSERT(oti-&gt;oti_r_locks == 0); LASSERT(oti-&gt;oti_w_locks == 0); } </pre> </div></div> <p>But code in mdd_close()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ..... 2457 mdd_write_lock(env, mdd_obj, MOR_TGT_CHILD); 2458 if (handle == NULL &amp;&amp; mdd_obj-&gt;mod_count == 1 &amp;&amp; 2459 (mdd_obj-&gt;mod_flags &amp; ORPHAN_OBJ) != 0) { 2460 mdd_write_unlock(env, mdd_obj); 2461 goto again; 2462 } ...... 2503 if (handle == NULL) { 2504 handle = mdd_trans_create(env, mdo2mdd(obj) ); 2505 if (IS_ERR(handle)) 2506 GOTO(out, rc = PTR_ERR(handle)); 2507 2508 rc = mdd_declare_object_kill(env, mdd_obj, ma, 2509 handle); 2510 if (rc) 2511 GOTO(out, rc); 2512 2513 rc = mdd_declare_changelog_store(env, mdd, 2514 NULL, hand le); 2515 if (rc) 2516 GOTO(stop, rc); 2517 2518 rc = mdd_trans_start(env, mdo2mdd(obj), han dle); </pre> </div></div> <p>We start transaction <em>after</em> locking the object, which violates the rule set in osd_trans_start().</p> <p>I've posted a patch for this: <a href="http://review.whamcloud.com/4145" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4145</a> , it's better to try it after it's reviewed.</p> <p>I made some changes for wangdi's review and rebased previous patch to 2.3 (previous one is for master), unfortunately I lost history of it by doing this. New patch is: <a href="http://review.whamcloud.com/#change,4152" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4152</a></p> <p>patch landed to 2.3 so it's not a 2.3 blocker anymore, but still need to fix it for master</p> <p>the patch against master is tracked at <a href="http://review.whamcloud.com/#change,4461" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4461</a></p> <p>I seem to be hitting this (in form of <a href="https://jira.whamcloud.com/browse/LU-2443" title="osd_object_write_lock()) ASSERTION( obj-&gt;oo_owner != env )" class="issue-link" data-issue-key="LU-2443"><del>LU-2443</del></a> during racer shutdown) fairly regularly now, please update the patch.</p> <p>reassign to hongchao because he ported the patch to master</p> <p>Since Change, 4461 has landed, can this ticket be closed?</p> <p>All patches are landed so closing ticket. Please reopen if more work is needed.</p> Development Rank 1|hzv4q7: Rank (Obsolete) 4299 Severity