https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6846 Lustre <p>Found this in 24 hours failover test in OpenSFS cluster.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: ==== Checking the clients loads AFTER failover -- failure NOT OK Lustre: DEBUG MARKER: mds7 has failed over 1 times, and counting... Lustre: 9887:0:(client.c:2018:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1436839484/real 1436839484] req@ffff88062c53a080 x1506632146392040/t0(0) o400-&gt;lustre-MDT0006-osp-MDT0001@192.168.2.127@o2ib:24/4 lens 224/224 e 1 to 1 dl 1436839486 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 Lustre: 9887:0:(client.c:2018:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: 9887:0:(client.c:2018:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1436839503/real 1436839503] req@ffff88062c53a080 x1506632146392208/t0(0) o400-&gt;lustre-MDT0006-osp-MDT0001@192.168.2.127@o2ib:24/4 lens 224/224 e 1 to 1 dl 1436839505 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 Lustre: 9887:0:(client.c:2018:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: lustre-MDT0006-osp-MDT0001: Connection restored to lustre-MDT0006 (at 192.168.2.127@o2ib) LustreError: 12030:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: LustreError: 12030:0:(dt_object.c:512:dt_record_write()) LBUG Pid: 12030, comm: mdt_out03_005 Call Trace: [&lt;ffffffffa0506875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0506e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa065978f&gt;] dt_record_write+0xbf/0x130 [obdclass] [&lt;ffffffffa08f4d0e&gt;] out_tx_write_exec+0x7e/0x300 [ptlrpc] [&lt;ffffffffa08ed30a&gt;] out_tx_end+0xda/0x5d0 [ptlrpc] [&lt;ffffffffa08f1e7b&gt;] out_handle+0xd9b/0x17e0 [ptlrpc] [&lt;ffffffffa083afb0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa08ea212&gt;] tgt_request_handle+0xa42/0x1230 [ptlrpc] [&lt;ffffffffa0892891&gt;] ptlrpc_main+0xe41/0x1920 [ptlrpc] [&lt;ffffffffa0891a50&gt;] ? ptlrpc_main+0x0/0x1920 [ptlrpc] [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1436839511.12030 </pre> </div></div> LU-6846

dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed:

Bug Major Resolved Fixed Di Wang Di Wang dne2 Tue, 14 Jul 2015 03:23:26 +0000 Tue, 28 Mar 2017 18:20:32 +0000 Sat, 19 Sep 2015 05:35:45 +0000 Lustre 2.8.0 Lustre 2.8.0 0 9 <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15595" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15595</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> osd: reset do_body_ops after creation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 981452bf05fbe3e7ac06f8289cd309abf12d41bb</p> <p>Clearly, this fix is not enough, i.e. even with this fix, we still found the LBUG in hyperion</p> {noforrmat} <p>2015-07-23 12:40:30 Lustre: 6294:0:(service.c:2102:ptlrpc_server_handle_request()) Skipped 1 previous similar message<br/> 2015-07-23 12:40:42 Lustre: lustre-MDT0003-osp-MDT0001: Connection restored to lustre-MDT0003 (at 192.168.120.10@o2ib)<br/> 2015-07-23 12:43:44 LustreError: 6700:0:(dt_object.c:513:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: FID <span class="error">&#91;0x280000402:0x3:0x0&#93;</span><br/> 2015-07-23 12:43:44 LustreError: 6700:0:(dt_object.c:513:dt_record_write()) LBUG<br/> 2015-07-23 12:43:44 Pid: 6700, comm: mdt_out02_002<br/> 2015-07-23 12:43:44<br/> 2015-07-23 12:43:44 Call Trace:<br/> 2015-07-23 12:43:44 <span class="error">&#91;&lt;ffffffffa049e875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2015-07-23 12:43:44 <span class="error">&#91;&lt;ffffffffa049ee77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-07-23 12:43:44 <span class="error">&#91;&lt;ffffffffa063849f&gt;&#93;</span> dt_record_write+0xcf/0x140 <span class="error">&#91;obdclass&#93;</span><br/> 2015-07-23 12:43:44 <span class="error">&#91;&lt;ffffffffa08d32de&gt;&#93;</span> out_tx_write_exec+0x7e/0x300 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:44 <span class="error">&#91;&lt;ffffffffa08cb8da&gt;&#93;</span> out_tx_end+0xda/0x5d0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffffa08d044b&gt;&#93;</span> out_handle+0xd9b/0x17e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffffa0819dd0&gt;&#93;</span> ? target_bulk_timeout+0x0/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffffa08c87e2&gt;&#93;</span> tgt_request_handle+0xa42/0x1230 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffffa0871201&gt;&#93;</span> ptlrpc_main+0xe41/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffff8152a2be&gt;&#93;</span> ? thread_return+0x4e/0x7d0<br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffffa08703c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffff8109e78e&gt;&#93;</span> kthread+0x9e/0xc0<br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffff8109e6f0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> 2015-07-23 12:43:45 <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 2015-07-23 12:43:45</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre></pre> </div></div> <p>It seems this is because of the race between llog cancel rec and llog destroy object. In llog_cancel_rec()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/* returns negative on error; 0 if success; 1 if success &amp; log destroyed */ int llog_cancel_rec(const struct lu_env *env, struct llog_handle *loghandle, int index) { struct llog_log_hdr *llh = loghandle-&gt;lgh_hdr; int rc = 0; ENTRY; CDEBUG(D_RPCTRACE, "Canceling %d in log "DOSTID"\n", index, POSTID(&amp;loghandle-&gt;lgh_id.lgl_oi)); if (index == 0) { CERROR("Can't cancel index 0 which is header\n"); RETURN(-EINVAL); } rc = llog_cancel_rec_internal(env, loghandle, index); if (rc &lt; 0) { CERROR("%s: fail to write header for llog #"DOSTID "#%08x: rc = %d\n", loghandle-&gt;lgh_ctxt-&gt;loc_obd-&gt;obd_name, POSTID(&amp;loghandle-&gt;lgh_id.lgl_oi), loghandle-&gt;lgh_id.lgl_ogen, rc); RETURN(rc); } if ((llh-&gt;llh_flags &amp; LLOG_F_ZAP_WHEN_EMPTY) &amp;&amp; (llh-&gt;llh_count == 1) &amp;&amp; (loghandle-&gt;lgh_last_idx == LLOG_HDR_BITMAP_SIZE(llh) - 1)) { rc = llog_destroy(env, loghandle); if (rc &lt; 0) { /* Sigh, can not destroy the final plain llog, but * the bitmap has been clearly, so the record can not * be accessed anymore, let's return 0 for now, and </pre> </div></div> <p>cancel record and log_destroy are in different transaction. So when cancel the remote record, if the RPC is out of order, i.e. cancel record arrives later than llog_destroy, then we will met this bug. So the fix should be merge them into a single transaction.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15730" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15730</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: combine cancel rec and destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 649af4d6ac28378b182d18f1dc4c45d46127f74d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15595/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15595/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> osd: reset do_body_ops after creation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 037759bc49ce7f6adc9acb4a748a25c0f4c4ab64</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15730/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15730/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: combine cancel rec and destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a96be7018d90585dce16ce4b126485ce41629cd2</p> <p>Landed for 2.8</p> <p>We are hitting this on Hyperion with 2.7.58</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 22 02:34:36 iws12 kernel: LustreError: 11526:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Aug 22 02:34:36 iws12 kernel: LustreError: 11526:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Aug 22 02:34:36 iws12 kernel: LustreError: 11526:0:(dt_object.c:512:dt_record_write()) LBUG Aug 22 02:34:36 iws12 kernel: LustreError: 11526:0:(dt_object.c:512:dt_record_write()) LBUG Aug 22 02:34:36 iws12 kernel: Pid: 11526, comm: mdt_out03_005 Aug 22 02:34:36 iws12 kernel: Aug 22 02:34:36 iws12 kernel: Call Trace: Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa04a6875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa04a6e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa09a658f&gt;] dt_record_write+0xbf/0x130 [obdclass] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0c428ce&gt;] out_tx_write_exec+0x7e/0x300 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0c3aeca&gt;] out_tx_end+0xda/0x5d0 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0c3fa3b&gt;] out_handle+0xd9b/0x17e0 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0b88010&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0c37dc2&gt;] tgt_request_handle+0xa42/0x1230 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0bdff91&gt;] ptlrpc_main+0xe41/0x1920 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffff8152a39e&gt;] ? thread_return+0x4e/0x7d0 Aug 22 02:34:36 iws12 kernel: [&lt;ffffffffa0bdf150&gt;] ? ptlrpc_main+0x0/0x1920 [ptlrpc] Aug 22 02:34:36 iws12 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Aug 22 02:34:36 iws12 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Aug 22 02:34:36 iws12 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Aug 22 02:34:36 iws12 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Aug 22 02:34:36 iws12 kernel: </pre> </div></div> <p>Hit again</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 1 17:42:47 iws12 kernel: LustreError: 7434:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Sep 1 17:42:47 iws12 kernel: LustreError: 7434:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Sep 1 17:42:47 iws12 kernel: LustreError: 7434:0:(dt_object.c:512:dt_record_write()) LBUG Sep 1 17:42:47 iws12 kernel: LustreError: 7434:0:(dt_object.c:512:dt_record_write()) LBUG Sep 1 17:42:47 iws12 kernel: Pid: 7434, comm: mdt_out03_020 Sep 1 17:42:47 iws12 kernel: Sep 1 17:42:47 iws12 kernel: Call Trace: Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa04c9875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa04c9e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa09c350f&gt;] dt_record_write+0xbf/0x130 [obdclass] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0c6144e&gt;] out_tx_write_exec+0x7e/0x300 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0c59a4a&gt;] out_tx_end+0xda/0x5d0 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0c5e5bb&gt;] out_handle+0xd9b/0x17e0 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0ba61d0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0c568ec&gt;] tgt_request_handle+0xa4c/0x1290 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0bfe5b1&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffffa0bfd770&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] Sep 1 17:42:47 iws12 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Sep 1 17:42:47 iws12 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Sep 1 17:42:47 iws12 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Sep 1 17:42:47 iws12 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Lustre-log dump</p> <p>I hit this bug over the week end as well.</p> <p>And we have hit it with 2.7.59</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 5 14:54:15 iws10 kernel: LustreError: 52672:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Sep 5 14:54:15 iws10 kernel: LustreError: 52672:0:(dt_object.c:512:dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: Sep 5 14:54:15 iws10 kernel: LustreError: 52672:0:(dt_object.c:512:dt_record_write()) LBUG Sep 5 14:54:15 iws10 kernel: LustreError: 52672:0:(dt_object.c:512:dt_record_write()) LBUG Sep 5 14:54:15 iws10 kernel: Pid: 52672, comm: mdt_out02_018 Sep 5 14:54:15 iws10 kernel: Sep 5 14:54:15 iws10 kernel: Call Trace: Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa07e4875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa07e4e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa09e12df&gt;] dt_record_write+0xbf/0x130 [obdclass] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c7f4ae&gt;] out_tx_write_exec+0x7e/0x300 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c77aaa&gt;] out_tx_end+0xda/0x5d0 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c7c61b&gt;] out_handle+0xd9b/0x17e0 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0bc41d0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c7494c&gt;] tgt_request_handle+0xa4c/0x1290 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c1c5b1&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffffa0c1b770&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] Sep 5 14:54:15 iws10 kernel: [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 Sep 5 14:54:15 iws10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Sep 5 14:54:15 iws10 kernel: [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 Sep 5 14:54:15 iws10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16333" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16333</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: create remote llog synchronously<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 22504242f5646e34feda559e33d544d8354e0fb1</p> <p>Cliff, James: Please try this patch. Thanks</p> <p>Just did. Now with sanity test 60 I get the follow exception:</p> <p>Sep 9 13:17:09 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12210.969737&#93;</span> Lustre: 9313:0:(llog_test.c:1010:llog_test_7()) <br/> 7a: test llog_logid_rec<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.888955&#93;</span> LustreError: 9313:0:(llog_test.c:80:verify_handl<br/> e()) 7_sub: expected 1 active recs after write, found 2<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.899565&#93;</span> LustreError: 9313:0:(osd_handler.c:2488:osd_obje<br/> ct_destroy()) ASSERTION( !lu_object_is_dying(dt-&gt;do_lu.lo_header) ) failed:<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.911973&#93;</span> LustreError: 9313:0:(osd_handler.c:2488:osd_obje<br/> ct_destroy()) LBUG<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.919387&#93;</span> Pid: 9313, comm: lctl<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.922812&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.922813&#93;</span> Call Trace:<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.926946&#93;</span> <span class="error">&#91;&lt;ffffffffa0448875&gt;&#93;</span> libcfs_debug_dumpstack+0x5<br/> 5/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.933996&#93;</span> <span class="error">&#91;&lt;ffffffffa0448e77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 [l<br/> ibcfs]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.940297&#93;</span> <span class="error">&#91;&lt;ffffffffa0e9e80b&gt;&#93;</span> osd_object_destroy+0x51b/0<br/> x5a0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.947683&#93;</span> <span class="error">&#91;&lt;ffffffffa0e9d177&gt;&#93;</span> ? osd_object_ref_del+0x187<br/> /0x4e0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.955277&#93;</span> <span class="error">&#91;&lt;ffffffffa058ddba&gt;&#93;</span> llog_osd_destroy+0x1ba/0x9<br/> d0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.962177&#93;</span> <span class="error">&#91;&lt;ffffffffa057d686&gt;&#93;</span> llog_destroy+0x2b6/0x470 [<br/> obdclass]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.968712&#93;</span> <span class="error">&#91;&lt;ffffffffa0527ccc&gt;&#93;</span> llog_test_7_sub+0x29c/0x92<br/> 0 <span class="error">&#91;llog_test&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.975597&#93;</span> <span class="error">&#91;&lt;ffffffffa0528eaf&gt;&#93;</span> llog_test_7+0x13f/0xa10 [l<br/> log_test]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.982111&#93;</span> <span class="error">&#91;&lt;ffffffffa052dbb5&gt;&#93;</span> llog_run_tests+0x1355/0x1d<br/> 40 <span class="error">&#91;llog_test&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.989099&#93;</span> <span class="error">&#91;&lt;ffffffffa0582b7e&gt;&#93;</span> ? llog_setup+0x4ae/0x8b0 [<br/> obdclass]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12211.995643&#93;</span> <span class="error">&#91;&lt;ffffffffa052ed49&gt;&#93;</span> llog_test_setup+0x7a9/0x8f<br/> 0 <span class="error">&#91;llog_test&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.002538&#93;</span> <span class="error">&#91;&lt;ffffffffa05b00fb&gt;&#93;</span> obd_setup+0x19b/0x290 [obd<br/> class]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.008810&#93;</span> <span class="error">&#91;&lt;ffffffff81175113&gt;&#93;</span> ? kmem_cache_alloc_trace+0<br/> x1b3/0x1c0<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.015463&#93;</span> <span class="error">&#91;&lt;ffffffffa0594a53&gt;&#93;</span> ? class_new_export+0x883/0<br/> x9e0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.022553&#93;</span> <span class="error">&#91;&lt;ffffffffa05b044f&gt;&#93;</span> class_setup+0x25f/0x940 [o<br/> bdclass]<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.029018&#93;</span> <span class="error">&#91;&lt;ffffffffa05b7631&gt;&#93;</span> class_process_config+0x115<br/> 1/0x2840 <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.036522&#93;</span> <span class="error">&#91;&lt;ffffffffa0575fd7&gt;&#93;</span> ? obd_ioctl_getdata+0x127/<br/> 0x1310 <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.043762&#93;</span> <span class="error">&#91;&lt;ffffffffa0454bb1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x<br/> 50 <span class="error">&#91;libcfs&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.050477&#93;</span> <span class="error">&#91;&lt;ffffffff8117546c&gt;&#93;</span> ? __kmalloc+0x21c/0x230<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.055985&#93;</span> <span class="error">&#91;&lt;ffffffffa058fbf2&gt;&#93;</span> class_handle_ioctl+0xbb2/0<br/> x21e0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.063131&#93;</span> <span class="error">&#91;&lt;ffffffff8148b5a1&gt;&#93;</span> ? netlink_create+0x191/0x2<br/> 00<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.069075&#93;</span> <span class="error">&#91;&lt;ffffffffa05752ab&gt;&#93;</span> obd_class_ioctl+0x4b/0x190<br/> <span class="error">&#91;obdclass&#93;</span><br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.075785&#93;</span> <span class="error">&#91;&lt;ffffffff811a4222&gt;&#93;</span> vfs_ioctl+0x22/0xa0<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.080930&#93;</span> <span class="error">&#91;&lt;ffffffff811acc5f&gt;&#93;</span> ? generic_drop_inode+0x4f/<br/> 0x80<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.087028&#93;</span> <span class="error">&#91;&lt;ffffffff811a43c4&gt;&#93;</span> do_vfs_ioctl+0x84/0x580<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.092516&#93;</span> <span class="error">&#91;&lt;ffffffff81448871&gt;&#93;</span> ? sock_release+0x71/0x90<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.098104&#93;</span> <span class="error">&#91;&lt;ffffffff814489bf&gt;&#93;</span> ? __sock_create+0xff/0x240<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.103849&#93;</span> <span class="error">&#91;&lt;ffffffff811a4941&gt;&#93;</span> sys_ioctl+0x81/0xa0<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.108989&#93;</span> <span class="error">&#91;&lt;ffffffff8100b0d2&gt;&#93;</span> system_call_fastpath+0x16/<br/> 0x1b<br/> Sep 9 13:17:10 ninja11.ccs.ornl.gov kernel: <span class="error">&#91;12212.115100&#93;</span></p> <p>Hi, James, I just update patch. Thanks for updating.</p> <p>Much better!!</p> <p>Still failing for sanity test 300k. I'm getting the following back trace:</p> <p>Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.502856] Lustre: DEBUG MARKER: == sanity test 300k: test large striped dir<br/> ectory == 12:35:19 (1441902919)<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.555631] Lustre: *** cfs_fail_loc=1703, val=0***<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.563480] Lustre: 29844:0:(osd_handler.c:1106:osd_trans_start()) lustre-MDT<br/> 0000: too many transaction credits (51379 &gt; 1024)<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.575080] Lustre: 29844:0:(osd_handler.c:1041:osd_trans_dump_creds()) cre<br/> ate: 146/1168/0, destroy: 0/0/0<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.585090] Lustre: 29844:0:(osd_handler.c:1048:osd_trans_dump_creds()) att<br/> r_set: 130/130/0, xattr_set: 259/16513/0<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.597106] Lustre: 29844:0:(osd_handler.c:1058:osd_trans_dump_creds()) wri<br/> te: 70/2146/0, punch: 0/0/0, quota 2/2/0<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.607934] Lustre: 29844:0:(osd_handler.c:1065:osd_trans_dump_creds()) ins<br/> ert: 932/30778/0, delete: 0/0/0<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.617951] Lustre: 29844:0:(osd_handler.c:1072:osd_trans_dump_creds()) ref<br/> _add: 642/642/0, ref_del: 0/0/0<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.627962] Pid: 29844, comm: mdt01_005<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.631922] <br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.631923] Call Trace:<br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.636098] <span class="error">&#91;&lt;ffffffffa040c875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.643201] <span class="error">&#91;&lt;ffffffffa0e57b3f&gt;&#93;</span> osd_trans_start+0x3ef/0x410 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.650441] <span class="error">&#91;&lt;ffffffffa0837977&gt;&#93;</span> top_trans_start+0x357/0x960 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.657134] <span class="error">&#91;&lt;ffffffffa1082c11&gt;&#93;</span> lod_trans_start+0x61/0x70 <span class="error">&#91;lod&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.663447] <span class="error">&#91;&lt;ffffffffa0f6c5c4&gt;&#93;</span> mdd_trans_start+0x14/0x20 <span class="error">&#91;mdd&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.669699] <span class="error">&#91;&lt;ffffffffa0f5b45d&gt;&#93;</span> mdd_create+0x9cd/0x1280 <span class="error">&#91;mdd&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.675797] <span class="error">&#91;&lt;ffffffffa0fcc77c&gt;&#93;</span> ? mdt_version_save+0x8c/0x1a0 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.682471] <span class="error">&#91;&lt;ffffffffa0fcef3c&gt;&#93;</span> mdt_reint_create+0xbbc/0xcc0 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.688988] <span class="error">&#91;&lt;ffffffffa05ac240&gt;&#93;</span> ? lu_ucred+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.695246] <span class="error">&#91;&lt;ffffffffa0fae675&gt;&#93;</span> ? mdt_ucred+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:35:19 spoon17.ccs.ornl.gov kernel: [ 283.701251] <span class="error">&#91;&lt;ffffffffa0fc785c&gt;&#93;</span> ? mdt_root_squash+0x2c/0x3f0 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.052812] <span class="error">&#91;&lt;ffffffffa07db532&gt;&#93;</span> ? __req_capsule_get+0x162/0x6e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.059811] <span class="error">&#91;&lt;ffffffffa0f3699d&gt;&#93;</span> mdt_reint_rec+0x5d/0x200 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.065938] <span class="error">&#91;&lt;ffffffffa0f227ab&gt;&#93;</span> mdt_reint_internal+0x62b/0xb80 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.072622] <span class="error">&#91;&lt;ffffffffa0f2319b&gt;&#93;</span> mdt_reint+0x6b/0x120 <span class="error">&#91;mdt&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.078449] <span class="error">&#91;&lt;ffffffffa081d72c&gt;&#93;</span> tgt_request_handle+0xa4c/0x1290 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.085555] <span class="error">&#91;&lt;ffffffffa07c53b1&gt;&#93;</span> ptlrpc_main+0xe41/0x1910 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.091995] <span class="error">&#91;&lt;ffffffffa07c4570&gt;&#93;</span> ? ptlrpc_main+0x0/0x1910 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.098371] <span class="error">&#91;&lt;ffffffff8109e88e&gt;&#93;</span> kthread+0x9e/0xc0<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.103354] <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.108422] <span class="error">&#91;&lt;ffffffff8109e7f0&gt;&#93;</span> ? kthread+0x0/0xc0<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.113536] <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.118808] <br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.120489] LustreError: 30312:0:(osd_internal.h:1030:osd_trans_exec_op()) lustre-MDT0000-osd: op = 3, rb = 4<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.423620] Lustre: 30312:0:(osd_handler.c:1041:osd_trans_dump_creds()) create: 146/1168/11, destroy: 0/0/0<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.433696] Lustre: 30312:0:(osd_handler.c:1041:osd_trans_dump_creds()) Skipped 419 previous similar messages<br/> Sep 10 12:21:13 spoon17.ccs.ornl.gov kernel: [ 515.443770] Lustre: 30312:0:(osd_handler.c:1048:osd_trans_dump_creds()) attr_set: 130/130/0, xattr_set: 259/16513/0</p> <p>Hi, James, this is a known issue, because credits required for creating a big striped directory is larger than maximum credits here. And I believe you can see this on current master as well.<br/> But it is just a warning for now, and this should not cause 300k fails. Did you see other error message there? thanks.</p> <p>I posted the full log under ticket <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a>. You can see it here : <a href="https://jira.hpdd.intel.com/secure/attachment/18890/kern-09-10-2015.log" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18890/kern-09-10-2015.log</a></p> <p>Hmm, I only saw 300g and 300h failed in your log.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 10 12:49:43 spoon17.ccs.ornl.gov kernel: [ 239.577103] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory == 12:49:43 (1441903783) Sep 10 12:49:44 spoon17.ccs.ornl.gov kernel: [ 240.291185] Lustre: DEBUG MARKER: sanity test_300g: @@@@@@ FAIL: stripe count 1 != 0 for /lustre/lustre/d300g.sanity/normal_dir/test1 Sep 10 12:49:45 spoon17.ccs.ornl.gov kernel: [ 241.048817] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory == 12:49:45 (1441903785) Sep 10 12:49:45 spoon17.ccs.ornl.gov kernel: [ 241.837071] Lustre: DEBUG MARKER: sanity test_300h: @@@@@@ FAIL: stripe count 1 != 0 for </pre> </div></div> <p>is <a href="http://review.whamcloud.com/#/c/16130/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16130/</a> in your series? I think there is a fix in that patch( see test_300_check_default_striped_dir) to deal with this failure.</p> <p>Yes it is my series. Ah I haven't updates my local test scripts. Still blows up the node with 300k.</p> <p>Hmm, I guess somewhere in ldiskfs still complain about this super giant credit reservation. Do you have stack trace when the panic happens?</p> <p>All have this what I posted to <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a> for the trace. The node just reboots with no kdumps or anything <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16333/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16333/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: create remote llog synchronously<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 36f59f94c06c74887a32f2a7757e7c962c6cf8dd</p> <p>Landed for 2.8</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17579" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17579</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: create remote llog synchronously<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b66f9d956c27b7f3b1ee7289ef2e230de93b71e4</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17616</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> llog: combine cancel rec and destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0e22e861ff702c7515f8716b29770874d1e230e1</p> Duplicate Related LU-6831 LU-7063 LU-9197 Development Rank 1|hzxi4v: Rank (Obsolete) 9223372036854775807 Severity