https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13195 Lustre <p>This issue was created by maloo for liuying &lt;emoly.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/ca28353a-46ca-11ea-91a9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ca28353a-46ca-11ea-91a9-52540065bddc</a></p> <p>test_118 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-single test 118: invalidate osp update will not cause update log corruption ================ 17:21:23 (1580750483) CMD: trevis-19vm4 lctl set_param fail_loc=0x1705 </pre> </div></div> <p>and the following stack trace on the console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10146.524712] LustreError: 19994:0:(dt_object.h:2191:dt_declare_record_write()) ASSERTION( dt-&gt;do_body_ops ) failed: [10146.525745] LustreError: 19994:0:(dt_object.h:2191:dt_declare_record_write()) LBUG [10146.526542] Pid: 19994, comm: mdt_out00_000 3.10.0-957.27.2.el7_lustre.x86_64 #1 SMP Sat Jan 18 23:01:59 UTC 2020 [10146.527632] Call Trace: [10146.527905] [&lt;ffffffffc0c348ac&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [10146.528605] [&lt;ffffffffc0c3495c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [10146.529291] [&lt;ffffffffc10f65eb&gt;] out_write_add_exec+0x13b/0x1b0 [ptlrpc] [10146.530275] [&lt;ffffffffc10eed43&gt;] out_write+0x333/0x370 [ptlrpc] [10146.530971] [&lt;ffffffffc10f1086&gt;] out_handle+0x1566/0x1bb0 [ptlrpc] [10146.531652] [&lt;ffffffffc10e7eca&gt;] tgt_request_handle+0x95a/0x1610 [ptlrpc] [10146.532417] [&lt;ffffffffc108b816&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [10146.533234] [&lt;ffffffffc108f8a4&gt;] ptlrpc_main+0xbb4/0x1550 [ptlrpc] </pre> </div></div> <p>This issue happened several times in Maloo testing but no more logs were collected.<br/> <a href="https://testing.whamcloud.com/sub_tests/121c5288-447b-11ea-bffa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/121c5288-447b-11ea-bffa-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/sub_tests/7cefb86c-4362-11ea-86b2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/7cefb86c-4362-11ea-86b2-52540065bddc</a></p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-single test_118 - trevis-19vm4 crashed during replay-single test_118</p> LU-13195

replay-single test_118: dt_declare_record_write() ASSERTION( dt->do_body_ops ) failed

Bug Major Resolved Fixed Alex Zhuravlev Maloo Tue, 4 Feb 2020 01:12:36 +0000 Fri, 6 Jan 2023 00:36:12 +0000 Tue, 9 Nov 2021 17:19:38 +0000 Lustre 2.14.0 Lustre 2.15.0 Lustre 2.12.10 0 12 <p>This looks like it is the same as <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a>.</p> <p>Another instance:<br/> <a href="https://testing.whamcloud.com/test_sets/2ee75f70-b67b-447d-ba2a-db2f67c0bd0f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2ee75f70-b67b-447d-ba2a-db2f67c0bd0f</a></p> <p>looking at the logs from sanity with the same symptom it looks like sometime llog cookie is shared by transactions:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 1586485804.112492:0:13975:0:(update_trans.c:75:top_multiple_thandle_dump()) lustre-MDT0001-osd tmt 000000004e86531f refcount 1 committed 1 result -5 batchid 17179870236 1586485804.112493:0:13975:0:(update_trans.c:85:top_multiple_thandle_dump()) st 000000001da13291 obd lustre-MDT0000-osp-MDT0001 committed 1 started 1 stopped 1 result -5 sub_th 000000003da75353 1586485804.112494:0:13975:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x2341:0x2].873 1586485804.112495:0:13975:0:(update_trans.c:85:top_multiple_thandle_dump()) st 000000000cc313ae obd lustre-MDT0001-osd committed 1 started 1 stopped 1 result 0 sub_th 000000000764ad4c 1586485804.112497:0:13975:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x40002340:0x2].921 1586485804.112497:0:13975:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x40002340:0x2].921 ... 1586485804.112723:0:13975:0:(update_trans.c:75:top_multiple_thandle_dump()) lustre-MDT0001-osd tmt 0000000046070051 refcount 1 committed 1 result 0 batchid 17179870237 1586485804.112725:0:13975:0:(update_trans.c:85:top_multiple_thandle_dump()) st 00000000a7f7ee4f obd lustre-MDT0000-osp-MDT0001 committed 1 started 1 stopped 1 result 0 sub_th 00000000222fbaf6 1586485804.112726:0:13975:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x2341:0x2].873 1586485804.112727:0:13975:0:(update_trans.c:85:top_multiple_thandle_dump()) st 00000000adeff453 obd lustre-MDT0001-osd committed 1 started 1 stopped 1 result 0 sub_th 0000000090f0b368 1586485804.112728:0:13975:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x40002340:0x2].922 </pre> </div></div> <p>notice cookie <b><span class="error">&#91;0x2:0x2341:0x2&#93;</span>.873</b></p> <p>the consequence is that cancellation of the first record (873) causes destroy of corresponding llog file and it's not supposed to be referenced any more, but then recovery process finds cookie duplicate and tries to repeat cancellation/destroy.<br/> while it's possible to handle this (checking whether the llog does exist) I think it's important to understand the root cause for duplicates.</p> <p>in another case it was the same transaction being cancelled twice in two different threads:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> .121703:0:5542:0:(update_trans.c:75:top_multiple_thandle_dump()) lustre-MDT0001-osd tmt 00000000cb6dafd8 refcount 2 committed 1 result 0 batchid 17179870235 .121705:0:5542:0:(update_trans.c:85:top_multiple_thandle_dump()) st 0000000084a8ca02 obd lustre-MDT0000-osp-MDT0001 committed 1 started 0 stopped 0 result 0 sub_th (<span class="code-keyword">null</span>) .121706:0:5542:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x2341:0x2].871 .121707:0:5542:0:(update_trans.c:85:top_multiple_thandle_dump()) st 00000000a1d7d236 obd lustre-MDT0001-osd committed 1 started 0 stopped 0 result 0 sub_th (<span class="code-keyword">null</span>) .121708:0:5542:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x40002340:0x2].902 .121919:0:5539:0:(update_trans.c:75:top_multiple_thandle_dump()) lustre-MDT0001-osd tmt 00000000cb6dafd8 refcount 1 committed 1 result 0 batchid 17179870235 .121920:0:5539:0:(update_trans.c:85:top_multiple_thandle_dump()) st 0000000084a8ca02 obd lustre-MDT0000-osp-MDT0001 committed 1 started 0 stopped 0 result 0 sub_th (<span class="code-keyword">null</span>) .121922:0:5539:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x2341:0x2].871 .121923:0:5539:0:(update_trans.c:85:top_multiple_thandle_dump()) st 00000000a1d7d236 obd lustre-MDT0001-osd committed 1 started 0 stopped 0 result 0 sub_th (<span class="code-keyword">null</span>) .121924:0:5539:0:(update_trans.c:92:top_multiple_thandle_dump()) cookie [0x2:0x40002340:0x2].902 </pre> </div></div> <p>dist_txn-1 D 0 5539 2 0x80000000<br/> Call Trace:<br/> ? __schedule+0x2ad/0xb00<br/> schedule+0x34/0x80<br/> lbug_with_loc+0x79/0x80 <span class="error">&#91;libcfs&#93;</span><br/> ? osp_md_write+0x4c3/0x540 <span class="error">&#91;osp&#93;</span><br/> ? dt_record_write+0x2a/0x110 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_osd_write_rec+0x717/0x1dd0 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_write_rec+0x36a/0x4e0 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_cancel_arr_rec+0x746/0xb20 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_cat_cancel_arr_rec+0x143/0x400 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_cat_cancel_records+0x4f/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> ? distribute_txn_commit_thread+0x580/0x11d0 <span class="error">&#91;ptlrpc&#93;</span><br/> ? __kthread_parkme+0x39/0x80</p> <p>00000100:00100000:1.0:1586499582.124714:0:5542:0:(client.c:1697:ptlrpc_send_new_req()) Sending RPC req@0000000053ab4ec6 pname:cluuid:pid:xid:nid:opc:job tgt_recover_1:lustre-MDT0001-mdtlov_UUID:5542:1663564960200128:192.168.122.97@tcp:1000:tgt_recover_1.0</p> <p>Is the mechanism for generating this number properly locked? Are you sure that it is being printed consistently?</p> <p>sorry, that was wrong understanding. I've got couple more logs and it's actually not duplicated id, but rather <b>correct</b> (in some sense) logic:<br/> llog is destroyed because the last record is just cancelled. normally this doesn't happen because this log is <em>current</em>, but after recovery it's not <em>current</em> anymore as we usually create a new llog (iirc).<br/> there is no current llog during recovery as we don't generate new llog records (though we can write records from update log), so current llog is NULL at recovery, this can be used to skip llog destroy, but then tricky thing is when to destroy those. will try to catch this..</p> <p>so far the change (postponing llog destroy) has been running for many hours and I saw this condition happeing sometimes.<br/> trying to construct a scripted test for the case..</p> <p>one identified issue is a race between llog append/cancel and invalidataion. basically any change to llog involves two changes at least: llh_count and bitmap (plus new record in case of append). to save on RPC size we modify llh_count and bitmap using two separate dt_write(). sometimes invalidation comes in between these two writes and one of them don't get into the transaction, but the transaction itself proceeds with just the first write.<br/> I think in this case llog should invalidate internal state (bitmap, counters) and re-initialize it (in this call or in the next one), the transaction then must be marked properly and aborted as a whole. working on a patch..<br/> this is not the only root cause for the issue, AFAICS. there is another one (or few) leading to similar symptoms.</p> <p>can't reproduce reliable, but this is my reconstruction: for a reason OSP invalidate failed request and all subsequent requests in the sending queue (<em>osp_send_update_thread()</em> -&gt; <em>osp_invalidate_request()</em>), which basically should invalidate all cached internal states like llog's bitmaps.<br/> all <em>new</em> transactions will notice that and refresh llog's data, but all <em>already-declared</em> transactions will still proceed. this is quite short window given everything is running in cache, all locks have been acquired already, etc, but still possible.<br/> then depending on luck the llog can be corrupted different ways (in majority of cases - llh_count is less than actual number of records, leading to early llog destroy)<br/> I'm trying to construct a test, but not sure how to bind two events: request invalidation and resuming a suspended-after-declaration transaction.</p> <p>few different issues have been identified so far:</p> <ul> <li>llog_cat_refresh() causes duplicated llog ids - if some write failed, then to reinitialize llog state the master MDS re-reads remote llog and gets old lgh_last_idx, which has been already returned to another transaction</li> <li>object invalidation doesn't work in one case at least - osp_md_write() doesn't put on object on invalidation list for subsequent transaction, then a race possible which in turn results in lost -ESTALE and then lgh_count won't match actual number of records</li> <li>partial writes - I've seen this in the logs, but not sure about exact sequence leading to this</li> </ul> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38385" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38385</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: track destroyed OSP object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 52bc4e0824a185e05551a3d95bb721c61baa6b40</p> <p>Alex, I have a draft fix for invalidation. I'm pushing it to gerrit today.</p> <blockquote><p>Alex, I have a draft fix for invalidation. I'm pushing it to gerrit today.</p></blockquote> <p>for xattr_set or for more ops? write path misses invalidation as well, I did that by adding an interpreter for osp_md_write() which doesn't depend on invalidation list.</p> <p>Fix for any operation.</p> <p>Alexander Boyko (alexander.boyko@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38386" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38386</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate objects for error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1107a36f44fd2d8c195c9bee5ad3db565efe766e</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38387" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38387</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 40eb50739b4f836cc6161e0efc866060cb267ac8</p> <p>Alex, I'm using the next debug patch to catch llog corruption</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/target/out_lib.c b/lustre/target/out_lib.c index e8bdfd1..9eac332 100644 --- a/lustre/target/out_lib.c +++ b/lustre/target/out_lib.c @@ -697,6 +697,20 @@ static int out_tx_write_exec(const struct lu_env *env, struct thandle *th, if (OBD_FAIL_CHECK(OBD_FAIL_OUT_ENOSPC)) { rc = -ENOSPC; } else { + struct lu_attr la; + unsigned int *buf = (unsigned int*) arg-&gt;u.write.buf.lb_buf; + dt_attr_get(env, dt_obj, &amp;la); + + if (arg-&gt;u.write.buf.lb_len &gt; LLOG_MIN_REC_SIZE &amp;&amp; + __le32_to_cpu(buf[2]) == UPDATE_REC &amp;&amp; + arg-&gt;u.write.pos &gt; la.la_size) { + CDEBUG(D_HA, "write with sparse "DFID" pos %llu buf %p, len %lu, size %llu\n", + PFID(lu_object_fid(&amp;dt_obj-&gt;do_lu)), arg-&gt;u.write.pos, + arg-&gt;u.write.buf.lb_buf, (unsigned long)arg-&gt;u.write.buf.lb_len, + la.la_size); + + LBUG(); + } dt_write_lock(env, dt_obj, MOR_TGT_CHILD); rc = dt_record_write(env, dt_obj, &amp;arg-&gt;u.write.buf, &amp;arg-&gt;u.write.pos, th); </pre> </div></div> <p>thanks for sharing. I used this one (to catch on remote side):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/osd-ldiskfs/osd_handler.c b/lustre/osd-ldiskfs/osd_handler.c index 8d8d8b4d7e..318e5aa4dc 100644 --- a/lustre/osd-ldiskfs/osd_handler.c +++ b/lustre/osd-ldiskfs/osd_handler.c @@ -1788,6 +1788,8 @@ <span class="code-keyword">static</span> struct thandle *osd_trans_create(<span class="code-keyword">const</span> struct lu_env *env, oti-&gt;oti_ins_cache_depth++; + oh-&gt;ot_bh = NULL; + RETURN(th); } @@ -1967,6 +1969,32 @@ <span class="code-keyword">static</span> void osd_trans_stop_cb(struct osd_thandle *oth, <span class="code-object">int</span> result) } } +void osd_llog_hdr_check(<span class="code-keyword">const</span> struct lu_fid *fid, void *buf) +{ + struct llog_log_hdr *llh = buf; + <span class="code-object">int</span> found = 0; + <span class="code-object">int</span> i = 0; + <span class="code-object">int</span> max; + <span class="code-object">int</span> last = -1, first = -1; + + max = 128 * 8; + <span class="code-keyword">while</span> (i &lt; max) { + i = find_next_bit((unsigned <span class="code-object">long</span> *)LLOG_HDR_BITMAP(llh), max, i); + <span class="code-keyword">if</span> (i &gt;= max) + <span class="code-keyword">break</span>; + <span class="code-keyword">if</span> (first &lt;= 0) + first = i; + last = i; + i++; + found++; + } + CDEBUG(D_HA, <span class="code-quote">"llog osd_write "</span>DFID<span class="code-quote">", llh_count=%d, found %d, first %d, last %d\n"</span>, + PFID(fid), (<span class="code-object">int</span>)llh-&gt;llh_count, found, first, last); + LASSERTF(llh-&gt;llh_count == found, + <span class="code-quote">"llog osd_write "</span>DFID<span class="code-quote">", llh_count=%d, found %d, first %d, last %d\n"</span>, + PFID(fid), (<span class="code-object">int</span>)llh-&gt;llh_count, found, first, last); +} + /* * Concurrency: shouldn't matter. */ @@ -1986,6 +2014,12 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> osd_trans_stop(<span class="code-keyword">const</span> struct lu_env *env, struct dt_device *dt, oh = container_of0(th, struct osd_thandle, ot_super); + <span class="code-keyword">if</span> (oh-&gt;ot_bh) { + osd_llog_hdr_check(&amp;oh-&gt;ot_fid, oh-&gt;ot_bh-&gt;b_data); + brelse(oh-&gt;ot_bh); + oh-&gt;ot_bh = NULL; + } + remove_agents = oh-&gt;ot_remove_agents; qtrans = oh-&gt;ot_quota_trans; diff --git a/lustre/osd-ldiskfs/osd_internal.h b/lustre/osd-ldiskfs/osd_internal.h index 6c62ca450b..8aedc5e9ff 100644 --- a/lustre/osd-ldiskfs/osd_internal.h +++ b/lustre/osd-ldiskfs/osd_internal.h @@ -426,6 +426,8 @@ struct osd_thandle { ktime_t oth_started; #endif struct list_head ot_trunc_locks; + struct buffer_head *ot_bh; + struct lu_fid ot_fid; }; /** diff --git a/lustre/osd-ldiskfs/osd_io.c b/lustre/osd-ldiskfs/osd_io.c index ce2af335ad..6563e8bd2b 100644 --- a/lustre/osd-ldiskfs/osd_io.c +++ b/lustre/osd-ldiskfs/osd_io.c @@ -1680,9 +1680,11 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> osd_ldiskfs_writelink(struct inode *inode, <span class="code-object">char</span> *buffer, <span class="code-object">int</span> buflen) <span class="code-keyword">return</span> 0; } +void osd_llog_hdr_check(<span class="code-keyword">const</span> struct lu_fid *fid, void *buf); + <span class="code-keyword">static</span> <span class="code-object">int</span> osd_ldiskfs_write_record(struct dt_object *dt, void *buf, <span class="code-object">int</span> bufsize, <span class="code-object">int</span> write_NUL, loff_t *offs, - handle_t *handle) + handle_t *handle, struct osd_thandle *oth) { struct inode *inode = osd_dt_obj(dt)-&gt;oo_inode; struct buffer_head *bh = NULL; @@ -1786,6 +1788,25 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> osd_ldiskfs_write_record(struct dt_object *dt, void *buf, sync = <span class="code-keyword">false</span>; } } + <span class="code-keyword">if</span> (block == 0 &amp;&amp; bh &amp;&amp; !create) { + <span class="code-keyword">const</span> struct lu_fid *fid = lu_object_fid(&amp;dt-&gt;do_lu); + <span class="code-keyword">if</span> (fid_seq(fid) == 0x200002341 || fid_seq(fid) == 0x200002342) { + <span class="code-keyword">if</span> (size == 4) { + unsigned <span class="code-object">char</span> *bb = buf; + CDEBUG(D_HA, <span class="code-quote">"write %d @ %d: %02x %02x %02x %02x\n"</span>, + (<span class="code-object">int</span>)size, boffs, bb[0], bb[1], bb[2], bb[3]); + } + + <span class="code-keyword">if</span> (oth-&gt;ot_bh == NULL) { + get_bh(bh); + oth-&gt;ot_fid = *fid; + oth-&gt;ot_bh = bh; + osd_llog_hdr_check(fid, bh-&gt;b_data); + } <span class="code-keyword">else</span> { + LASSERT(oth-&gt;ot_bh == bh); + } + } + } memcpy(bh-&gt;b_data + boffs, buf, size); err = ldiskfs_handle_dirty_metadata(handle, NULL, bh); <span class="code-keyword">if</span> (err) @@ -1856,7 +1878,7 @@ <span class="code-keyword">static</span> ssize_t osd_write(<span class="code-keyword">const</span> struct lu_env *env, struct dt_object *dt, result = osd_ldiskfs_writelink(inode, buf-&gt;lb_buf, buf-&gt;lb_len); <span class="code-keyword">else</span> result = osd_ldiskfs_write_record(dt, buf-&gt;lb_buf, buf-&gt;lb_len, - is_link, pos, oh-&gt;ot_handle); + is_link, pos, oh-&gt;ot_handle, oh); <span class="code-keyword">if</span> (result == 0) result = buf-&gt;lb_len; </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38387/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38387/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e1071b517578ed3752efb1412017c8f93cd333b</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38977" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38977</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> obdclass: show FID for corrupted llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b381c6b220c888b7eacd0abf947add60bbf11231</p> <p>+1 on master with this patch applied <a href="https://review.whamcloud.com/38387" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38387</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error</tt>"<br/> <a href="https://testing.whamcloud.com/test_sets/16bbb988-9976-40c3-86d3-e673eec6e820" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/16bbb988-9976-40c3-86d3-e673eec6e820</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/3d28761f-80fe-402e-9d21-53f977d88d40" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/3d28761f-80fe-402e-9d21-53f977d88d40</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38977/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38977/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> obdclass: show FID for corrupted llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5b74e0466edbfbbf9f336171de1adc5e583e9475</p> <p>+1 <a href="https://testing.whamcloud.com/test_sets/eca8222e-939a-437a-be41-0df8325c411d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/eca8222e-939a-437a-be41-0df8325c411d</a></p> <p>Alex - </p> <p>Is this the same issue <a href="https://testing.whamcloud.com/test_sets/dfff72b1-060d-4d53-b15b-b8a989d5c8d5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/dfff72b1-060d-4d53-b15b-b8a989d5c8d5</a> ?</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/757f2536-f148-46a7-8992-2ddef45293bf" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/757f2536-f148-46a7-8992-2ddef45293bf</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/27e54b38-cf46-466f-8fec-4cf25e9bb57a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/27e54b38-cf46-466f-8fec-4cf25e9bb57a</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1705 Lustre: *** cfs_fail_loc=1705, val=0*** Lustre: Skipped 1 previous similar message LustreError: 760439:0:(dt_object.h:2274:dt_declare_record_write()) ASSERTION( dt-&gt;do_body_ops ) failed: [0x20003ccc2:0x1:0x0] doesn't exit </pre> </div></div> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/cb680395-862a-482a-98bb-c7b3da79cf68" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/cb680395-862a-482a-98bb-c7b3da79cf68</a></p> <p>Alex, is patch <a href="https://review.whamcloud.com/38385" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38385</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: track destroyed OSP object</tt>" expected to fix this problem, or is that more of a diagnostic patch?</p> <p>This subtest crashed about 10 of 200 runs in the past week.</p> <p>It seems also to happen on runtest test_1:<br/> <a href="https://testing.whamcloud.com/test_sets/842d914c-2cce-427d-a5a7-4aadb1a36d21" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/842d914c-2cce-427d-a5a7-4aadb1a36d21</a></p> <p>This seems to be failing fairly often on master recently - "15.15% of most recent 66 runs, 34 skipped (all branches)"</p> <p>yes, looking at this one in background..</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44853" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44853</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> obdclass: catch llog corruption<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 513265c887b42247b93c23efe8e8d17c84013841</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/38385/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38385/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: track destroyed OSP object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f5a8f1bcf5563f96cf6ba0e5de5a99a1ea524cc6</p> <p>Landed for 2.15</p> <p><del>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/44993" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44993</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: object creation needs a version</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: a79be4c9521b4c35f5efcf813350e7097cfd4210</del></p> <p>this still happen to master. my understanding is that eviction in 118 can cause different operations to abort. the last few times I checked it was llog object creation invalidated. I'm working on a reliable reproducer.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45042" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45042</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> obdclass: check another theory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3affa067b1ff7a5a4577eadbd269ae300ccb22cd</p> <p>with the latest <a href="https://review.whamcloud.com/45042" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45042</a> I can't reproduce LBUG() anymore. basically it's a race - failed (by intention) create doesn't invalidate request-in-progress properly, then that survived request (containing a write to object just failed to create) flies to remote MDT and we get the LBUG().<br/> with LBUG resolved I observed another issue - few OSP structures from that inproperly invalidated request can leak. I <em>think</em> this is slightly different issue and plan to fix that with another patch.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45042/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45042/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: osp_send_update_req() should check generation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: dff1e0d21c8c6bb20d63669252190795198bc49f</p> <p>So... complete it seems</p> <p>+1 in runtest (with all the fix patches):<br/> <a href="https://testing.whamcloud.com/test_sets/7110dd40-b540-469d-a773-874769fe527b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7110dd40-b540-469d-a773-874769fe527b</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 4327.451873] Lustre: DEBUG MARKER: copying 607 files from /etc /bin to /mnt/lustre/d1.runtests/etc /bin at Fri Oct 22 22:51:39 UTC 2021 [ 4334.578564] LustreError: 12949:0:(dt_object.h:2310:dt_declare_record_write()) ASSERTION( dt-&gt;do_body_ops ) failed: [0x200011571:0x1:0x0] doesn't exit [ 4334.581584] LustreError: 12949:0:(dt_object.h:2310:dt_declare_record_write()) LBUG [ 4334.583011] Pid: 12949, comm: mdt_out00_001 4.18.0-240.22.1.el8_lustre.x86_64 #1 SMP Mon Oct 4 16:46:22 UTC 2021 [ 4334.585079] Call Trace TBD: [ 4334.585913] [&lt;0&gt;] libcfs_call_trace+0x6f/0x90 [libcfs] [ 4334.586882] [&lt;0&gt;] lbug_with_loc+0x43/0x80 [libcfs] [ 4334.588281] [&lt;0&gt;] out_write_add_exec+0x17d/0x1e0 [ptlrpc] [ 4334.589374] [&lt;0&gt;] out_write+0x166/0x380 [ptlrpc] [ 4334.590282] [&lt;0&gt;] out_handle+0x16af/0x20e0 [ptlrpc] [ 4334.591293] [&lt;0&gt;] tgt_request_handle+0xc93/0x1a00 [ptlrpc] [ 4334.592391] [&lt;0&gt;] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc] [ 4334.593680] [&lt;0&gt;] ptlrpc_main+0xc06/0x1550 [ptlrpc] [ 4334.594667] [&lt;0&gt;] kthread+0x112/0x130 [ 4334.595389] [&lt;0&gt;] ret_from_fork+0x35/0x40 </pre> </div></div> <p>damn... looking</p> <p>Sorry, new failure was <a href="https://jira.whamcloud.com/browse/LU-15139" title="sanity test_160h: dt_record_write() ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed" class="issue-link" data-issue-key="LU-15139"><del>LU-15139</del></a>, which is similar, but not identical. </p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46863" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46863</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a6555d5b7f9a76250e8460adcb3d8a089356f490</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47010" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47010</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: osp_send_update_req() should check generation<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: e555f223a6ad340e7f326478cd09ba36b4b8bbb2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46863/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46863/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: be237a523e1208888f8f7d10e2a88709ea823a74</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47010/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47010/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: osp_send_update_req() should check generation<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: b18246d8b78a308c32a5f78eee581f16dae5dc44</p> Duplicate LU-10143 LU-13295 Related LU-13411 LU-15769 LU-13469 LU-15139 Development Rank 1|i00t47: Rank (Obsolete) 9223372036854775807 Severity