https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14918 Lustre <p>Removing widely overstriped files from an ldiskfs MDT causes excessively many transaction credits to be reserved. This can be seen in the MDS console logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ======== Lustre: 25401:0:(osd_handler.c:1934:osd_trans_start()) lustre-MDT0000: credits 54595 &gt; trans_max 2592 Lustre: 25401:0:(osd_handler.c:1863:osd_trans_dump_creds()) create: 800/6400/0, destroy: 1/4/0 Lustre: 25401:0:(osd_handler.c:1870:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 804/148/0 Lustre: 25401:0:(osd_handler.c:1880:osd_trans_dump_creds()) write: 4001/34410/0, punch: 0/0/0, quota 6/6/0 Lustre: 25401:0:(osd_handler.c:1887:osd_trans_dump_creds()) insert: 801/13616/0, delete: 2/5/0 Lustre: 25401:0:(osd_handler.c:1894:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 Pid: 25401, comm: mdt00_004 3.10.0-1160.36.2.el7_lustre.x86_64 #1 SMP Tue Aug 3 23:03:31 UTC 2021 Call Trace: libcfs_call_trace+0x90/0xf0 [libcfs] libcfs_debug_dumpstack+0x26/0x30 [libcfs] osd_trans_start+0x4bb/0x4e0 [osd_ldiskfs] top_trans_start+0x702/0x940 [ptlrpc] lod_trans_start+0x34/0x40 [lod] mdd_trans_start+0x1a/0x20 [mdd] mdd_unlink+0x4ee/0xae0 [mdd] mdo_unlink+0x1b/0x1d [mdt] mdt_reint_unlink+0xb64/0x1890 [mdt] mdt_reint_rec+0x83/0x210 [mdt] mdt_reint_internal+0x720/0xaf0 [mdt] mdt_reint+0x67/0x140 [mdt] tgt_request_handle+0x7ea/0x1750 [ptlrpc] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] Lustre: 25401:0:(osd_internal.h:1325:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 2589 &lt; left 34410, rollback = 7 </pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 27Cd: test maximum stripe count ======== Lustre: 12686:0:(osd_handler.c:1934:osd_trans_start()) lustre-MDT0003: credits 136195 &gt; trans_max 2592 Lustre: 12686:0:(osd_handler.c:1863:osd_trans_dump_creds()) create: 2000/16000/0, destroy: 1/4/0 Lustre: 12686:0:(osd_handler.c:1870:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 2004/148/0 Lustre: 12686:0:(osd_handler.c:1880:osd_trans_dump_creds()) write: 10001/86010/0, punch: 0/0/0, quota 6/6/0 Lustre: 12686:0:(osd_handler.c:1887:osd_trans_dump_creds()) insert: 2001/34016/0, delete: 2/5/0 Lustre: 12686:0:(osd_handler.c:1894:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 Pid: 12686, comm: mdt00_000 3.10.0-1160.36.2.el7_lustre.x86_64 #1 SMP Tue Aug 3 23:03:31 UTC 2021 Call Trace: libcfs_call_trace+0x90/0xf0 [libcfs] libcfs_debug_dumpstack+0x26/0x30 [libcfs] osd_trans_start+0x4bb/0x4e0 [osd_ldiskfs] top_trans_start+0x702/0x940 [ptlrpc] lod_trans_start+0x34/0x40 [lod] mdd_trans_start+0x1a/0x20 [mdd] mdd_unlink+0x4ee/0xae0 [mdd] mdo_unlink+0x1b/0x1d [mdt] mdt_reint_unlink+0xb64/0x1890 [mdt] mdt_reint_rec+0x83/0x210 [mdt] mdt_reint_internal+0x720/0xaf0 [mdt] mdt_reint+0x67/0x140 [mdt] tgt_request_handle+0x7ea/0x1750 [ptlrpc] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] </pre> </div></div> <p>and similarly in sanity test_130e, sanity-pfl test_0b, test_1c, always during unlink.</p> <p>The two examples shown are trying to reserve a whopping 213MiB and 532MiB of journal space, respectively. Since the maximum xattr size for an overstriped file is 64KiB, this is pretty excessive. </p> LU-14918

too many ldiskfs transaction credits when unlinking overstriped files

Bug Major Resolved Fixed Alex Zhuravlev Andreas Dilger Fri, 6 Aug 2021 22:58:54 +0000 Fri, 29 Dec 2023 11:10:38 +0000 Tue, 14 Feb 2023 13:25:41 +0000 Lustre 2.14.0 Lustre 2.15.0 Lustre 2.16.0 0 11 <p>There was previously patch <a href="http://review.whamcloud.com/8881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8881</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-4648" title="inode deletion transaction does not count large EA inode deletion credits" class="issue-link" data-issue-key="LU-4648"><del>LU-4648</del></a> ldiskfs: delete external EA in another transaction</tt>" that pushed the unlink of the xattr inode to a separate transaction, so either that code is not working properly anymore, or <tt>mdd_declare_unlink()</tt> (and sub-functions) is counting the entire xattr size in the credits for the unlink, even though it is done in a separate transaction.</p> <p>This may also be fixed by the <a href="https://jira.whamcloud.com/browse/LU-14965" title="ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x290 [ldiskfs]" class="issue-link" data-issue-key="LU-14965"><del>LU-14965</del></a> patch. </p> <p>I'm confused where these "create/write" requests come from, since mdd_declare_unlink() is quite simple </p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>mdd_declare_unlink()</b></div><div class="codeContent panelContent"> <pre class="code-java"> 1628 <span class="code-keyword">if</span> (c != NULL) { 1629 rc = mdo_declare_ref_del(env, c, handle); 1630 <span class="code-keyword">if</span> (rc) 1631 <span class="code-keyword">return</span> rc; 1632 1633 rc = mdo_declare_ref_del(env, c, handle); 1634 <span class="code-keyword">if</span> (rc) 1635 <span class="code-keyword">return</span> rc; 1636 1637 la-&gt;la_valid = LA_CTIME; 1638 rc = mdo_declare_attr_set(env, c, la, handle); 1639 <span class="code-keyword">if</span> (rc) 1640 <span class="code-keyword">return</span> rc; 1641 1642 rc = mdd_declare_finish_unlink(env, c, handle); 1643 <span class="code-keyword">if</span> (rc) 1644 <span class="code-keyword">return</span> rc; 1645 1646 <span class="code-comment">/* FIXME: need changelog <span class="code-keyword">for</span> remove entry */</span> 1647 rc = mdd_declare_changelog_store(env, mdd, CL_UNLINK, name, 1648 NULL, handle); 1649 } </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>mdd_declare_finish_unlink()</b></div><div class="codeContent panelContent"> <pre class="code-java"> 1519 rc = mdd_mark_orphan_object(env, obj, handle, <span class="code-keyword">true</span>); 1520 <span class="code-keyword">if</span> (rc != 0) 1521 <span class="code-keyword">return</span> rc; 1522 1523 rc = mdo_declare_destroy(env, obj, handle); 1524 <span class="code-keyword">if</span> (rc != 0) 1525 <span class="code-keyword">return</span> rc; 1526 1527 rc = mdd_orphan_declare_insert(env, obj, mdd_object_type(obj), handle); 1528 <span class="code-keyword">if</span> (rc != 0) 1529 <span class="code-keyword">return</span> rc; 1530 1531 <span class="code-keyword">return</span> mdd_declare_links_del(env, obj, handle); </pre> </div></div> <p>mdd_declare_changelog_store() should not generate so much write requests I think.</p> <p>The debug message shows that it is logging thousands of separate operations:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>create: 800/6400/0, destroy: 1/4/0 write: 4001/34410/0, punch: 0/0/0, quota 6/6/0 insert: 801/13616/0, delete: 2/5/0 </pre> </div></div> <p>This means 800 creates for 6400 credits, 4000 writes (5 per create/insert) for 34410 credits, and 801 inserts for 13616 credits. The sanity <tt>test_130g</tt> itself is unlinking an overstriped file with <tt>$((OSTCOUNT * 100)) = 800</tt> stripes, and <tt>test_27Cd</tt> is unlinking a 2000-stripe file, but those objects are on the OSTs, not on the MDT. It looks like the LOD layer is declaring local credits for each remote object being unlinked? I could understand that LOD needs a few credits to update llog records for each unlinks (e.g. enough to allocate a few blocks for each OST), but it doesn't need hundreds of MB of credits.</p> <p>It wouldn't be too hard to add extra debugging into <tt>osd_declare_create()</tt>, <tt>osd_declare_write()</tt>, and <tt>osd_declare_insert()</tt> that if there are more than, say, 50 such operations in a single handle to print a <tt>CWARN()</tt> and <tt>libcfs_debug_dumpstack()</tt> (and set a flag so that it doesn't happen another 750 or 3950 times) to see where this is happening in the code. However, looking at <tt>lod_declare_destroy()</tt> it appears clear that this is doing a declare for every stripe, and not limiting itself when there are multiple objects per OST. </p> <p>Firstly, it is worthwhile to confirm that there is not a "sub handle" for each <b>stripe</b> instead of for each <b>OST</b>. That would both increase the RAM usage, and make it difficult to aggregate credits for a single OST's objects. Next, if <tt>ldo_dir_stripe_count &gt; ost_count</tt> to declare a bitmap for the number of OSTs and then limit the number of credits if that OST has already been assigned credits for logging the unlink of the objects.</p> <p>Alex should know the details, but I expect the number of credits needed to unlink an object are essentially one llog block with indirects to write an llog record (64 bytes for <tt>llog_unlink64_rec</tt>, if that is the right one, but close enough), plus a rare chance of creating a new llog file and inserting it into the <tt>O/1/dN</tt> directory. A new llog file might happen <b>once</b> per unlink per OST, but not 2000 times. Similarly, destroying 2000 objects doesn't take 4000 llog blocks with 34000 blocks. Instead, it would take either one llog block+ overhead per OST, or about 2000x64/4096 <b>bytes</b> =32KB = 8 blocks + 32 indirects if all the objects were on a single OST.</p> <p>Yes, the write/create declaration come from the llog record for unlinking all stripe objects.</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>like this example</b></div><div class="panelContent"> <p><span class="error">&#91;10033.829237&#93;</span> <span class="error">&#91;&lt;ffffffffa757a3ae&gt;&#93;</span> dump_stack+0x19/0x1b<br/> <span class="error">&#91;10033.829244&#93;</span> <span class="error">&#91;&lt;ffffffffc1908d74&gt;&#93;</span> osd_declare_write+0x434/0x540 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;10033.829255&#93;</span> <span class="error">&#91;&lt;ffffffffc104a150&gt;&#93;</span> llog_osd_declare_write_rec+0xe0/0x3a0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;10033.829265&#93;</span> <span class="error">&#91;&lt;ffffffffc103e196&gt;&#93;</span> llog_declare_write_rec+0x1e6/0x240 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;10033.829289&#93;</span> <span class="error">&#91;&lt;ffffffffc104475c&gt;&#93;</span> llog_cat_declare_add_rec+0x9c/0x260 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;10033.829297&#93;</span> <span class="error">&#91;&lt;ffffffffc103b3ef&gt;&#93;</span> llog_declare_add+0x14f/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;10033.829302&#93;</span> <span class="error">&#91;&lt;ffffffffc1bb75ba&gt;&#93;</span> osp_sync_declare_add+0x11a/0x490 <span class="error">&#91;osp&#93;</span><br/> <span class="error">&#91;10033.829306&#93;</span> <span class="error">&#91;&lt;ffffffffc1ba6443&gt;&#93;</span> osp_declare_destroy+0x1a3/0x200 <span class="error">&#91;osp&#93;</span><br/> <span class="error">&#91;10033.829312&#93;</span> <span class="error">&#91;&lt;ffffffffc1b54a6e&gt;&#93;</span> lod_sub_declare_destroy+0xce/0x2d0 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;10033.829318&#93;</span> <span class="error">&#91;&lt;ffffffffc1b2a51d&gt;&#93;</span> lod_obj_stripe_destroy_cb+0x8d/0xa0 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;10033.829325&#93;</span> <span class="error">&#91;&lt;ffffffffc1b37c8e&gt;&#93;</span> lod_obj_for_each_stripe+0x11e/0x2c0 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;10033.829330&#93;</span> <span class="error">&#91;&lt;ffffffffc1b38bf2&gt;&#93;</span> lod_declare_destroy+0x5c2/0x630 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;10033.829344&#93;</span> <span class="error">&#91;&lt;ffffffffc19c55f3&gt;&#93;</span> mdd_declare_finish_unlink+0x83/0x200 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;10033.829349&#93;</span> <span class="error">&#91;&lt;ffffffffc19d63af&gt;&#93;</span> mdd_unlink+0x4af/0xaf0 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;10033.829359&#93;</span> <span class="error">&#91;&lt;ffffffffc1a9cb9c&gt;&#93;</span> mdo_unlink+0x1b/0x1d <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;10033.829368&#93;</span> <span class="error">&#91;&lt;ffffffffc1a5893f&gt;&#93;</span> mdt_reint_unlink+0x9ff/0x1460 <span class="error">&#91;mdt&#93;</span></p> </div></div> <p> <br/> I think the "sub handle" for each <b>stripe</b> is eventually derived from the <b>device</b> of the stripes, so that's comforting.</p> <p>lod_sub_declare_destroy(env, dt, th)-&gt;lod_sub_get_thandle(env, th, dt, ...)</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lod_sub_declare_destroy(env, dt, th)</b></div><div class="codeContent panelContent"> <pre class="code-java"> 350 sub_th = lod_sub_get_thandle(env, th, dt, &amp;record_update); ... 357 rc = dt_declare_destroy(env, dt, sub_th); </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lod_sub_get_thandle(env, th, dt, ...)</b></div><div class="codeContent panelContent"> <pre class="code-java"> 104 sub_th = thandle_get_sub(env, th, dt); </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>thandle_get_sub(env, th, dt)</b></div><div class="codeContent panelContent"> <pre class="code-java"> 474 { 475 <span class="code-keyword">return</span> thandle_get_sub_by_dt(env, th, lu2dt_dev(dt-&gt;do_lu.lo_dev)); 476 } </pre> </div></div> <p>So the issue locates in that unlinking a wide-striped file reserves too much credits for llog records of unlinking its sub objects.</p> <p>Bobijam, are you able to work on a fix for this, or should Mike look into it?</p> <p>well, it's still possible (though not likely) that, for example, after some manual manipulations such an unlink will need to create that many llog files.</p> <p>I think we can postpone llog object removal using a dedicated thread, this would save some credits.</p> <p>Alex, in the overstriping case, how would it be possible for the unlink to have eg. 100 different llog files for a single OST? I am understand it would be possible to create one new llog file, and several new blocks in that file, but with 65000 records per llog file it isn't possible even under crazy cases to need so many credits. </p> <p>ok, I got it, how many different OSTs were actually involved?</p> <p>Needs Mike's input, llog_osd_declare_write_rec() just use loghandle-&gt;lgh_ctxt-&gt;loc_chunk_size instead of llog records len to write the llog, which is wasteful.</p> <p>a single new record involves:</p> <ul> <li>record itself, which can't be less than blocksize (4K for ldiskfs)</li> <li>llog header, which in turn can be two different blocks: 1st with llh_count (4K), 2nd with bitmap (4K)</li> </ul> <p>Alex, this issue can be seen in any review-dne-part-1 sanity test_27Cd (or other subtest using overstriping), for example:<br/> <a href="https://testing.whamcloud.com/test_sets/afe2d412-628d-4a7f-bda4-1c3359a97ff1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/afe2d412-628d-4a7f-bda4-1c3359a97ff1</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1863.395427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27Cd: test maximum stripe count =========== 11:50:04 \(1630583404\) [ 1863.871341] Lustre: DEBUG MARKER: == sanity test 27Cd: test maximum stripe count =========== 11:50:04 (1630583404) [ 1870.327497] Lustre: 11292:0:(osd_handler.c:1938:osd_trans_start()) lustre-MDT0003: credits 128195 &gt; trans_max 2592 [ 1870.329179] Lustre: 11292:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2000/8000/0, destroy: 1/4/0 [ 1870.330663] Lustre: 11292:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 2004/148/0 [ 1870.332184] Lustre: 11292:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 10001/86010/0, punch: 0/0/0, quota 6/6/0 [ 1870.333817] Lustre: 11292:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2001/34016/0, delete: 2/5/0 [ 1870.335318] Lustre: 11292:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 </pre> </div></div> <p>There are 4 MDTs and 8 OSTs in that case, and a 512MB transaction handle request.</p> <p>Tried a patch to use llog rec len instread of chunk_size in llog_osd_declare_write_rec(), the write credits demand decreased a little</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>patch</b></div><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/obdclass/llog_osd.c b/lustre/obdclass/llog_osd.c index 5aed12e9b9..18b1b80fdf 100644 --- a/lustre/obdclass/llog_osd.c +++ b/lustre/obdclass/llog_osd.c @@ -320,7 +320,6 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> llog_osd_declare_write_rec(<span class="code-keyword">const</span> struct lu_env *env, <span class="code-object">int</span> idx, struct thandle *th) { struct llog_thread_info *lgi = llog_info(env); - __u32 chunk_size; struct dt_object *o; <span class="code-object">int</span> rc; @@ -335,8 +334,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> llog_osd_declare_write_rec(<span class="code-keyword">const</span> struct lu_env *env, o = loghandle-&gt;lgh_obj; LASSERT(o); - chunk_size = loghandle-&gt;lgh_ctxt-&gt;loc_chunk_size; - lgi-&gt;lgi_buf.lb_len = chunk_size; + lgi-&gt;lgi_buf.lb_len = sizeof(struct llog_log_hdr); lgi-&gt;lgi_buf.lb_buf = NULL; <span class="code-comment">/* each time we update header */</span> rc = dt_declare_record_write(env, o, &amp;lgi-&gt;lgi_buf, 0, @@ -348,7 +346,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> llog_osd_declare_write_rec(<span class="code-keyword">const</span> struct lu_env *env, * the pad record can be inserted so take into account <span class="code-object">double</span> * record size */ - lgi-&gt;lgi_buf.lb_len = chunk_size * 2; + lgi-&gt;lgi_buf.lb_len = rec-&gt;lrh_len * 2; lgi-&gt;lgi_buf.lb_buf = NULL; <span class="code-comment">/* XXX: implement declared window or multi-chunks approach */</span> rc = dt_declare_record_write(env, o, &amp;lgi-&gt;lgi_buf, -1, th); </pre> </div></div> <p>sanity.sh test_27Cd<br/> credits demand before the patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2954.917326] Lustre: 20362:0:(osd_handler.c:1938:osd_trans_start()) lustre-MDT0001: credits 128195 &gt; trans_max 2496 [ 2954.917330] Lustre: 20362:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2000/8000/0, destroy: 1/4/0 [ 2954.917333] Lustre: 20362:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 2004/148/0 [ 2954.917335] Lustre: 20362:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 10001/86010/0, punch: 0/0/0, quota 6/6/0 [ 2954.917337] Lustre: 20362:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2001/34016/0, delete: 2/5/0 [ 2954.917339] Lustre: 20362:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 </pre> </div></div> <p>credits demand after the patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 3371.600695] Lustre: 30759:0:(osd_handler.c:1938:osd_trans_start()) lustre-MDT0000: credits 92195 &gt; trans_max 2496 [ 3371.600699] Lustre: 30759:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2000/8000/0, destroy: 1/4/0 [ 3371.600701] Lustre: 30759:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 2004/148/0 [ 3371.600704] Lustre: 30759:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 10001/50010/0, punch: 0/0/0, quota 6/6/0 [ 3371.600706] Lustre: 30759:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2001/34016/0, delete: 2/5/0 [ 3371.600708] Lustre: 30759:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 </pre> </div></div> <p>this means that loc_chunk_size is not LLOG_MIN_CHUNK_SIZE for unlink llogs?</p> <p>Alex, as I understand patch uses just header size and <tt>rec-&gt;lrh_len</tt> to declare llog write but not <tt>loc_chunk_size</tt> so yes, both are not <tt>LLOG_MIN_CHUNK_SIZE</tt>.</p> <p>As I remember we were using <tt>loc_chunk_size</tt> in declaration just because we don't know where record (and bit in bitmap) will be really written due to llog append nature, that is why we are declaring whole chunk for header and current chunk along with next one for the record. So we can't replace that with declaration of just record size and header size.</p> <p>Note that write itself is more narrowed, it is done with exact offset/len values as they are known at the moment of write.</p> <p>yes, I don't think this change is correct:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + lgi-&gt;lgi_buf.lb_len = rec-&gt;lrh_len * 2; </pre> </div></div> <p>as lrh_len can be more than remaining space in the last block we may need to write a padding record (last block N), then add the new record to block N+1</p> <p>what we can do, I guess, is to declare that potential padding block not as an append, but a record within llog's size which would mean a single block to modify (in contrast of append which may need many more blocks - allocation, externt tree, etc)</p> <p>but I think this way we don't save much, like Andreas said instead we should be looking for ways to declare at most OSTCOUNT new plain llogs and #stripes records. what about the case when lots of MDT threads are unlinking such overstriped files concurrently?<br/> 64K records / 128 threads = 512 records per thread</p> <p>also as I remember that was mostly due to ZFS which needs exact offset/len to be declared and complains if write doesn't match. So as we can't say for sure exact offset at the declaration time we assume range where it can go - the current chunk and the next one. And also we should take case about non-atomic 'declare'-'write' nature, I think there could be concurrent record additions in between, can't they?</p> <p>Mike, I think that was later fixed by Alex to remove the "specific declare" since that hurt performance and didn't save much space...</p> <p>As for concurrent records, that would be mostly fine. As I wrote before, there <b>may</b> be some interleaving of records, but not 65000 different records written from other threads between each delete of the overstriped object that would trigger a new llog file each time.</p> <p>I'd like to still keep this under consideration for 2.15.0 since it is hit very often during usage, and the transaction size (over 500MB) can hurt performance significantly.</p> <p>Assign to Alex to balance 2.15 tickets. </p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45765" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45765</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14918" title="too many ldiskfs transaction credits when unlinking overstriped files" class="issue-link" data-issue-key="LU-14918"><del>LU-14918</del></a> osd: don't declare similar writes twice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 54aef59ac5bd349a3d42b71b3d0bdc9cda93066e</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49701" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49701</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14918" title="too many ldiskfs transaction credits when unlinking overstriped files" class="issue-link" data-issue-key="LU-14918"><del>LU-14918</del></a> osd: don't declare similar writes twice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4a86ff32d18b6598880df1ca19e16af5a8b781b</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45765/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45765/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14918" title="too many ldiskfs transaction credits when unlinking overstriped files" class="issue-link" data-issue-key="LU-14918"><del>LU-14918</del></a> osd: don't declare similar ldiskfs writes twice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e6225b2e7385cbb7be0474df01075fafc4966d5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49701/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14918" title="too many ldiskfs transaction credits when unlinking overstriped files" class="issue-link" data-issue-key="LU-14918"><del>LU-14918</del></a> osd: don't declare similar zfs writes twice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c1936c9d294d53ff39741e1b07ffc74f51fcddb6</p> <p>Landed for 2.16</p> Related LU-14965 LU-15380 LU-5890 Development Rank 1|i0215z: Rank (Obsolete) 9223372036854775807 Severity