https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4479 Lustre <p>Lustre: lustre-OST0027: deleting orphan objects from 0x0:216 to 0x0:257<br/> Lustre: 23291:0:(osd_handler.c:897:osd_trans_start()) lustre-MDT0000: too many transaction credits (7166 &gt; 1024)<br/> Lustre: 23291:0:(osd_handler.c:904:osd_trans_start()) create: 40/1000, delete: 2/35, destroy: 1/25<br/> Lustre: 23291:0:(osd_handler.c:909:osd_trans_start()) attr_set: 2/2, xattr_set: 41/574<br/> Lustre: 23291:0:(osd_handler.c:916:osd_trans_start()) write: 322/4508, punch: 80/320, quota 2/2<br/> Lustre: 23291:0:(osd_handler.c:921:osd_trans_start()) insert: 41/696, delete: 1/25<br/> Lustre: 23291:0:(osd_handler.c:926:osd_trans_start()) ref_add: 1/1, ref_del: 3/3<br/> Pid: 23291, comm: mdt00_004</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa03ea8c5&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c1f393&gt;&#93;</span> osd_trans_start+0x653/0x670 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e26529&gt;&#93;</span> lod_trans_start+0x1b9/0x250 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d1b187&gt;&#93;</span> mdd_trans_start+0x17/0x20 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d0ff6f&gt;&#93;</span> mdd_unlink+0x42f/0xd20 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d77d91&gt;&#93;</span> ? mdt_reint_unlink+0x871/0x10f0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d74738&gt;&#93;</span> mdo_unlink+0x18/0x50 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d77dc8&gt;&#93;</span> mdt_reint_unlink+0x8a8/0x10f0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d74431&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d5ad53&gt;&#93;</span> mdt_reint_internal+0x4c3/0x780 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d5b59b&gt;&#93;</span> mdt_reint+0x6b/0x120 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa081b995&gt;&#93;</span> tgt_request_handle+0x245/0xad0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07cc261&gt;&#93;</span> ptlrpc_main+0xce1/0x1960 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07cb580&gt;&#93;</span> ? ptlrpc_main+0x0/0x1960 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810996b6&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c1ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffff81534750&gt;&#93;</span> ? _spin_unlock_irq+0x30/0x40<br/> <span class="error">&#91;&lt;ffffffff8100bb10&gt;&#93;</span> ? restore_args+0x0/0x30<br/> <span class="error">&#91;&lt;ffffffff81099620&gt;&#93;</span> ? kthread+0x0/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c1c0&gt;&#93;</span> ? child_rip+0x0/0x20s</p> OSTCOUNT=40 sh sanity.sh LU-4479

Credit error with unlink for large number OSTs

Bug Critical Resolved Duplicate Niu Yawei Alexey Lyashkov Tue, 14 Jan 2014 08:53:41 +0000 Mon, 20 Jun 2016 06:24:40 +0000 Mon, 20 Jun 2016 06:24:39 +0000 Lustre 2.6.0 0 6 <p>Note that this debug message is only printed when transaction credit checking is enabled during test-framework test runs (i.e. llmount.sh and friends). This debugging is not enabled by default for normal users. </p> <p>The core problem is that your MDT journal is too small for so many OSTs - only 16 MB. At the same time, 7166 blocks is probably far more than what is actually needed for transaction credits on a small MDT filesystem. </p> <p>Two options exist to fix this :</p> <ul class="alternate" type="square"> <li>improve credit calculations to avoid overhead that is counted multiple times (e.g. superblock, extra llog files, and such), but I don't think that will remove 6000+ blocks from the transaction. The benefit is that could actually improve MDS performance, since it will not request as many extra blocks, and avoid committing/checkpointing transactions early.</li> <li>fix test-framework.sh to format a larger journal when there are so many OSTs. This is more like real life, but doesn't fix the root problem of such large transactions (they only get worse with more MDTs)</li> </ul> <p>It's don't only limited to the unlink</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 26062:0:(osd_handler.c:926:osd_trans_start()) ref_add: 0/0, ref_del: 0/0 Pid: 26062, comm: mdt01_005 Call Trace: [&lt;ffffffffa03ea8c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0c1f393&gt;] osd_trans_start+0x653/0x670 [osd_ldiskfs] [&lt;ffffffffa0e26529&gt;] lod_trans_start+0x1b9/0x250 [lod] [&lt;ffffffffa0d1b187&gt;] mdd_trans_start+0x17/0x20 [mdd] [&lt;ffffffffa0d0ecc9&gt;] mdd_create+0x929/0x17a0 [mdd] [&lt;ffffffffa0c27cb7&gt;] ? osd_xattr_get+0x97/0x2e0 [osd_ldiskfs] [&lt;ffffffffa0d7f968&gt;] mdo_create+0x18/0x50 [mdt] [&lt;ffffffffa0d89bda&gt;] mdt_reint_open+0x12aa/0x2120 [mdt] [&lt;ffffffff8153480b&gt;] ? _spin_unlock+0x2b/0x40 [&lt;ffffffffa04073fc&gt;] ? upcall_cache_get_entry+0x3dc/0x8a0 [libcfs] [&lt;ffffffffa05aab00&gt;] ? lu_ucred_global_init+0x20/0x30 [obdclass] [&lt;ffffffffa0d74431&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0d5ad53&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0d5b205&gt;] mdt_intent_reint+0x1f5/0x520 [mdt] [&lt;ffffffffa0d59929&gt;] mdt_intent_policy+0x499/0xca0 [mdt] [&lt;ffffffffa0771502&gt;] ldlm_lock_enqueue+0x2f2/0x8b0 [ptlrpc] [&lt;ffffffffa079acff&gt;] ldlm_handle_enqueue0+0x4ff/0x10d0 [ptlrpc] [&lt;ffffffffa081b5e2&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] [&lt;ffffffffa081b995&gt;] tgt_request_handle+0x245/0xad0 [ptlrpc] [&lt;ffffffffa07cc261&gt;] ptlrpc_main+0xce1/0x1960 [ptlrpc] [&lt;ffffffffa07cb580&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [&lt;ffffffff810996b6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81534750&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff81099620&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 Pid: 10821, comm: ldlm_elt Call Trace: [&lt;ffffffffa03ea8c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0c1f393&gt;] osd_trans_start+0x653/0x670 [osd_ldiskfs] [&lt;ffffffffa0e26529&gt;] lod_trans_start+0x1b9/0x250 [lod] [&lt;ffffffffa0d1b187&gt;] mdd_trans_start+0x17/0x20 [mdd] [&lt;ffffffffa0d02ec0&gt;] mdd_close+0x250/0xb80 [mdd] [&lt;ffffffffa0d83269&gt;] mdt_mfd_close+0x349/0x1bb0 [mdt] [&lt;ffffffffa058a16b&gt;] ? keys_fini+0xcb/0x110 [obdclass] [&lt;ffffffffa058a1dd&gt;] ? lu_context_fini+0x2d/0xb0 [obdclass] [&lt;ffffffffa0d56551&gt;] ? mdt_ctxt_add_dirty_flag+0x141/0x190 [mdt] [&lt;ffffffffa0d5693a&gt;] mdt_obd_disconnect+0x39a/0x530 [mdt] [&lt;ffffffffa0554140&gt;] class_fail_export+0x250/0x560 [obdclass] [&lt;ffffffffa07959f4&gt;] expired_lock_main+0x4a4/0x870 [ptlrpc] [&lt;ffffffff81064650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0795550&gt;] ? expired_lock_main+0x0/0x870 [ptlrpc] [&lt;ffffffff810996b6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81534750&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff81099620&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>it's global problem in lustre.</p> <p>See comments for overall issues in transaction accounting in <a href="https://jira.whamcloud.com/browse/LU-4611" title="too many transaction credits (32279 &gt; 25600) " class="issue-link" data-issue-key="LU-4611"><del>LU-4611</del></a>. I think there are a number of cases where the OSD operations are accounting for transaction credits that are only useful for the OST or the MDT, but not the reverse.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>create: 40/1000, delete: 2/35, destroy: 1/25 attr_set: 2/2, xattr_set: 41/574 write: 322/4508, punch: 80/320, quota 2/2 insert: 41/696, delete: 1/25 ref_add: 1/1, ref_del: 3/3 </pre> </div></div> <p>It looks like one obvious fix for the unlink transaction accounting here is to avoid registering for punch operations when the inode has no blocks allocated. That would avoid accounting for punch operations on the MDT objects, which almost never have blocks allocated except for widely-striped files.</p> <p>Similarly, it isn't necessary that there are credits for create (for llog destroy records) reserved for every unlink, only when there is no current llog file for that OSP or if the current llog is close to being full.</p> <p>I think this can be fixed at the same time as <a href="https://jira.whamcloud.com/browse/LU-4611" title="too many transaction credits (32279 &gt; 25600) " class="issue-link" data-issue-key="LU-4611"><del>LU-4611</del></a>, but I think that some of the changes are unlink specific so I'm keeping this bug separate for now.</p> <p>Just hit this in my testing. It was with sanity test 300k</p> <p>Hmm...hit similar problem on our local testing (racer.sh on rhel7.2):<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1092:osd_trans_start()) lustre-MDT0001: too many transaction credits (533 &gt; 1024)<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1097:osd_trans_start()) create: 6/24, destroy: 0/0<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1102:osd_trans_start()) attr_set: 2/2, xattr_set: 5/57<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1109:osd_trans_start()) write: 2/4, punch: 0/0, quota 2/2<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1114:osd_trans_start()) insert: 18/240, delete: 0/0<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1119:osd_trans_start()) ref_add: 4/4, ref_del: 0/0<br/> Jun 15 12:00:31 localhost kernel: Pid: 30911, comm: mdt00_002<br/> Jun 15 12:00:31 localhost kernel: #012Call Trace:<br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa08bb853&gt;&#93;</span> libcfs_debug_dumpstack+0x53/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0f49c92&gt;&#93;</span> osd_trans_start+0x642/0x670 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa1133537&gt;&#93;</span> lod_trans_start+0x67/0xf0 <span class="error">&#91;lod&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa11be5a4&gt;&#93;</span> mdd_trans_start+0x14/0x20 <span class="error">&#91;mdd&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa11a864d&gt;&#93;</span> mdd_create+0x9ed/0x12a0 <span class="error">&#91;mdd&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa109f2ce&gt;&#93;</span> mdt_reint_create+0xb3e/0xc70 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0b0937e&gt;&#93;</span> ? lu_ucred+0x1e/0x30 <span class="error">&#91;obdclass&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa108ff35&gt;&#93;</span> ? mdt_ucred+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa1090211&gt;&#93;</span> ? mdt_root_squash+0x21/0x410 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0d4c0bf&gt;&#93;</span> ? lustre_pack_reply_flags+0x6f/0x1e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa10a0ce0&gt;&#93;</span> mdt_reint_rec+0x80/0x210 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa1082699&gt;&#93;</span> mdt_reint_internal+0x5d9/0xb40 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa108cc27&gt;&#93;</span> mdt_reint+0x67/0x140 <span class="error">&#91;mdt&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0db583b&gt;&#93;</span> tgt_request_handle+0x8fb/0x11f0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0d5896b&gt;&#93;</span> ptlrpc_server_handle_request+0x21b/0xa90 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0d55a38&gt;&#93;</span> ? ptlrpc_wait_event+0x98/0x330 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffff810af018&gt;&#93;</span> ? __wake_up_common+0x58/0x90<br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0d5c290&gt;&#93;</span> ptlrpc_main+0xc00/0x1f50 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffffa0d5b690&gt;&#93;</span> ? ptlrpc_main+0x0/0x1f50 <span class="error">&#91;ptlrpc&#93;</span><br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffff810a5acf&gt;&#93;</span> kthread+0xcf/0xe0<br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffff810a5a00&gt;&#93;</span> ? kthread+0x0/0xe0<br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffff81646018&gt;&#93;</span> ret_from_fork+0x58/0x90<br/> Jun 15 12:00:31 localhost kernel: <span class="error">&#91;&lt;ffffffff810a5a00&gt;&#93;</span> ? kthread+0x0/0xe0<br/> Jun 15 12:00:31 localhost kernel:</p> <p>Lustre is based on 2.7.15.3, with additional patch, 1 MGS, 2 OSTs, 2 Clients</p> <blockquote> <p>Hmm...hit similar problem on our local testing (racer.sh on rhel7.2):<br/> Jun 15 12:00:31 localhost kernel: Lustre: 30911:0:(osd_handler.c:1092:osd_trans_start()) lustre-MDT0001: too many transaction credits (533 &gt; 1024)</p></blockquote> <p>This looks a tiny testing system with 1024 maximum journal buffers, it's not surprise to see such kind of warning messages. In the latest master code, such messages have been suppressed for tiny systems.</p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-4611" title="too many transaction credits (32279 &gt; 25600) " class="issue-link" data-issue-key="LU-4611"><del>LU-4611</del></a>.</p> Duplicate LU-4611 Development Rank 1|hzwcrb: Rank (Obsolete) 12270 Severity