https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5794 Lustre <p>the current schema can't detect how the credits are consumed while a transaction is going. so, it's hard to understand which specific operation consumes too much. also, the schema doesn't support rollback operations properly which may cause false assertions.</p> ldiskfs LU-5794

improve credit checks in osd-ldiskfs

Improvement Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Thu, 23 Oct 2014 09:31:13 +0000 Tue, 8 Nov 2016 18:59:20 +0000 Fri, 10 Jul 2015 12:10:26 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 0 7 <p>the proposed patch - <a href="http://review.whamcloud.com/#/c/12169/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12169/</a> - tracks how many credits every operation has used and asserts when too many credits were consumed. it reuses the existing rollback table of the operations to allow zero-credits reverse operations (e.g. create <del>&gt; destroy, insert</del>&gt;delete, etc).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:14:23:Lustre: DEBUG MARKER: == recovery-small test 110g: drop reply during migration == 05:14:10 (1412745250) 23:14:23:Lustre: 7852:0:(osd_internal.h:1048:osd_trans_exec_check()) op 9: used 4, used now 4, reserved 1 23:14:23:Lustre: 7852:0:(osd_handler.c:985:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 23:14:23:Lustre: 7852:0:(osd_handler.c:992:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 1/64/0 23:14:23:Lustre: 7852:0:(osd_handler.c:1002:osd_trans_dump_creds()) write: 2/4/0, punch: 0/0/0, quota 2/2/0 23:14:23:Lustre: 7852:0:(osd_handler.c:1009:osd_trans_dump_creds()) insert: 3/50/0, delete: 1/1/4 23:14:23:Lustre: 7852:0:(osd_handler.c:1016:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 23:14:23:LustreError: 7852:0:(osd_internal.h:1050:osd_trans_exec_check()) LBUG 23:14:23:Pid: 7852, comm: mdt_out00_002 23:14:23: 23:14:23:Call Trace: 23:14:23: [&lt;ffffffffa0760895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 23:14:23: [&lt;ffffffffa0760e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 23:14:23: [&lt;ffffffffa1396dfc&gt;] osd_index_ea_delete+0x9cc/0xd80 [osd_ldiskfs] 23:14:23: [&lt;ffffffffa0dff316&gt;] out_obj_index_delete+0x156/0x1f0 [ptlrpc] 23:14:23: [&lt;ffffffffa0e061f8&gt;] out_tx_index_delete_exec+0x28/0x170 [ptlrpc] 23:14:23: [&lt;ffffffffa0dfea2a&gt;] out_tx_end+0xda/0x5c0 [ptlrpc] 23:14:23: [&lt;ffffffffa0e05480&gt;] out_handle+0x7c0/0xe50 [ptlrpc] 23:14:23: [&lt;ffffffffa0dfba9e&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] 23:14:23: [&lt;ffffffffa0dab384&gt;] ptlrpc_main+0xe64/0x1990 [ptlrpc] 23:14:23: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:41:14:Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ================ 07:40:48 (1413358848) 01:41:14:Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 01:41:14:LustreError: 7196:0:(fail.c:132:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms 01:41:14:LustreError: 7196:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake 01:41:14:Lustre: 7195:0:(osd_internal.h:1048:osd_trans_exec_check()) op 9: used 6, used now 6, reserved 1 01:41:14:Lustre: 7195:0:(osd_handler.c:985:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 01:41:14:Lustre: 7195:0:(osd_handler.c:992:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 1/64/0 01:41:14:Lustre: 7195:0:(osd_handler.c:1002:osd_trans_dump_creds()) write: 2/4/0, punch: 0/0/0, quota 2/2/0 01:41:14:Lustre: 7195:0:(osd_handler.c:1009:osd_trans_dump_creds()) insert: 0/0/0, delete: 1/1/6 01:41:14:Lustre: 7195:0:(osd_handler.c:1016:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0 01:41:14:LustreError: 7195:0:(osd_internal.h:1050:osd_trans_exec_check()) LBUG 01:41:14:Pid: 7195, comm: mdt_out00_002 01:41:14: 01:41:14:Call Trace: 01:41:14: [&lt;ffffffffa0483895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 01:41:14: [&lt;ffffffffa0483e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 01:41:14: [&lt;ffffffffa0d2cdfc&gt;] osd_index_ea_delete+0x9cc/0xd80 [osd_ldiskfs] 01:41:14: [&lt;ffffffffa0884316&gt;] out_obj_index_delete+0x156/0x1f0 [ptlrpc] 01:41:14: [&lt;ffffffffa088b1f8&gt;] out_tx_index_delete_exec+0x28/0x170 [ptlrpc] 01:41:14: [&lt;ffffffffa0883a2a&gt;] out_tx_end+0xda/0x5c0 [ptlrpc] 01:41:14: [&lt;ffffffffa088a480&gt;] out_handle+0x7c0/0xe50 [ptlrpc] 01:41:14: [&lt;ffffffffa0880a9e&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] 01:41:14: [&lt;ffffffffa0830384&gt;] ptlrpc_main+0xe64/0x1990 [ptlrpc] </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:47:27:Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) == 00:47:21 (1412729241) 18:47:27:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161e 18:47:27:Lustre: *** cfs_fail_loc=161e, val=0*** 18:47:27:Lustre: Skipped 3 previous similar messages 18:47:27:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 18:47:27:Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r 18:47:27:Lustre: 21068:0:(osd_internal.h:1048:osd_trans_exec_check()) op 9: used 4, used now 4, reserved 1 18:47:27:LustreError: 21068:0:(osd_internal.h:1050:osd_trans_exec_check()) LBUG 18:47:27:Pid: 21068, comm: lfsck_namespace 18:47:27: 18:47:27:Call Trace: 18:47:27: [&lt;ffffffffa0483895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 18:47:27: [&lt;ffffffffa0483e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 18:47:27: [&lt;ffffffffa0d2cdfc&gt;] osd_index_ea_delete+0x9cc/0xd80 [osd_ldiskfs] 18:47:27: [&lt;ffffffffa0fbe2d7&gt;] lod_index_delete+0xa7/0x140 [lod] 18:47:27: [&lt;ffffffffa0e50005&gt;] dt_delete+0x95/0x1a0 [lfsck] 18:47:27: [&lt;ffffffffa0e56182&gt;] lfsck_namespace_repair_unmatched_pairs+0x6e2/0x840 [lfsck] 18:47:27: [&lt;ffffffffa0e56939&gt;] lfsck_namespace_dsd_single+0x659/0x7e0 [lfsck] 18:47:27: [&lt;ffffffffa0e57309&gt;] lfsck_namespace_double_scan_dir+0x849/0x1590 [lfsck] 18:47:27: [&lt;ffffffffa0e5cb44&gt;] lfsck_namespace_double_scan_one+0x2f4/0x1250 [lfsck] 18:47:27: [&lt;ffffffffa0e5e278&gt;] lfsck_namespace_assistant_handler_p2+0x7d8/0xa50 [lfsck] 18:47:27: [&lt;ffffffffa0e4b9ed&gt;] lfsck_assistant_engine+0x130d/0x1c50 [lfsck] </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:05:40:Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 10:05:29 (1412589929) 04:05:40:Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000440000400-0x0000000480000400):1:mdt] 04:05:40:Lustre: 3448:0:(osd_internal.h:1048:osd_trans_exec_check()) op 9: used 6, used now 6, reserved 1 04:05:40:Lustre: 3448:0:(osd_handler.c:985:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 04:05:40:Lustre: 3448:0:(osd_handler.c:992:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 1/64/0 04:05:40:Lustre: 3448:0:(osd_handler.c:1002:osd_trans_dump_creds()) write: 2/4/0, punch: 0/0/0, quota 2/2/0 04:05:40:Lustre: 3448:0:(osd_handler.c:1009:osd_trans_dump_creds()) insert: 0/0/0, delete: 1/1/6 04:05:40:Lustre: 3448:0:(osd_handler.c:1016:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0 04:05:40:LustreError: 3448:0:(osd_internal.h:1050:osd_trans_exec_check()) LBUG 04:05:40:Pid: 3448, comm: mdt_out00_001 04:05:40: 04:05:40:Call Trace: 04:05:40: [&lt;ffffffffa0483895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 04:05:40: [&lt;ffffffffa0483e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 04:05:41: [&lt;ffffffffa0d2cdfc&gt;] osd_index_ea_delete+0x9cc/0xd80 [osd_ldiskfs] 04:05:41: [&lt;ffffffffa0884316&gt;] out_obj_index_delete+0x156/0x1f0 [ptlrpc] 04:05:41: [&lt;ffffffffa088b1f8&gt;] out_tx_index_delete_exec+0x28/0x170 [ptlrpc] 04:05:41: [&lt;ffffffffa0883a2a&gt;] out_tx_end+0xda/0x5c0 [ptlrpc] 04:05:41: [&lt;ffffffffa088a480&gt;] out_handle+0x7c0/0xe50 [ptlrpc] 04:05:41: [&lt;ffffffffa0880a9e&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] 04:05:41: [&lt;ffffffffa0830384&gt;] ptlrpc_main+0xe64/0x1990 [ptlrpc] </pre> </div></div> <p>if I understand correctly, this is an delete in the index where an agent inode needs to be removed.</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12169" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12169</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5794" title="improve credit checks in osd-ldiskfs" class="issue-link" data-issue-key="LU-5794"><del>LU-5794</del></a> osd: additional checks to verify credits calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 13<br/> Commit: 6e77a93dca837d9f395bced9262ad0de2e2c81f7</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12169" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12169</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5794" title="improve credit checks in osd-ldiskfs" class="issue-link" data-issue-key="LU-5794"><del>LU-5794</del></a> osd: additional checks to verify credits calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 14<br/> Commit: 3db6478913dac65b7bb8c9585f0ab56c39e1b192</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12169/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12169/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5794" title="improve credit checks in osd-ldiskfs" class="issue-link" data-issue-key="LU-5794"><del>LU-5794</del></a> osd: additional checks to verify credits calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 92d28a7f35bcdc718e2bdcec48bcdeacd0e8f4fe</p> <p>Landed for 2.8</p> Duplicate Related LU-5770 LU-4528 Development Rank 1|hzwzbz: Rank (Obsolete) 16258