https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5604 Lustre <p>It looks to me that lots of FAIL_ID checking are lost from time to time, take the replay-single.sh as an example:</p> <ul class="alternate" type="square"> <li>test_73c() checked OBD_FAIL_TGT_LAST_REPLAY, but this FAIL_ID is never being checked in Lustre code from the day one it was introduced.</li> </ul> <ul class="alternate" type="square"> <li>test_73b() checked OBD_FAIL_LDLM_REPLY, but this FAIL_ID is now only checked in mdt_reint_open(), I think it should be checked for every lock enqueue as well.</li> </ul> <ul class="alternate" type="square"> <li>test_73a() checked OBD_FAIL_LDLM_ENQUEUE_NET, but this FAIL_ID is not being checked in Lustre code anymore.</li> </ul> <ul class="alternate" type="square"> <li>test_80c() checked OBD_FAIL_UPDATE_OBJ_NET_REP, but this FAIL_ID has been removed from Lustre code.</li> </ul> <ul class="alternate" type="square"> <li>test_83a() checked OBD_FAIL_MDS_FAIL_LOV_LOG_ADD, but this FAIL_ID isn't checked in Lustre code.<br/> ...</li> </ul> <p>To make sure the error injection test working as expected, I think we'd go through all the fail IDs, and add back all the missed fail_id checking. If some FAIL_ID is obsolete already, we'd remove or improve the corresponding test case.</p> LU-5604

Lots of FAIL_ID checking are lost

Bug Critical Resolved Fixed Mikhail Pershin Niu Yawei MB Wed, 10 Sep 2014 13:59:11 +0000 Mon, 25 Nov 2019 16:30:08 +0000 Tue, 24 Feb 2015 07:34:43 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.7.0 Lustre 2.13.0 0 10 <p>Note that OBD_FAIL_LDLM_ENQUEUE_NET is referenced but unfortunately in a nonobvious way:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/* * Unified target <span class="code-keyword">generic</span> handers macros and <span class="code-keyword">generic</span> functions. */ #define TGT_RPC_HANDLER_HP(base, flags, opc, fn, hp, fmt, version) \ [opc - base] = { \ .th_name = #opc, \ .th_fail_id = OBD_FAIL_ ## opc ## _NET, \ .th_opc = opc, \ .th_flags = flags, \ .th_act = fn, \ .th_fmt = fmt, \ .th_version = version, \ .th_hp = hp, \ } </pre> </div></div> <p>I see that the following are referenced somewhere in lustre/tests/ but never used in the lustre source.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OBD_FAIL_MDS_OST_SETATTR OBD_FAIL_MDS_LLOG_SYNC_TIMEOUT OBD_FAIL_MDS_BLOCK_QUOTA_REQ OBD_FAIL_MDS_DROP_QUOTA_REQ OBD_FAIL_MDS_FAIL_LOV_LOG_ADD OBD_FAIL_MDS_LOV_PREP_CREATE OBD_FAIL_MDS_OPEN_WAIT_CREATE OBD_FAIL_OST_SETATTR_CREDITS OBD_FAIL_OST_HOLD_WRITE_RPC OBD_FAIL_OST_LLOG_RECOVERY_TIMEOUT OBD_FAIL_OST_CANCEL_COOKIE_TIMEOUT OBD_FAIL_OST_PAUSE_CREATE OBD_FAIL_OST_NOMEM OBD_FAIL_OST_BRW_PAUSE_BULK2 OBD_FAIL_OSC_DIO_PAUSE OBD_FAIL_PTLRPC_DELAY_RECOV OBD_FAIL_PTLRPC_DELAY_IMP_FULL OBD_FAIL_OBD_DQACQ OBD_FAIL_TGT_DELAY_PRECREATE OBD_FAIL_TGT_LAST_REPLAY OBD_FAIL_MDS_SYNC_CAPA_SL </pre> </div></div> <p>Mike, since you did most of the changes to the target unification, and will be further changing the client/server for DoM I think it makes sense for you to work on this to ensure we are not adding (more?) regressions in this area.</p> <p>I can take this, yes, and check all current FAIL_IDs, though I am not sure how to ensure there will not be new regressions.</p> <p>Comment from duplicated ticket by Vitaly:</p> <blockquote> <p>afaics, there are a number of "unified commit handlers" commits which broke some of IT tests:<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> mdt: call MDT handlers via unified request handler<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> ofd: use unified handler for OST requests<br/> <a href="https://jira.whamcloud.com/browse/LU-2145" title="The llog_origin_handle_create operation failed with -2" class="issue-link" data-issue-key="LU-2145"><del>LU-2145</del></a> server: use unified request handler for MGS<br/> etc<br/> there were different fail_loc which were set&amp;checked per RPC operation code (e.g. OBD_FAIL_LDLM_REPLY, OBD_FAIL_OST_LDLM_REPLY_NET, OBD_FAIL_OST_LDLM_REPLY_NET, etc) which were removed. also looking at the code, it is not clear that all the OBD/CFS_FAIL macros and defines are properly ported to the new code.<br/> as an example - OBD_FAIL_LDLM_REPLY is not used anymore, but the following tests expect it to be used:<br/> recovery-small 53<br/> recovery-small 113<br/> replay-dual 19<br/> replay-single 52<br/> replay-single 73b<br/> therefore tests do nothing now.</p></blockquote> <p>Inspected commits<br/> ----------------------------<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> target: generic hpreq handler in target<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> target: use osc_reply_portal for OUT services<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> ptlrpc: initialize request session early<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> target: unified transaction callbacks<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> ofd: use unified handler for OST requests<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> mdt: call MDT handlers via unified request handler<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> seq: unified SEQ handler<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> target: FLD to use unified request handler<br/> <a href="https://jira.whamcloud.com/browse/LU-3467" title="Unified request handler on OST" class="issue-link" data-issue-key="LU-3467"><del>LU-3467</del></a> target: move OUT to the unified target code<br/> <a href="https://jira.whamcloud.com/browse/LU-2145" title="The llog_origin_handle_create operation failed with -2" class="issue-link" data-issue-key="LU-2145"><del>LU-2145</del></a> server: use unified request handler for MGS<br/> <a href="https://jira.whamcloud.com/browse/LU-2145" title="The llog_origin_handle_create operation failed with -2" class="issue-link" data-issue-key="LU-2145"><del>LU-2145</del></a> target: use tgt_ prefix for target function<br/> <a href="https://jira.whamcloud.com/browse/LU-2145" title="The llog_origin_handle_create operation failed with -2" class="issue-link" data-issue-key="LU-2145"><del>LU-2145</del></a> target: move target code to the separate directory</p> <p>After going through above mention commits, found that following fail_id macros (changed)</p> <p>1. Assignment Removed<br/> -----------------------------------<br/> OBD_FAIL_LDLM_REPLY<br/> OBD_FAIL_OST_ALL_REPLY_NET<br/> OBD_FAIL_OST_LDLM_REPLY_NET<br/> OBD_FAIL_MDS_ALL_REPLY_NET<br/> OBD_FAIL_MGS_ALL_REPLY_NET</p> <p>2. Conditional check removed<br/> -----------------------------------------<br/> OBD_FAIL_OST_EROFS<br/> OBD_FAIL_OST_CONNECT_NET<br/> OBD_FAIL_OST_CONNECT_NET2<br/> OBD_FAIL_OST_DISCONNECT_NET<br/> OBD_FAIL_OST_CREATE_NET<br/> OBD_FAIL_OST_DESTROY_NET<br/> OBD_FAIL_OST_GETATTR_NET<br/> OBD_FAIL_OST_SETATTR_NET<br/> OBD_FAIL_OST_ENOSPC<br/> OBD_FAIL_OST_PUNCH_NET<br/> OBD_FAIL_OST_STATFS_NET<br/> OBD_FAIL_OST_SYNC_NET<br/> OBD_FAIL_OST_QUOTACHECK_NET<br/> OBD_FAIL_OST_QUOTACTL_NET<br/> OBD_FAIL_LDLM_ENQUEUE_NET<br/> OBD_FAIL_LDLM_CONVERT_NET<br/> OBD_FAIL_LDLM_CANCEL_NET<br/> OBD_FAIL_MDS_ALL_REQUEST_NET<br/> OBD_FAIL_MGS_CONNECT_NET</p> <p>Checked the above macros and found that following marcos need to be ported to repair the tests.</p> <p>OBD_FAIL_OST_LDLM_REPLY_NET<br/> OBD_FAIL_LDLM_REPLY<br/> OBD_FAIL_OST_CREATE_NET<br/> OBD_FAIL_LDLM_ENQUEUE</p> <p>Hence created the patch for the same.<br/> Patch link : <a href="http://review.whamcloud.com/#/c/12203/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12203/</a></p> <p>Please review and let me know if any thing is missed or needs correction.</p> <p>Please note that all fail ids to simulate lost request are not missed but defined in TGT_RPC_HANDLER_HP macro, see John comment above. I know it is done in non-obvious way and you can't find those fail ids doing grep, but that is so since Lustre 2.0 when new MDT stack was introduced. I am going to add comment containing fail id name to each handler, so anyone can find it by grep.</p> <blockquote> <p>1. Assignment Removed<br/> -----------------------------------<br/> OBD_FAIL_LDLM_REPLY<br/> OBD_FAIL_OST_ALL_REPLY_NET<br/> OBD_FAIL_OST_LDLM_REPLY_NET<br/> OBD_FAIL_MDS_ALL_REPLY_NET<br/> OBD_FAIL_MGS_ALL_REPLY_NET</p></blockquote> <p>OBD_FAIL_OST_ALL_REPLY_NET<br/> OBD_FAIL_MDS_ALL_REPLY_NET<br/> OBD_FAIL_MGS_ALL_REPLY_NET<br/> all IDs above are passed to tgt_init() by ofd, mdt and mgs. They are stored in lu_target::lut_reply_fail_id for further use.</p> <blockquote> <p>2. Conditional check removed<br/> -----------------------------------------<br/> OBD_FAIL_OST_EROFS<br/> OBD_FAIL_OST_CONNECT_NET<br/> OBD_FAIL_OST_CONNECT_NET2<br/> OBD_FAIL_OST_DISCONNECT_NET<br/> OBD_FAIL_OST_CREATE_NET<br/> OBD_FAIL_OST_DESTROY_NET<br/> OBD_FAIL_OST_GETATTR_NET<br/> OBD_FAIL_OST_SETATTR_NET<br/> OBD_FAIL_OST_ENOSPC<br/> OBD_FAIL_OST_PUNCH_NET<br/> OBD_FAIL_OST_STATFS_NET<br/> OBD_FAIL_OST_SYNC_NET<br/> OBD_FAIL_OST_QUOTACHECK_NET<br/> OBD_FAIL_OST_QUOTACTL_NET<br/> OBD_FAIL_LDLM_ENQUEUE_NET<br/> OBD_FAIL_LDLM_CONVERT_NET<br/> OBD_FAIL_LDLM_CANCEL_NET<br/> OBD_FAIL_MDS_ALL_REQUEST_NET<br/> OBD_FAIL_MGS_CONNECT_NET</p></blockquote> <p>Most of them are constructed from opcode name and stored on tgt_handler::th_fail_id. They are checked in tgt_handle_request0() properly. </p> <p>OBD_FAIL_OST_EROFS<br/> OBD_FAIL_OST_ENOSPC<br/> both are checked inside tgt_brw_write()</p> <p>OBD_FAIL_OST_CONNECT_NET2 is checked in tgt_request_handle().</p> <p>So for now I see the missed fail_ids are:<br/> OBD_FAIL_LDLM_REPLY<br/> OBD_FAIL_OST_LDLM_REPLY_NET<br/> OBD_FAIL_UPDATE_OBJ_NET_REP<br/> OBD_FAIL_MDS_FAIL_LOV_LOG_ADD</p> <p><a href="http://review.whamcloud.com/12232" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12232</a> - patch adds missing checks for the following FAIL ids:<br/> OBD_FAIL_LDLM_REPLY - fail id is set after tgt_enqueue() for non-OST requests.<br/> OBD_FAIL_OST_LDLM_REPLY_NET - the same but for OST enqueue only.<br/> OBD_FAIL_UPDATE_OBJ_NET_REP - it was not lost and functional but renamed, rename reference to it tests.<br/> OBD_FAIL_MDS_FAIL_LOV_LOG_ADD - added back in code along with workaround.<br/> OBD_FAIL_TGT_LAST_REPLAY it was never used and useless in fact, patch removes it along with test.</p> <p>Therefore only 3 FAIL IDs were really lost after all. I am going to inspect all fail ids in obd_support.h and find out which are not used anymore or miss checks too.</p> <p>Tappro, I'm very sorry that I didn't realise you already have a patch for OBD_FAIL_LDLM_REPLY, I submitted another one <a href="http://review.whamcloud.com/#/c/12780/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12780/</a><br/> could you take a look? If you prefer to choose yours, I'm OK to abandon mine, thanks.</p> <p>Patch was updated to include the latest master commits which fix replay-dual issues.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12232/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12232/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5604" title="Lots of FAIL_ID checking are lost" class="issue-link" data-issue-key="LU-5604"><del>LU-5604</del></a> tgt: return missed fail ids<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4de90170e2573321e7691364d1d527aedfd25ff9</p> <p>patch was landed, ticket can be closed</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16846" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5604" title="Lots of FAIL_ID checking are lost" class="issue-link" data-issue-key="LU-5604"><del>LU-5604</del></a> tests: fix usage of drop_ldlm_reply() in tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d5c616e494e2e46b348162db410dacc3046f2fb7</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/16846/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/16846/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5604" title="Lots of FAIL_ID checking are lost" class="issue-link" data-issue-key="LU-5604"><del>LU-5604</del></a> tests: fix usage of drop_ldlm_reply() in tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7130551d63c155534ab62cb4942b7c88556daad4</p> Blocker LU-5579 Duplicate LU-5709 Related LU-10816 Development Rank 1|hzwvxj: Rank (Obsolete) 15676 Severity