https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9066 Lustre <p>During the OST failover, it was reported:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2016-12-05 02:42:07 [115035.688184] LustreError: 138-a: fs1-OST0001: A client on nid 172.18.1.103@o2ib was evicted due to a lock completion callback time out: rc -19 </pre> </div></div> <p>On the client side, IOR got I/O failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bluepill-client03: IOR-3.0.1: MPI Coordinated Test of Parallel I/O bluepill-client03: bluepill-client03: Began: Sun Dec 4 18:28:38 2016 bluepill-client03: Command line used: /usr/local/bin/IOR -o /mnt/fs1//ha.sh-111769/bluepill-client03-ior/f.ior -f /test-tools/grev/Cray/2016_snx2k_fvt.ior.shared_file.p ... bluepill-client03: clients = 88 (8 per node) ... bluepill-client03: Commencing read performance test: Mon Dec 5 02:34:58 2016 bluepill-client03: ior ERROR: read() failed, errno 5, Input/output error (aiori-POSIX.c:250) </pre> </div></div> LU-9066

ior ERROR: read() failed, Input/output error; client was evicted after OST failover

Bug Minor Closed Fixed nasf nasf Tue, 31 Jan 2017 13:45:31 +0000 Wed, 14 Jun 2017 11:11:15 +0000 Thu, 23 Mar 2017 01:46:06 +0000 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.10.0 0 4 <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25173" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25173</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9066" title="ior ERROR: read() failed, Input/output error; client was evicted after OST failover" class="issue-link" data-issue-key="LU-9066"><del>LU-9066</del></a> ldlm: NOT evict client when target stopping<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1495b69d8bf5d901c372ec91fa316e9649b31866</p> <p>It seems this is the same as <a href="https://jira.whamcloud.com/browse/LU-8359" title="Wrong evict during failover" class="issue-link" data-issue-key="LU-8359">LU-8359</a>, isn't it? </p> <p>The issue will be handled in <a href="https://jira.whamcloud.com/browse/LU-8860" title="lock callback errors after client umount" class="issue-link" data-issue-key="LU-8860"><del>LU-8860</del></a>.</p> <blockquote> <p>It seems this is the same as <a href="https://jira.whamcloud.com/browse/LU-8359" title="Wrong evict during failover" class="issue-link" data-issue-key="LU-8359">LU-8359</a>, isn't it? </p></blockquote> <p>I think your patch <a href="https://review.whamcloud.com/#/c/23921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/23921</a> has already handled this case.</p> <p>reopen to keep patch tracking under this ticket</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23921/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23921/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9066" title="ior ERROR: read() failed, Input/output error; client was evicted after OST failover" class="issue-link" data-issue-key="LU-9066"><del>LU-9066</del></a> ldlm: don't evict client on umount if AST fails<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eda984e7cb4e6a97310ed0f5e81f398dc48b56bf</p> <p>We observe same error in 2.7 with <a href="https://review.whamcloud.com/23921/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23921/</a> applied</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-06-09 14:53:58 [77116.822290] LustreError: 138-a: fs1-OST0000: A client on nid 172.18.1.104@o2ib was evicted due to a lock blocking callback time out: rc -19 </pre> </div></div> <p>with patch from <a href="https://jira.whamcloud.com/browse/LU-8359" title="Wrong evict during failover" class="issue-link" data-issue-key="LU-8359">LU-8359</a> it isn't reproducible.</p> <p>Sorry, I cannot imagine how this message can be printed with the <a href="https://review.whamcloud.com/23921/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23921/</a> applied. The logic of ldlm_handle_ast_error() with such patch is as following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int ldlm_handle_ast_error(struct ldlm_lock *lock, struct ptlrpc_request *req, int rc, const char *ast_type) { ... } else if (rc == -ENODEV || rc == -ESHUTDOWN || (rc == -EIO &amp;&amp; req-&gt;rq_import-&gt;imp_state == LUSTRE_IMP_CLOSED)) { /* Upon umount process the AST fails because cannot be * sent. This shouldn't lead to the client eviction. * -ENODEV error is returned by ptl_send_rpc() for * new request in such import. * -SHUTDOWN is returned by ptlrpc_import_delay_req() * if imp_invalid is set or obd_no_recov. * Meanwhile there is also check for LUSTRE_IMP_CLOSED * in ptlrpc_import_delay_req() as well with -EIO code. * In all such cases errors are ignored. */ LDLM_DEBUG(lock, "%s AST can't be sent due to a server" " %s failure or umount process: rc = %d\n", ast_type, req-&gt;rq_import-&gt;imp_obd-&gt;obd_name, rc); } else { LDLM_ERROR(lock, "client (nid %s) %s %s AST (req@%p x%llu status %d rc %d), evict it", libcfs_nid2str(peer.nid), req-&gt;rq_replied ? "returned error from" : "failed to reply to", ast_type, req, req-&gt;rq_xid, (req-&gt;rq_repmsg != NULL) ? lustre_msg_get_status(req-&gt;rq_repmsg) : 0, rc); ldlm_failed_ast(lock, rc, ast_type); } return rc; ... } </pre> </div></div> <p>Please note that when the "rc == -19 (ENODEV)", if will goto the branch LDLM_DEBUG(), but not the branch ldlm_failed_ast(). Please correct me if I missed anything.</p> <p>Ah, in fact the fix wasn't applied during our testing. Sorry for the confusion.</p> Duplicate LU-8860 Related LU-8359 LU-8860 Development Rank 1|hzz253: Rank (Obsolete) 9223372036854775807 Severity