https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-868 Lustre <p>In current handle_recovery_req, when setting the timeout for waiting next replay</p> <p> to = lustre_msg_get_timeout(req-&gt;rq_reqmsg);<br/> extend_recovery_timer(class_exp2obd(req-&gt;rq_export), to);</p> <p>It did not consider the netlatency and early reply might extend the req timeout on client side, which might cause some replay tests failed like replay-dual 9, 10. </p> LU-868

Add early reply and netlatency into the timeout of waiting next replay.

Improvement Minor Resolved Fixed Di Wang Di Wang Sun, 20 Nov 2011 23:06:04 +0000 Fri, 3 Jun 2016 07:51:40 +0000 Fri, 3 Jun 2016 07:51:40 +0000 Lustre 2.2.0 0 0 <p>There exists a quite similar scenario at: <a href="https://maloo.whamcloud.com/test_sets/eb23c4ba-1510-11e1-b669-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eb23c4ba-1510-11e1-b669-52540025f9af</a></p> <p>The failure on patch set 2 is quite similar to <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a>.</p> <p>0. server started recovery window at: 1321951273.840132, and recovery timeout was set to 60s;<br/> 1. client replayed the open request at 1321951273.829810, with timeout value to 60 seconds;<br/> 2. server received this request at 1321951273.840592, and reply was dropped at 1321951273.845728;<br/> 3. server evicted this client at 1321951333.841817 because it couldn't finish recovery in time;<br/> 4. the replaying request was expired on the client side at 1321951334.831938;</p> <p>It looks like that it's not enough to extend the recovery window by rq_timeout only. We should add RECONNECT time as well so that it has enough time for the client to reconnect and replay again. Another question is why rq_timeout of replaying request was set to 60 seconds, I think it should be much shorter.</p> <p>I'd agree that reconnect time is good to have included but it should be also added to the initial recovery timeout then. As for 60s timeout, this can be due to network latency, I recall from discussion with Eric that it is normal to have delays up to 60s, e.g. with routers. In current situation the recovery time in 60s looks more unnatural than request timeout, it is just too short.</p> <p>The check_and_start_recovery_timer() looks pretty reasonable but at the end it has:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">1487 service_time -= obd-&gt;obd_recovery_timeout; 1488 <span class="code-keyword">if</span> (service_time &gt; 0) 1489 extend_recovery_timer(obd, service_time); </pre> </div></div> <p>I think that Maloo issue you are referring was before your fixes for extend_recovery_timer(), so I suspect that can cause not correct total recovery time making it smaller, what do you think?</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/client.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/client.c</li> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/client.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/ptlrpc/client.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/ptlrpc/client.c</li> <li>lustre/include/lustre/lustre_idl.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/422/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #422</a><br/> <a href="https://jira.whamcloud.com/browse/LU-868" title="Add early reply and netlatency into the timeout of waiting next replay. " class="issue-link" data-issue-key="LU-868"><del>LU-868</del></a> ptlrpc: Fix the timeout for waiting next replay (Revision 06f41b84901ad3bb4901b51a24e1553f9eeb6f1c)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=06f41b84901ad3bb4901b51a24e1553f9eeb6f1c" class="external-link" target="_blank" rel="nofollow noopener">06f41b84901ad3bb4901b51a24e1553f9eeb6f1c</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/include/lustre/lustre_idl.h</li> <li>lustre/ptlrpc/client.c</li> </ul> Development Rank 1|hzv7mv: Rank (Obsolete) 4771