https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11128 Lustre <p>This issue was created by maloo for bobijam &lt;bobijam@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5c95f0b2-8186-11e8-b441-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5c95f0b2-8186-11e8-b441-52540065bddc</a></p> <p>test_115 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 216 mins, last suite running was replay-single, restarting cluster to continue tests </pre> </div></div> <p>MDS dmesg keeps showing following error messages during several tests, and the test takes too much time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2545.541360] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.571570] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.210@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.618732] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.212@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.618926] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.212@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.619112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.212@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. ... </pre> </div></div> <p>another hit also happens at <a href="https://testing.whamcloud.com/test_sets/08372d04-8188-11e8-97ff-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/08372d04-8188-11e8-97ff-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_80c 'Timeout occurred after 159 mins, last suite running was replay-single, restarting cluster to continue tests' </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-single test_115 - Timeout occurred after 216 mins, last suite running was replay-single, restarting cluster to continue tests<br/>  </p> LU-11128

replay-single test timeout

Bug Minor Resolved Fixed Alex Zhuravlev Maloo Sun, 8 Jul 2018 13:33:12 +0000 Tue, 28 May 2019 17:29:53 +0000 Tue, 2 Oct 2018 21:55:51 +0000 Lustre 2.12.0 Lustre 2.12.0 0 5 <p>There are a significant number of test timeouts on replay-single (60 in the past 40 weeks), but they are spread across a large number of different subtests, so may be harder to notice. Doing a subtest query for <tt>TIMEOUT</tt> failures, but leaving the subtest field blank will show this.</p> <p>I'm working on this as it seem to affect many patches I've been working on and more importantly it seem to be related to <a href="https://jira.whamcloud.com/browse/LU-7236" title="connections on demand" class="issue-link" data-issue-key="LU-7236"><del>LU-7236</del></a> (idle connections can disconnect).</p> <p><a href="https://review.whamcloud.com/#/c/32980/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32980/</a> is a patch to reproduce/debug the issue. so far it looks like late DISCONNECT reply resets just reinitiated connection and it gets stuck.</p> <p> </p> <p><a href="https://review.whamcloud.com/#/c/32980/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32980</a> has passed many replay-single runs, I think it's ready for inspection and regular testing.</p> <p> </p> <p>One question I had about this failure - how short/long is the idle connection timeout? It seems like we shouldn't be getting so many timeouts in the middle of actively running tests. Is there some correlation between the tests failing with this issue and the length of time the client is idle?</p> <p>When you think about it, we don't want the connections to be dropping after only a few seconds of idle time, or we may get big reconnection storms if the system is still mostly in use, which will also hurt performance because of dropped grant and such. </p> <p>20s by default</p> <p>I wonder if 20s is too short by default? Especially in the case of large systems where there may be thousands of clients that have nearly identical behaviour (e.g. active/idle at the same time, though possibly to different OSTs).</p> <p>On the one hand, the 20s timeout definitely good for finding issues with this code during testing, but I think the default should be longer (e.g. 60s or 300s) depending on how long it takes for a large number of clients to reconnect. We could still set a shorter time in the test-framework to ensure the code continues to be tested. For testing purposes, it might also make sense to have an option (e.g. "<tt>lctl set_param osc.&#42;.idle_timeout=debug</tt>" and "<tt>...=nodebug</tt>" or similar) to print a message to the console when the client disconnects (e.g. "<tt>testfs-OST0004: disconnect after 50s idle</tt>" and "<tt>testfs-OST0004: reconnect after 650s idle</tt>" or similar) so that we can help debug problems related to this feature. The console message should be enabled during testing.</p> <p>I see in the code that <tt>idle_timeout</tt> has a maximum value of <tt>CONNECT_SWITCH_MAX = 50s</tt>, which seems a bit short to me? Is that because the <tt>OBD_PING</tt> RPCs will keep the connection alive if it is longer than this? What happens if <tt>ping_interval</tt> (default 25s) is shorter than <tt>idle_timeout</tt>? Is that why the default idle_timeout is 20s?</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33168" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33168</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11128" title="replay-single test timeout" class="issue-link" data-issue-key="LU-11128"><del>LU-11128</del></a> ptlrpc: add debugging for idle connections<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 216c9c7cbcd38fa56ee2240c8a13066ad66b3f77</p> <p>Andreas, I'm fine to change the defaults and yes, one of the reason to rather have it short is to hit the code more frequently.</p> <p>ping reply is not counted:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lustre_msg_get_opc(req-&gt;rq_reqmsg) != OBD_PING) req-&gt;rq_import-&gt;imp_last_reply_time = ktime_get_real_seconds();</pre> </div></div> <p>then check for idle:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (now - imp-&gt;imp_last_reply_time &lt; imp-&gt;imp_idle_timeout) <span class="code-keyword">return</span> <span class="code-keyword">false</span>;</pre> </div></div> <p>In that case, can you submit a patch to increase the default idle_timeout value, and set it lower in the test config for testing.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33168/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33168/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11128" title="replay-single test timeout" class="issue-link" data-issue-key="LU-11128"><del>LU-11128</del></a> ptlrpc: add debugging for idle connections<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0aa58d26f5df2b71a040ed6f0f419b925528b6ad</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32980/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32980/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11128" title="replay-single test timeout" class="issue-link" data-issue-key="LU-11128"><del>LU-11128</del></a> ptlrpc: new request vs disconnect race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 93d20d171c20491a96e5e85d7442a002f300619d</p> <p>Landed for 2.12</p> Duplicate LU-11126 LU-11183 Related LU-7236 LU-11362 LU-11269 LU-11405 Development Rank 1|hzzyvr: Rank (Obsolete) 9223372036854775807 Severity