https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8582 Lustre <p>This issue was created by maloo for Saurabh Tandan &lt;saurabh.tandan@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/a3f680b4-7276-11e6-8afd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a3f680b4-7276-11e6-8afd-5254006e85c2</a>.</p> <p>The sub-test test_255a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Client dmesg:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 7909.928568] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 19:40:57 (1472870457) [ 7916.983080] Lustre: 13272:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1472870457/real 1472870457] req@ffff880043d21b00 x1544415137983184/t0(0) o21-&gt;lustre-OST0003-osc-ffff880000b5c800@10.2.4.226@tcp:6/4 lens 504/432 e 0 to 1 dl 1472870464 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 7916.993670] Lustre: lustre-OST0003-osc-ffff880000b5c800: Connection to lustre-OST0003 (at 10.2.4.226@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7917.002143] Lustre: lustre-OST0003-osc-ffff880000b5c800: Connection restored to 10.2.4.226@tcp (at 10.2.4.226@tcp) [ 7917.004970] Lustre: Skipped 1 previous similar message [ 7938.019143] Lustre: lustre-OST0003-osc-ffff880000b5c800: Connection to lustre-OST0003 (at 10.2.4.226@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7938.030107] Lustre: Skipped 2 previous similar messages [ 7973.060145] Lustre: lustre-OST0003-osc-ffff880000b5c800: Connection to lustre-OST0003 (at 10.2.4.226@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7973.070547] Lustre: Skipped 10 previous similar messages [ 7987.075859] Lustre: lustre-OST0003-osc-ffff880000b5c800: Connection restored to 10.2.4.226@tcp (at 10.2.4.226@tcp) [ 7987.082573] Lustre: Skipped 15 previous similar messages [ 8040.574146] INFO: task lfs:24449 blocked for more than 120 seconds. [ 8040.579853] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8040.582026] lfs D ffff88003cb47b10 0 24449 24265 0x00000080 [ 8040.584167] ffff88005462fbe0 0000000000000082 ffff880036479700 ffff88005462ffd8 [ 8040.586392] ffff88005462ffd8 ffff88005462ffd8 ffff880036479700 ffff8800378a5790 [ 8040.588540] ffff8800378a5798 7fffffffffffffff ffff880036479700 ffff88003cb47b10 [ 8040.590683] Call Trace: [ 8040.592422] [&lt;ffffffff8163bc59&gt;] schedule+0x29/0x70 [ 8040.594318] [&lt;ffffffff81639949&gt;] schedule_timeout+0x209/0x2d0 [ 8040.596362] [&lt;ffffffffa0945c5c&gt;] ? ptlrpcd_add_req+0x1ec/0x2e0 [ptlrpc] [ 8040.598446] [&lt;ffffffffa0923c30&gt;] ? lustre_swab_ladvise_hdr+0x40/0x40 [ptlrpc] [ 8040.600571] [&lt;ffffffff8163c026&gt;] wait_for_completion+0x116/0x170 [ 8040.602607] [&lt;ffffffff810b8910&gt;] ? wake_up_state+0x20/0x20 [ 8040.604615] [&lt;ffffffffa0b4f3b0&gt;] ? lov_io_iter_fini_wrapper+0x50/0x50 [lov] [ 8040.606715] [&lt;ffffffffa0aeee68&gt;] osc_io_ladvise_end+0x38/0x50 [osc] [ 8040.608765] [&lt;ffffffffa075a95d&gt;] cl_io_end+0x5d/0x150 [obdclass] [ 8040.610725] [&lt;ffffffffa0b4f48b&gt;] lov_io_end_wrapper+0xdb/0xe0 [lov] [ 8040.612715] [&lt;ffffffffa0b4f9a6&gt;] lov_io_call.isra.9+0x86/0x140 [lov] [ 8040.614647] [&lt;ffffffffa0b4fa96&gt;] lov_io_end+0x36/0xb0 [lov] [ 8040.616502] [&lt;ffffffffa075a95d&gt;] cl_io_end+0x5d/0x150 [obdclass] [ 8040.618371] [&lt;ffffffffa075d103&gt;] cl_io_loop+0xb3/0x190 [obdclass] [ 8040.620279] [&lt;ffffffffa0bd1c29&gt;] ll_file_ioctl+0x3069/0x37f0 [lustre] [ 8040.622168] [&lt;ffffffff811f2585&gt;] do_vfs_ioctl+0x2e5/0x4c0 [ 8040.624012] [&lt;ffffffff811fd057&gt;] ? __alloc_fd+0xa7/0x130 [ 8040.625813] [&lt;ffffffff811fd167&gt;] ? __fd_install+0x47/0x60 [ 8040.627680] [&lt;ffffffff811f2801&gt;] SyS_ioctl+0xa1/0xc0 [ 8040.629464] [&lt;ffffffff81646c49&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>OST console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:43:40:[ 7912.118286] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 19:40:57 (1472870457) 19:43:40:[ 7912.161108] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0002: no handlers for opcode 0x15 19:43:40:[ 7912.166191] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) Skipped 6 previous similar messages 19:43:40:[ 7919.176071] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 7919.185035] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7926.185274] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 7926.189136] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7933.189463] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 7933.196824] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7940.211153] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7947.211321] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 7947.216921] Lustre: Skipped 1 previous similar message 19:43:40:[ 7954.222110] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7954.228583] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) Skipped 1 previous similar message 19:43:40:[ 7968.227179] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 7968.233536] Lustre: Skipped 8 previous similar messages 19:43:40:[ 7975.251445] LustreError: 23546:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 7975.257113] LustreError: 23546:0:(tgt_handler.c:593:tgt_handler_find_check()) Skipped 8 previous similar messages 19:43:40:[ 8000.268676] Lustre: lustre-OST0000: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 19:43:40:[ 8000.275601] Lustre: Skipped 5 previous similar messages 19:43:40:[ 8010.282726] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 19:43:40:[ 8010.287623] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) Skipped 10 previous similar messages 19:43:40:[ 8066.316121] Lustre: lustre-OST0003: Client 95455275-3e21-dec7-8ff1-6b8c07c6d061 (at 10.2.4.219@tcp) reconnecting 20:41:39:********** Timeout by autotest system ********** </pre> </div></div> <p>This issue was first seen on 08/16/2016</p> Interop: 2.8.0 EL7.2 Server/EL7.2 Client<br/> Server: b2_8, build# 12<br/> Client: master, build# 3431 LU-8582

Interop: master<->b2_8 - sanity test_255a: test failed to respond and timed out

Bug Minor Resolved Fixed Li Xi Maloo MON Tue, 6 Sep 2016 19:00:15 +0000 Fri, 14 Jul 2023 03:11:07 +0000 Mon, 18 Jul 2022 13:52:18 +0000 Lustre 2.9.0 Lustre 2.16.0 0 6 <p>There is a version check in this test that should avoid this interop issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ $(lustre_version_code ost1) -lt $(version_code 2.8.54) ] &amp;&amp; skip <span class="code-quote">"lustre &lt; 2.8.54 does not support ladvise "</span> &amp;&amp; <span class="code-keyword">return</span> </pre> </div></div> <p>If this problem is hit again, please investigate why the interop version check is not working.</p> <p>For now I'm going to close this issue, since I believe it is fixed, and there is a different failure being hit on sanity test_255a that is being annotated as this one.</p> <p>The version check is not at the start of the test, so some of the <tt>lfs ladvise</tt> code was still being run before the version was checked.</p> <p>Li Xi, could you please take a look at this issue. While it is possible to skip this test during interop testing (which I will push a patch for), the real problem is that the <tt>-ENOTSUPP</tt> error is not being returned to <tt>lfs ladvise</tt> and instead the client is hanging forever. The same would be true for regular users if they have a new client with <tt>lfs ladvise</tt> support connecting to an older server, so this isn't just a testing problem.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25362" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25362</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> tests: skip sanity test_255a for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a5fa2a6f2f2ee7138d9bdd3560fd73dd8c6c040d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25362/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25362/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> tests: skip sanity test_255a for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 35958f8c07beb7cff0763232c1aa16f0080ec38f</p> <p>Closing again</p> <p>This was not actually fixed, just the test was disabled.</p> <p>James wrote:</p> <blockquote> <p>For sanity test 255a, an ladvise test, we have</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_255a() { [ $(lustre_version_code ost1) -lt $(version_code 2.8.54) ] &amp;&amp; skip "lustre &lt; 2.8.54 does not support ladvise " &amp;&amp; return </pre> </div></div> <p>We need to add the same check to test 255b.</p></blockquote> <p>Hi Li Xi,<br/> could you please look into this problem. Running <tt>lfs ladvise</tt> with a new (2.9+) client causes it to hang when using an old (2.8-) server. It should just return an <tt>-EOPNOTSUPP</tt> error to the caller.</p> <p>A quick look at the code shows it is stuck in <tt>wait_for_completion()</tt> in:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> void osc_io_ladvise_end(<span class="code-keyword">const</span> struct lu_env *env, <span class="code-keyword">const</span> struct cl_io_slice *slice) { struct cl_io *io = slice-&gt;cis_io; struct osc_io *oio = cl2osc_io(env, slice); struct osc_async_cbargs *cbargs = &amp;oio-&gt;oi_cbarg; <span class="code-object">int</span> result = 0; struct cl_ladvise_io *lio = &amp;io-&gt;u.ci_ladvise; <span class="code-keyword">if</span> ((!(lio-&gt;li_flags &amp; LF_ASYNC)) &amp;&amp; cbargs-&gt;opc_rpc_sent) { wait_for_completion(&amp;cbargs-&gt;opc_sync); result = cbargs-&gt;opc_rc; } slice-&gt;cis_io-&gt;ci_result = result; } </pre> </div></div> <p>but my knowledge of CLIO isn't strong enough to know if it is waiting there indefinitely, or just repeating this wait for each failed RPC that it is sending? In any case, the OSS is returning <tt>-EOPNOTSUPP</tt> to the client, but the client is not handling the RPC error properly and returning it to the client.</p> <p>Also, it would seem to me that these handlers should be using <tt>wait_for_completion_interruptible()</tt> so that user processes can be killed if the server is not responsive. Ideally, there would be a <tt>wait_for_completion_io_interruptible()</tt> so that the time could be accounted as IO wait instead of just sleeping, but that doesn't seem to exist.</p> <p>This is going to become an issue with patch <a href="https://review.whamcloud.com/43170" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43170</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14380" title="Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)" class="issue-link" data-issue-key="LU-14380">LU-14380</a> statahead: divide hash space evenly among the ranks</tt>" adding a new "<tt>LU_LADVISE_STATAHEAD</tt>" method.</p> <p>The need for <tt>wait_for_completion_interruptible()</tt> in <tt>osc_io_ladvise_end()</tt> still exists also.</p> <p>The server of Lustre 2.8 does not have ladvise support, thus is not able to handle ladvise RPC, so it complains:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:43:40:[ 7954.222110] LustreError: 12896:0:(tgt_handler.c:593:tgt_handler_find_check()) lustre-OST0003: no handlers for opcode 0x15 </pre> </div></div> <p>opcode 0x15 is OST_LADVISE.</p> <p>It seems no error message about the wrong opcode RPC will be sent back to client, thus client hang there waiting forever. There is no mechanism of sending error reply back if RPC has a wrong opcode? If so, adding new opcode in the protocol might hit similar problem.</p> <p>I don't think this will cause any problem of new ladvise types. <a href="https://jira.whamcloud.com/browse/LU-14380" title="Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)" class="issue-link" data-issue-key="LU-14380">LU-14380</a> should work fine when testing interop with old servers (not &lt; 2.9). But it worth to confirm that.</p> <p>I will push a patch to use wait_for_completion_interruptible() in osc_io_ladvise_end().</p> <p>Hmm, seems like changing to wait_for_completion_interruptible() does not fix the problem. It just enables users to interrupt the problem.</p> <p>Client was not able to get reply, so it stuck there and got disconnected.</p> <p>In the case of new client connects to server, what would be the mechanisms to check whether server supports a feature? The client could add another OBD_CONNECT_ e.g. OBD_CONNECT2_LADVISE to check whether the feature is supported on server. But is there any way for the server to sent a error reply to a PRC with unknown opcode?</p> <p>We had a similar issue with fallocate - the bad opcode caused clients to retry forever. It would be better to fix the server to return -EOPNOTSUPP for bad opcodes (which will be returned to userspace), but also to allow clients to handle -EPROTO and abort instead of retry. </p> <p>You are correct that wait_event_idle_interruptible() is only to allow users to interrupt the process (which should preferably be used in places that user applications are waiting), but that will only help the <b>next</b> problem that is hit in this code. The above two changes should prevent old/new client/server from getting stuck at all. </p> <p>A fix patch is comming. If we apply the following debug change:</p> <p>osc_ladvise_base()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> - rc = ptlrpc_request_pack(req, LUSTRE_OST_VERSION, OST_LADVISE); + rc = ptlrpc_request_pack(req, LUSTRE_OST_VERSION, OST_QUOTACHECK); </pre> </div></div> <p>Without the fix patch:<br/> The ladise command will stuck there forever</p> <p>With the fix patch, command quit immediately:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs ladvise -a willread a lfs ladvise: cannot give advice: Operation not supported (95) ladvise: cannot give advice 'willread' to file 'a': Operation not supported </pre> </div></div> <p>"Li Xi &lt;lixi@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47761" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47761</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> target: send error reply on wrong opcode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b089334b46d19b5efe4e2197ffdca9f3dbc70916</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47761/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47761/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> target: send error reply on wrong opcode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 566edb8c43e65ed4cb99614c7c753965bb201366</p> <p>Landed for 2.16</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51568" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51568</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> tests: skip sanity/905 for old OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2ced1e0898aacd741c95c25d44350dfefa953853</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51568/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51568/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8582" title="Interop: master&lt;-&gt;b2_8 - sanity test_255a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8582"><del>LU-8582</del></a> tests: skip sanity/905 for old OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 839b43515004828dec212ed0183fe51929c91b5b</p> Duplicate Related LU-14139 LU-14380 LU-8811 LU-9097 Development Rank 1|hzyniv: Rank (Obsolete) 9223372036854775807 Severity