Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-4181] lnet-selftest test_smoke: lst error found https://jira.whamcloud.com/browse/LU-4181 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/794b62ea-40b4-11e3-af95-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/794b62ea-40b4-11e3-af95-52540035b04c</a>.</p> <p>The sub-test test_smoke failed with the following error:</p> <blockquote> <p>Total 1 error nodes in c<br/> Total 3 error nodes in s<br/> lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found </p></blockquote> <p>Info required for matching: lnet-selftest smoke</p> LU-4181 lnet-selftest test_smoke: lst error found Bug Blocker Resolved Fixed Isaac Huang Maloo Tue, 29 Oct 2013 20:41:08 +0000 Thu, 13 Oct 2016 18:45:18 +0000 Tue, 5 Jan 2016 07:52:05 +0000 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.7.0 Lustre 2.4.3 Lustre 2.8.0 Lustre 2.7.0 0 15 <p>Amir</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Just disable lnet_selftest now for DNE <a href="http://review.whamcloud.com/8130" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8130</a></p> <p>It looks like lnet-selftest has only been failing since 2013-10-25, but only for review-dne and review-dne-zfs tests.</p> <p>In <a href="https://maloo.whamcloud.com/test_sets/794b62ea-40b4-11e3-af95-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/794b62ea-40b4-11e3-af95-52540035b04c</a> it looks like there is some kind of communication problem between the DNE nodes, possibly because there are more of nodes? It seems some of the nodes are having problems with .218, except for the one that reports problems with .215.</p> <p>MDS 1 (wtm-21vm3):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:51:19:LNet: 12592:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.218@tcp, timeout 64. 02:51:19:LNet: 12592:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.218@tcp, timeout 64. 02:51:19:LustreError: 12591:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 02:51:19:LustreError: 12591:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 </pre> </div></div> <p>Client 2 (wtm-21vm6):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:24:09:LNet: 18251:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.215@tcp, timeout 64. 02:24:09:LNet: 18251:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.215@tcp, timeout 64. 02:24:09:LustreError: 18249:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.215@tcp failed with -110 02:24:09:LustreError: 18249:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.215@tcp failed with -110 02:24:09:LustreError: 18249:0:(ping_test.c:135:ping_client_done_rpc()) Unable to ping 12345-10.10.16.220@tcp (21): -110 02:24:09:LustreError: 18249:0:(ping_test.c:135:ping_client_done_rpc()) Unable to ping 12345-10.10.16.220@tcp (22): -110 02:51:29:Lustre: 18248:0:(ping_test.c:82:ping_client_fini()) 16 pings have failed. 02:51:29:Lustre: DEBUG MARKER: /usr/sbin/lctl mark lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found </pre> </div></div> <p>OST 9-16 (wtm-21vm8):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:51:15:LNet: 581:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.218@tcp, timeout 64. 02:51:15:LustreError: 580:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 02:51:15:LustreError: 580:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 02:51:15:Lustre: 579:0:(ping_test.c:82:ping_client_fini()) 8 pings have failed. 02:51:15:Lustre: 579:0:(ping_test.c:82:ping_client_fini()) 8 pings have failed. </pre> </div></div> <p>MDS 4 (wtm-21vm1):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:51:15:LNet: 28972:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.218@tcp, timeout 64. 02:51:15:LNet: 28972:0:(rpc.c:1119:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.10.16.218@tcp, timeout 64. 02:51:15:LustreError: 28971:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 02:51:15:LustreError: 28971:0:(brw_test.c:333:brw_client_done_rpc()) BRW RPC to 12345-10.10.16.218@tcp failed with -110 02:51:15:Lustre: 28970:0:(ping_test.c:82:ping_client_fini()) 5 pings have failed. 02:51:15:Lustre: 28970:0:(ping_test.c:82:ping_client_fini()) 5 pings have failed. </pre> </div></div> <p>Amir, any chance to look at this failure?</p> <p>Is there a way to rerun the test and add some debug statements in the test. the test gets the list of nids of servers and clients, I want to make sure that this list is sane and what is expected. Could their be some assumptions based on the order of the nids, that is no longer true for DNE systems?</p> <p>Amir, the best bet would be to submit a patch to lnet-selftest.sh that adds the debugging you want and the run the test with:</p> <p> Test-Parameters: fortestonly testlist=lnet-selftest mdscount=4 osscount=2</p> <p>Or similar (please check wiki page to verify exact parameter names).</p> <p>The patch to disable lnet-selftest for DNE shows as landed on 11-08, but I still see four failures in review-dne in the past few days. Is that because those patches have not been rebased, or is the test not being skipped for some reason?</p> <p>maloo says <a href="https://maloo.whamcloud.com/test_sets/2d3946f0-539d-11e3-9901-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2d3946f0-539d-11e3-9901-52540035b04c</a> is this bug, but it was seen in review not in review-dne</p> <p>Discussed this issue with Isaac, and we're suspecting that it could be a network issue. In the original report there is many ping failures, which would indicate that one of the clients is unable to communicate with the server. Is there a way to hop onto the test system that's having the problem and try and do lctl ping from the client to the server. <br/> Beside that, I'm making a few changes to lnet-selftest to dump out more debug information, so I can tell if one of the clients is simply failing to communicate with the servers.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/69/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/69/</a> (2.4.2 RC1)<br/> MDSCOUNT=4</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/6af5f312-6879-11e3-a9a3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6af5f312-6879-11e3-a9a3-52540035b04c</a></p> <p>Since Chris changed the review-dne node configuration to 2x MDS + 1x OSS on 2013-12-26 as part of TEI-1312, it seems this is allowing lnet-selftest can pass again (on b2_5 where it is not excluded):</p> <p><a href="https://maloo.whamcloud.com/test_sets/b040595e-7269-11e3-ab15-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b040595e-7269-11e3-ab15-52540035b04c</a></p> <p>It is probably worthwhile to submit a patch to re-enable this test to see if it is now working for review-dne runs on master.</p> <p>I just submit a patch to re-enable it: <a href="http://review.whamcloud.com/#/c/8823/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8823/</a></p> <p>the following result shows that this test can pass with current DNE config</p> <p><a href="https://maloo.whamcloud.com/test_sessions/4753c032-7dab-11e3-91f7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/4753c032-7dab-11e3-91f7-52540035b04c</a></p> <p>So it seems the problem is due to the number of nodes in the old review-dne config. We have since reduced the node count in the review-dne config to just one more than review-ldiskfs. </p> <p>However, since review-dne already takes much more time to run than review-ldiskfs (14h vs. 3h) it doesn't make sense to enable this on review-dne at this time, since the testing is equivalent.</p> <p>Hit this error in zfs testing</p> <p><a href="https://maloo.whamcloud.com/test_sets/9235abcc-9f18-11e3-934b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9235abcc-9f18-11e3-934b-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/73/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/73/</a> (2.4.3 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> MDSCOUNT=4</p> <p><a href="https://maloo.whamcloud.com/test_sets/00d71476-abee-11e3-bcad-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/00d71476-abee-11e3-bcad-52540035b04c</a></p> <p>another, in b2_5<br/> <a href="https://maloo.whamcloud.com/test_sets/0cf3118c-be8c-11e3-b5bd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0cf3118c-be8c-11e3-b5bd-52540035b04c</a></p> <p>Digging through some of the dmesg log files I see the error messages below. Some of these errors are repeated all over the the log. Probably will need someone more familiar with the lustre codebase to take a look and determine the cause of these errors.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 11-0: lustre-MDT0000-mdc-ffff880037c15400: Communicating with 10.2.4.158@tcp, operation mds_reint failed with -19. LustreError: 17968:0:(file.c:171:ll_close_inode_openhandle()) inode 144115205272520296 mdc close failed: rc = -4 LustreError: 166-1: MGC10.2.4.158@tcp: Connection to MGS (at 10.2.4.158@tcp) was lost; in progress operations using <span class="code-keyword">this</span> service will fail LustreError: 21700:0:(lmv_obd.c:1524:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff880037c15400), error -5 LustreError: 21700:0:(llite_lib.c:1610:ll_statfs_internal()) md_statfs fails: rc = -5 LustreError: 167-0: lustre-OST0000-osc-ffff880037c15400: This client was evicted by lustre-OST0000; in progress operations using <span class="code-keyword">this</span> service will fail. LustreError: 11-0: MGC10.2.4.158@tcp: Communicating with 10.2.4.158@tcp, operation obd_ping failed with -107. LustreError: 26751:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 sleeping <span class="code-keyword">for</span> 20000ms LustreError: 26751:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 awake LustreError: 11-0: lustre-OST0000-osc-ffff8800701a3400: Communicating with 10.2.4.159@tcp, operation ost_connect failed with -16. LustreError: 167-0: lustre-OST0000-osc-ffff8800701a3400: This client was evicted by lustre-OST0000; in progress operations using <span class="code-keyword">this</span> service will fail. LustreError: 29724:0:(osc_lock.c:833:osc_ldlm_completion_ast()) lock@ffff88007ac36ed0[3 3 0 1 1 00000000] W(2):[0, 18446744073709551615]@[0x100000000:0xe22:0x0] { LustreError: 29724:0:(osc_lock.c:833:osc_ldlm_completion_ast()) lovsub@ffff88006cec7a20: [0 ffff88007bd6a470 W(2):[0, 18446744073709551615]@[0x200002b11:0x14:0x0]] LustreError: 29724:0:(osc_lock.c:833:osc_ldlm_completion_ast()) osc@ffff88006efaca28: ffff88007cf06d40 0x20040000001 0x27bfcba3c3cbb9cb 3 ffff8800698c9dd8 size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0 LustreError: 29724:0:(osc_lock.c:833:osc_ldlm_completion_ast()) } lock@ffff88007ac36ed0 LustreError: 29724:0:(osc_lock.c:833:osc_ldlm_completion_ast()) dlmlock returned -5 LustreError: 29724:0:(ldlm_resource.c:809:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800701a3400: namespace resource [0xe22:0x0:0x0].0 (ffff88006af141c0) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 29724:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0xe22:0x0:0x0].0 (ffff88006af141c0) refcount = 2 LustreError: 29724:0:(ldlm_resource.c:1451:ldlm_resource_dump()) Granted locks (in reverse order): LustreError: 29724:0:(ldlm_resource.c:1454:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff8800701a3400 lock: ffff88007cf06d40/0x27bfcba3c3cbb9cb lrc: 4/0,1 mode: PW/PW res: [0xe22:0x0:0x0].0 rrc: 2 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;4095) flags: 0x126400000000 nid: local remote: 0xbb105070585d6d1c expref: -99 pid: 29703 timeout: 0 lvb_type: 1 LustreError: 29703:0:(cl_lock.c:1422:cl_unuse_try()) result = -5, <span class="code-keyword">this</span> is unlikely! Lustre: lustre-OST0000-osc-ffff8800701a3400: Connection restored to lustre-OST0000 (at 10.2.4.159@tcp) LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) lock@ffff88007ac36df8[2 0 0 1 0 00000000] W(2):[0, 18446744073709551615]@[0x200002b11:0x14:0x0] { LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) vvp@ffff880037c86650: LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) lov@ffff88007bd6a470: 1 LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) 0 0: --- LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) } lock@ffff88007ac36df8 LustreError: 29703:0:(cl_lock.c:1437:cl_unuse_locked()) unuse <span class="code-keyword">return</span> -5 LustreError: 9132:0:(mdc_locks.c:918:mdc_enqueue()) ldlm_cli_enqueue: -2 </pre> </div></div> <p>I thought when we test lnet_selftest, we are only loading the modules lnet and lnet_selftest. Those errors are from Lustre. What is it doing running during an lnet_selftest?</p> <p>Continuing to look at the LustreErrors in the dmesg on the client on the latest failures. Currently looking at: <br/> <a href="https://maloo.whamcloud.com/test_sessions/2bf8aa7e-4ad7-11e4-8d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/2bf8aa7e-4ad7-11e4-8d48-5254006e85c2</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 11-0: lustre-MDT0000-mdc-ffff88007c2b8800: Communicating with 10.2.4.131@tcp, operation obd_ping failed with -107. ## 107 ENOTCONN LustreError: 166-1: MGC10.2.4.131@tcp: Connection to MGS (at 10.2.4.131@tcp) was lost; in progress operations using <span class="code-keyword">this</span> service will fail LustreError: 2373:0:(client.c:2804:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007bb64c00 x1480913582033212/t4294967470(4294967470) o101-&gt;lustre-MDT0000-mdc-ffff88007c2b8800@10.2.4.131@tcp:12/10 lens 632/544 e 0 to 0 dl 1412309449 ref 2 fl Interpret:RP/4/0 rc 301/301 LustreError: 5833:0:(mgc_request.c:517:do_requeue()) failed processing log: -5 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007c2b8800: Communicating with 10.2.4.131@tcp, operation ldlm_enqueue failed with -95. LustreError: 22804:0:(fail.c:132:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 sleeping <span class="code-keyword">for</span> 1000ms LustreError: 22804:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 awake LustreError: 29859:0:(osc_request.c:1983:osc_build_rpc()) prep_req failed: -12 LustreError: 133-1: lustre-OST0004-osc-ffff88007c2b8800: BAD READ CHECKSUM: from 10.2.4.132@tcp inode [0x0:0x0:0x0] object 0x0:1925 extent [0-1048575] </pre> </div></div> <p>These errors are repeated through out the log.</p> <p>One observation, is that all the errors are on the tcp network not IB. I'm not familiar with the test configuration, so that could simply be because the test runs only on tcp network.</p> <p>On the server side I see the following LustreErrors:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 11-0: lustre-MDT0000-lwp-OST0000: Communicating with 10.2.4.95@tcp, operation mds_connect failed with -16. LustreError: 11-0: MGC10.2.4.95@tcp: Communicating with 10.2.4.95@tcp, operation obd_ping failed with -107. LustreError: 137-5: lustre-OST0006_UUID: not available <span class="code-keyword">for</span> connect from 10.2.4.98@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 31991:0:(ldlm_lib.c:2106:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery LustreError: 168-f: BAD WRITE CHECKSUM: lustre-OST0000 from 12345-10.2.4.98@tcp inode [0x200000bd0:0xe626:0x0] object 0x0:1899 extent [0-1048575]: client csum f1e7c95b, server csum f1e7c95a </pre> </div></div> <p>We created a patch, where lnet-selftest.sh unmounts the file system (which it already does) and and unload the lustre modules (new), then ran lnet-selftest.sh on it 22 times. And it passed. I would like to commit that change to see if resolves the errors that are occuring.</p> <p><a href="http://review.whamcloud.com/#/c/12214/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12214/</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/f9be7532-4f0b-11e4-92ea-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f9be7532-4f0b-11e4-92ea-5254006e85c2</a></p> <p>looked at the test failure from Jian Yu on 4/Oct/14 and it looks like there are many 108 (ESHUTDOWN) errors on one of the nodes, which could indicate that the lnet/self-test modules are being unloaded before the test is completed. This would result in the lst failure.</p> <p>Does the test system attempt to unload modules after the test? Could explain the errors. Ad mentioned before, when I an lnetselftest on its own 20 consecutive times, everything works.</p> <p>Please see <a href="http://review.whamcloud.com/12332" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12332</a>.</p> <p><a href="http://review.whamcloud.com/#/c/12469/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12469/</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/034d0816-60b4-11e4-a66b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/034d0816-60b4-11e4-a66b-5254006e85c2</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/94b38298-61a0-11e4-8b49-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/94b38298-61a0-11e4-8b49-5254006e85c2</a></p> <p>Patch landed to Master.</p> <p>The patch landed was a debug patch.</p> <p>I saw an instance of this today with the new debug patch.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>https://testing.hpdd.intel.com/test_sets/8f6d3750-6a53-11e4-a7b0-5254006e85c2 https://testing.hpdd.intel.com/test_logs/9b2922d4-6a53-11e4-a7b0-5254006e85c2 00000001:02000400:1.0:1415666453.647581:0:10468:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: -----============= acceptance-small: lnet-selftest ============----- Tue Nov 11 00:40:53 UTC 2014 00000001:02000400:1.0:1415666454.125144:0:10613:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark == lnet-selftest test complete, duration -o sec == 00:40:53 \(1415666453\) 00000001:02000400:0.0:1415666454.269086:0:10672:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: == lnet-selftest test complete, duration -o sec == 00:40:53 (1415666453) 00000001:02000400:0.0:1415666454.844848:0:10856:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' 00000001:02000400:0.0:1415666455.483429:0:11174:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 00000001:02000400:1.0:1415666455.618475:0:11232:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: excepting tests: 00000001:02000400:0.0:1415666458.017200:0:11346:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark == lnet-selftest test smoke: lst regression test == 00:40:57 \(1415666457\) 00000001:02000400:0.0:1415666458.154280:0:11404:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: == lnet-selftest test smoke: lst regression test == 00:40:57 (1415666457) 00000001:02000400:0.0:1415666458.424825:0:11506:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u 00000001:02000400:1.0:1415666459.001668:0:11759:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u 00000400:00020000:1.0:1415666764.476159:0:414:0:(rpc.c:1470:srpc_lnet_ev_handler()) ev-&gt;status = -125, ev-&gt;type = 5, errors = 1, rpcs_sent = 17427, rpcs_rcvd = 16841, rpcs_dropped = 0, rpcs_expired = 0 00000001:02000400:1.0:1415666768.744246:0:12074:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found 00000001:02000400:1.0:1415666768.878468:0:12132:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found 00000001:02000400:1.0:1415666769.107358:0:12233:0:(debug.c:345:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl dk &gt; /logdir/test_logs/2014-11-10/lustre-reviews-el6-x86_64--review-ldiskfs--2_4_1__28281__-70078148216900-145241/lnet-selftest.test_smoke.debug_log.$(hostname -\ s).1415666768.log; ev-&gt;type == 5 == LNET_EVENT_SEND t:lustre-release# errno 125 #define ECANCELED 125 /* Operation Canceled */ </pre> </div></div> <p>Since the status is -ECANCELED and is reported about 300 seconds after the start of test smoke this supports the idea that the error is due to the test being shutdown.</p> <p>More instance on master: <a href="https://testing.hpdd.intel.com/test_sets/c9b21b00-70bc-11e4-95d2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c9b21b00-70bc-11e4-95d2-5254006e85c2</a></p> <p>I looked at the other instances of these issues over the past few days and it appears that the failure is exactly the same across the different tests:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ev-&gt;status = -125 and ev-&gt;type = 5 </pre> </div></div> <p>Also the failure rate hasn't really changed after the patch landed. Approximately, 1 failure/day</p> <p>ECANCELED is set when the operation is canceled either due to a test being shutdown or selftest module being unloaded. I don't think there is any issues with selftest, as far as I could tell. </p> <p>Another possibility is that it could be a test system issue.</p> <p>All occurrences of the debug message added is towards the end of the log file. Could it be that the test system is shutting down the test after it thinks it is complete, but the traffic hasn't been stopped yet, causing this error to occur?</p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8f027262-76ce-11e4-b1ab-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8f027262-76ce-11e4-b1ab-5254006e85c2</a></p> <blockquote><p>Could it be that the test system is shutting down the test after it thinks it is complete, but the traffic hasn't been stopped yet, causing this error to occur?</p></blockquote> <p>In lnet-selftest.sh test_smoke():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> run_lst $runlst | tee $log rc=${PIPESTATUS[0]} [ $rc = 0 ] || error <span class="code-quote">"$runlst failed: $rc"</span> lst_end_session --verbose | tee -a $log </pre> </div></div> <p>We can add some codes before running lst_end_session to make sure the sessions are really finished. And it looks like the changes need to be made in the following codes in test_smoke_sub():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> echo $LST run b echo sleep 1 echo <span class="code-quote">"$LST stat --delay 10 --timeout 10 c s &amp;"</span> echo <span class="code-quote">'pid=$!'</span> echo <span class="code-quote">'trap <span class="code-quote">"cleanup $pid"</span> INT TERM'</span> echo sleep $smoke_DURATION echo <span class="code-quote">'cleanup $pid'</span> </pre> </div></div> <p>Isaac,<br/> Can you please take this one?<br/> Thank you</p> <p>I believe the root cause was that there's no way for a script to know whether tests are still running when end_session is called. In other words a script can't figure out when to end_session. The lst was mainly designed for interactive use - when the r/w stats dwindle down to near zero, the tests have completed and it's time for further actions (e.g. running more tests, or end_session and stop).</p> <p>The fix here is not to count it as an error, if lst tests are stopped from administrative actions (e.g. end_session from the remote test console, or local root user trying to unload the lnet_selftest kernel module). Such aren't network errors, and shouldn't be counted anyway.</p> <p>Increasing smoke_DURATION would reduce the chances of hitting such bogus errors but not eliminate them. It also comes at the cost of increasing test run time unnecessarily. So I wouldn't recommend it.</p> <p>Since test_smoke is just to verify that lnet selftest is working, it would probably be better to reduce the amount of work that is being done, instead of increasing the test duration further (it is already taking 2000s to complete).</p> <p>It might be possible to make the test size conditional on whether it is running in a VM or not. </p> <p>Hit this in tag-2.6.92 zfs testing:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/d45d80fc-9363-11e4-b7aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d45d80fc-9363-11e4-b7aa-5254006e85c2</a></p> <p>Isaac Huang (he.huang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13279" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13279</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4181" title="lnet-selftest test_smoke: lst error found" class="issue-link" data-issue-key="LU-4181"><del>LU-4181</del></a> lnet_selftest: bogus lst errors<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 827440ea6db076aceb2be3a65fd17cbcdcee0c2e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13279/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13279/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4181" title="lnet-selftest test_smoke: lst error found" class="issue-link" data-issue-key="LU-4181"><del>LU-4181</del></a> lnet_selftest: bogus lst errors<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8cfa7f80d3975298d5738091da7d9c28d6f5c9f5</p> <p>Didn't see this issue in build #2821(include the fix) and #2822, verified.</p> <p>Thanks Sarah.</p> <p>Hit this master branch build #2983 with EL7 client<br/> <a href="https://testing.hpdd.intel.com/test_sets/f7370aee-df8a-11e4-bf2e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f7370aee-df8a-11e4-bf2e-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[LNet Rates of c] [R] Avg: 272 RPC/s Min: 272 RPC/s Max: 272 RPC/s [W] Avg: 265 RPC/s Min: 265 RPC/s Max: 265 RPC/s [LNet Bandwidth of c] [R] Avg: 23.06 MB/s Min: 23.06 MB/s Max: 23.06 MB/s [W] Avg: 20.04 MB/s Min: 20.04 MB/s Max: 20.04 MB/s killing 5477 ... RPC failure, can't show error on 12345-10.2.4.205@tcp 12345-10.2.4.206@tcp: [Session 0 brw errors, 0 ping errors] [RPC: 2 errors, 0 dropped, 0 expired] c: Total 2 error nodes in c 12345-10.2.4.203@tcp: [Session 120 brw errors, 15 ping errors] [RPC: 62 errors, 58 dropped, 131 expired] 12345-10.2.4.204@tcp: [Session 120 brw errors, 15 ping errors] [RPC: 1 errors, 0 dropped, 135 expired] s: Total 2 error nodes in s stop batch RPC failed on 12345-10.2.4.205@tcp: Unknown error -110 session is ended Total 2 error nodes in c Total 2 error nodes in s lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found </pre> </div></div> <p>This appeared to be a different issue: while previously the errors were bogus, here there were more than one hundred expired RPCs.</p> <p>Close this old ticket again. New test failures are being tracked under <a href="https://jira.whamcloud.com/browse/LU-6622" title="lnet-selftest test_smoke: @@@@@@ FAIL: lst Error found" class="issue-link" data-issue-key="LU-6622"><del>LU-6622</del></a>.</p> Related LU-8705 Development Rank 1|hzw78v: Rank (Obsolete) 11318 Severity