https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2194 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/a7b63102-1737-11e2-afe1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a7b63102-1737-11e2-afe1-52540035b04c</a>.</p> <p>The sub-test test_19b failed with the following error:</p> <blockquote> <p>no eviction</p></blockquote> <p>Info required for matching: recovery-small 19b</p> LU-2194

Test failure on test suite recovery-small, subtest test_19b

Bug Blocker Resolved Fixed Hongchao Zhang Maloo LB Tue, 16 Oct 2012 04:02:08 +0000 Sun, 3 Apr 2016 03:25:55 +0000 Tue, 7 Jul 2015 00:06:11 +0000 Lustre 2.4.0 Lustre 2.4.0 Lustre 2.8.0 0 15 <p><a href="http://review.whamcloud.com/4652" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4652</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c</a></p> <p>same error as above link <a href="https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/3e897880-432c-11e2-b57a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3e897880-432c-11e2-b57a-52540035b04c</a></p> <p>the reason is the client is evicted by OST, which is the result of previous subtest "test_19a"<br/> in "test_19a", fail_loc OBD_FAIL_LDLM_CANCEL_NET is set at MDT &amp; OSTs, and it detect an eviction (was from MDT), <br/> then it assume the test passed and continue to run test_19b, which encountered the eviction from OST.</p> <p>Landed for 2.4</p> <p>The same error as comment 2/3 above, is seen again with patch in comment 1 (commit cc980df563ef86847aae1e0a3f0a5b17589e6297) applied.</p> <p><a href="https://maloo.whamcloud.com/test_sets/070528a6-4cea-11e2-bf7d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/070528a6-4cea-11e2-bf7d-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c</a></p> <p>I don't know if this is exactly the same error originally reported but it is the same test and the same errors Peng Tao is seeing. </p> <p>Error: 'test_19b failed with 1' </p> <p><a href="https://maloo.whamcloud.com/test_sessions/4d2113e8-471a-11e2-9537-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/4d2113e8-471a-11e2-9537-52540035b04c</a></p> <p>In my example the on the MDS the lnet service thead gets tied up: <br/> "LNet: Service thread pid 7757 completed after 150.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources)."<br/> There is a 40 second thread hung stack dump and plenty of other errors in the dmesg log about this Lnet service thread 7757. </p> <p>It seems test_19b is still having issues. </p> <p>same error: <br/> <a href="https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c</a></p> <p>Another failure:</p> <p><a href="https://maloo.whamcloud.com/test_sets/ac8aa71c-6034-11e2-84d4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ac8aa71c-6034-11e2-84d4-52540035b04c</a></p> <p>Is the environment not set before the tests are starting in some way?</p> <p>another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/119c11ae-6233-11e2-b20c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/119c11ae-6233-11e2-b20c-52540035b04c</a></p> <p><a href="http://review.whamcloud.com/5141" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5141</a></p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/da6627c0-64b1-11e2-a1aa-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/da6627c0-64b1-11e2-a1aa-52540035b04c</a></p> <p>Yet another failure instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/1120911e-69a9-11e2-954a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1120911e-69a9-11e2-954a-52540035b04c</a></p> <p>Another one:<br/> <a href="https://maloo.whamcloud.com/test_sets/145ca582-6ab4-11e2-bea0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/145ca582-6ab4-11e2-bea0-52540035b04c</a></p> <p>Another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/1fb3c870-6ae1-11e2-bea0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1fb3c870-6ae1-11e2-bea0-52540035b04c</a></p> <p>Has this reoccurred since the extra debugging has been landed?</p> <p>Found one post debugging:<br/> <a href="https://maloo.whamcloud.com/test_sets/e3c10bd4-7878-11e2-9928-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e3c10bd4-7878-11e2-9928-52540035b04c</a></p> <p>Client error for multiop:</p> <blockquote> <p>Lustre: DEBUG MARKER: multiop /mnt/lustre/f.recovery-small.19b Ow<br/> LustreError: 11-0: lustre-OST0000-osc-ffff88007c0b8000: Communicating with 10.10.17.16@tcp, operation ldlm_enqueue failed with -107.<br/> LustreError: 23456:0:(ldlm_request.c:1267:ldlm_cli_cancel_req()) Got rc -5 from cancel RPC: canceling anyway<br/> LustreError: 23456:0:(ldlm_request.c:1904:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -5<br/> LustreError: 23456:0:(ldlm_request.c:1904:ldlm_cli_cancel_list()) Skipped 1 previous similar message<br/> Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_19b: @@@@@@ FAIL: failed to run multiop: 5 </p></blockquote> <p>another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/48d621c0-7271-11e2-9b41-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/48d621c0-7271-11e2-9b41-52540035b04c</a></p> <p>another hit at <a href="https://maloo.whamcloud.com/test_sets/87356fc4-7f6f-11e2-afad-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/87356fc4-7f6f-11e2-afad-52540035b04c</a></p> <p>from OST console message, it says</p> <blockquote> <p>07:12:30:Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) == 07:12:23 (1361805143)<br/> 07:12:30:LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 153s: evicting client at 10.10.17.14@tcp ns: filter-ffff88006b003000 lock: ffff8800679ac000/0xf574bdda33b2fbc1 lrc: 3/0,0 mode: PR/PR res: 36/0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x10020 nid: 10.10.17.14@tcp remote: 0x2e77606da0e4cfde expref: 4 pid: 29406 timeout: 4307850023 lvb_type: 1<br/> 07:12:30:LNet: Service thread pid 29405 completed after 150.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> 07:12:30:Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_19b: @@@@@@ FAIL: failed to run multiop: 5 </p></blockquote> <p>and OST evicts client</p> <p>another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/eeae8204-85d6-11e2-9f8d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eeae8204-85d6-11e2-9f8d-52540035b04c</a></p> <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,5679" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5679</a></p> <p>Landed for 2.4</p> <p>recovery-small test_19b is still being skipped due to this bug.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13671" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13671</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2194" title="Test failure on test suite recovery-small, subtest test_19b" class="issue-link" data-issue-key="LU-2194"><del>LU-2194</del></a> test: remove test_19b from except list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7531f2ac0495f021d79427bf9dfab41a2d269db6</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13671/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13671/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2194" title="Test failure on test suite recovery-small, subtest test_19b" class="issue-link" data-issue-key="LU-2194"><del>LU-2194</del></a> test: remove test_19b from except list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 87e12095044c905078008efe11b3476374088c3e</p> Development Rank 1|hzvahz: Rank (Obsolete) 5236 Severity