Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-14947] recovery-small test_26a: client not evicted from OST https://jira.whamcloud.com/browse/LU-14947 Lustre <p>This issue was created by maloo for Qian Yingjin <qian@ddn.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/acf1ad3c-16d3-4cac-9376-40485f9ed5f1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/acf1ad3c-16d3-4cac-9376-40485f9ed5f1</a></p> <p>test_26a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client not evicted from OST </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> recovery-small test_26a - client not evicted from OST</p> LU-14947

recovery-small test_26a: client not evicted from OST

Bug Major Open Unresolved Mikhail Pershin Maloo failing_tests Tue, 17 Aug 2021 08:02:00 +0000 Thu, 21 Dec 2023 00:59:33 +0000 Lustre 2.12.8 Lustre 2.15.0 0 8 <p>Found similar issue in recovery-small test set on 2.12.8: <a href="https://testing.whamcloud.com/test_sets/48b57407-656d-46a2-bcb5-2809ffc48c29" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/48b57407-656d-46a2-bcb5-2809ffc48c29</a> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5919:error() = /usr/lib64/lustre/tests/test-framework.sh:9154:check_clients_evicted() = /usr/lib64/lustre/tests/recovery-small.sh:1094:test_26a() = /usr/lib64/lustre/tests/test-framework.sh:6222:run_one() = /usr/lib64/lustre/tests/test-framework.sh:6271:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:6111:run_test() = /usr/lib64/lustre/tests/recovery-small.sh:1097:main() Dumping lctl log to /autotest/autotest-1/2021-11-20/lustre-b2_12_failover-part-1_150_1_40_13db9919-f21e-4132-8be9-3d11b4f5908e<span class="code-comment">//recovery-small.test_26a.*.1637428776.log </span>CMD: onyx-109vm10,onyx-24vm6,onyx-64vm1.onyx.whamcloud.com,onyx-64vm3,onyx-64vm4 /usr/sbin/lctl dk > /autotest/autotest-1/2021-11-20/lustre-b2_12_failover-part-1_150_1_40_13db9919-f21e-4132-8be9-3d11b4f5908e<span class="code-comment">//recovery-small.test_26a.debug_log.\$(hostname -s).1637428776.log; </span></pre> </div></div> <p>I think this is the same issue as reported in the closed ticket <a href="https://jira.whamcloud.com/browse/LU-10632" title="recovery-small test 26a fails with ‘client not evicted from OST’" class="issue-link" data-issue-key="LU-10632"><del>LU-10632</del></a>. </p> <p>For 2.12.8 we’ve see it a couple of times in November<br/> 2.12.7.28 - <a href="https://testing.whamcloud.com/test_sets/0ee7439d-1b67-436e-bbb9-2bc9d0561594" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0ee7439d-1b67-436e-bbb9-2bc9d0561594</a><br/> 2.12.8 - <a href="https://testing.whamcloud.com/test_sets/48b57407-656d-46a2-bcb5-2809ffc48c29" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/48b57407-656d-46a2-bcb5-2809ffc48c29</a><br/> and for master once in the past two months<br/> 2.14.55.43 - <a href="https://testing.whamcloud.com/test_sets/d594a8a7-9153-4741-8289-1258864c9c80" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d594a8a7-9153-4741-8289-1258864c9c80</a></p> <p>From the master failure, we see that no eviction took place in the time frame expected</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Check state for lustre-OST0000-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0000-osc-ffff97ad5fb89800.state lustre-OST0000-osc-ffff97ad5fb89800 is evicted at 1634938357 Check state for lustre-OST0001-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0001-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0001-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0001-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] Check state for lustre-OST0002-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0002-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0002-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0002-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] Check state for lustre-OST0003-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0003-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0003-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0003-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] Check state for lustre-OST0004-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0004-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0004-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0004-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] Check state for lustre-OST0005-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0005-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0005-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0005-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] Check state for lustre-OST0006-osc-ffff97ad5fb89800 CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0006-osc-ffff97ad5fb89800.state /usr/lib64/lustre/tests/test-framework.sh: line 9488: ((: > 1634938297: syntax error: operand expected (error token is "> 1634938297") lustre-OST0006-osc-ffff97ad5fb89800 was not evicted after 1634938297: CMD: trevis-209vm1.trevis.whamcloud.com /usr/sbin/lctl get_param osc.lustre-OST0006-osc-ffff97ad5fb89800.state - [ 1634938350, DISCONN ] - [ 1634938350, CONNECTING ] - [ 1634938351, DISCONN ] - [ 1634938351, CONNECTING ] - [ 1634938352, DISCONN ] - [ 1634938352, CONNECTING ] - [ 1634938352, RECOVER ] - [ 1634938352, FULL ] recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6320:error() = /usr/lib64/lustre/tests/test-framework.sh:9498:check_clients_evicted() = /usr/lib64/lustre/tests/recovery-small.sh:1087:test_26a() </pre> </div></div> <p>The 2.12.8 failure at <a href="https://testing.whamcloud.com/test_sets/0ee7439d-1b67-436e-bbb9-2bc9d0561594" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0ee7439d-1b67-436e-bbb9-2bc9d0561594</a> looks like an eviction did take place in time</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Check state for lustre-OST0001-osc-ffff98697bf0b000 CMD: onyx-66vm8 /usr/sbin/lctl get_param osc.lustre-OST0001-osc-ffff98697bf0b000.state /usr/lib64/lustre/tests/test-framework.sh: line 9144: ((: > 1636932232: syntax error: operand expected (error token is "> 1636932232") lustre-OST0001-osc-ffff98697bf0b000 was not evicted after 1636932232: CMD: onyx-66vm8 /usr/sbin/lctl get_param osc.lustre-OST0001-osc-ffff98697bf0b000.state - [ 1636932288, CONNECTING ] - [ 1636932288, EVICTED ] - [ 1636932288, RECOVER ] - [ 1636932288, FULL ] - [ 1636932308, CONNECTING ] - [ 1636932308, IDLE ] - [ 1636932313, CONNECTING ] - [ 1636932313, FULL ] </pre> </div></div> <p>+1 on master<br/> <a href="https://testing.whamcloud.com/test_sets/f4a1c4f1-b91b-4648-abcd-19f452939b6c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f4a1c4f1-b91b-4648-abcd-19f452939b6c</a></p> <p>+1 on master<br/> <a href="https://testing.whamcloud.com/test_sets/d0a65bcc-9af0-4224-9727-41507dfce00f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d0a65bcc-9af0-4224-9727-41507dfce00f</a></p> <blockquote> <p>+1 on master<br/> <a href="https://testing.whamcloud.com/test_sets/d0a65bcc-9af0-4224-9727-41507dfce00f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d0a65bcc-9af0-4224-9727-41507dfce00f</a></p></blockquote> <p>This was induced by <a href="https://review.whamcloud.com/#/c/43834/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43834/</a></p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tappro" class="user-hover" rel="tappro">tappro</a> </p> <p>Here's another one we're seeing. Can you please take a look? Thank you!</p> <p>There is a patch in <a href="https://jira.whamcloud.com/browse/LU-12066" title="recovery-small test 26b fails with “Client was not evicted by ost rc=1”" class="issue-link" data-issue-key="LU-12066">LU-12066</a> (duplicate of this one) which may fix this problem for test_26b, and the same change may also fix test_26a. I would close this as a duplicate, but the error message has changed slightly and this ticket matches the current message, and I don't want to close <a href="https://jira.whamcloud.com/browse/LU-12066" title="recovery-small test 26b fails with “Client was not evicted by ost rc=1”" class="issue-link" data-issue-key="LU-12066">LU-12066</a> because it has the patch. First step should be to review/rebase that patch to see if it fixes the problem.</p> Related LU-10632 LU-14748 LU-12066 Development Rank 1|i021yn: Rank (Obsolete) 9223372036854775807 Severity