Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-15123] sanity-quota: test_7a Error: 'reintegration failed' https://jira.whamcloud.com/browse/LU-15123 Lustre <p>This issue was created by maloo for paf <pfarrell@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/ded8a9a9-b77c-41ca-bb53-105290b2709e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ded8a9a9-b77c-41ca-bb53-105290b2709e</a></p> LU-15123

sanity-quota: test_7a Error: 'reintegration failed'

Bug Minor Resolved Fixed Alex Zhuravlev Maloo Mon, 18 Oct 2021 14:50:33 +0000 Wed, 2 Aug 2023 13:08:53 +0000 Wed, 26 Apr 2023 13:33:48 +0000 Lustre 2.16.0 Lustre 2.15.3 Lustre 2.16.0 Lustre 2.15.4 0 11 <p>+1 on master - <a href="https://testing.whamcloud.com/test_sets/2d408dbe-88f6-428a-ae7e-0c8796fb3207" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2d408dbe-88f6-428a-ae7e-0c8796fb3207</a></p> <p>+1 on master - <a href="https://testing.whamcloud.com/test_sets/f07f8d61-0faa-41ad-9217-c238bd4c2bb0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f07f8d61-0faa-41ad-9217-c238bd4c2bb0</a></p> <p>I guess It fails because OST0000 waits for client 1 in a recovery blocking reintegration to start(and finish). Finally it evicts this client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15386.910732] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 5 clients reconnect ... [15485.424725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-quota test_7a: @@@@@@ FAIL: reintegration failed [15485.858740] Lustre: DEBUG MARKER: sanity-quota test_7a: @@@@@@ FAIL: reintegration failed [15486.354991] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-1/2021-12-15/lustre-reviews_review-dne-zfs-part-4_85112_1_13_4b065c95-2177-4a6a-b5c8-32b025199627//sanity-quota.test_7a.debug_log.$(hostname -s).1639605011.log; dmesg > /autotest/autotest-1/2021-12-15/lustre-review [15488.876821] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [15488.878099] Lustre: lustre-OST0000: disconnecting 1 stale clients [15489.069798] Lustre: lustre-OST0000: Recovery over after 1:42, of 5 clients 4 recovered and 1 was evicted.</pre> </div></div> <p>Logs from client1 dmesg:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15387.000236] Lustre: lustre-OST0000-osc-ffff9247ca727000: Connection to lustre-OST0000 (at 10.240.26.143@tcp) was lost; in progress operations using this service will wait for recovery to complete [15407.479024] Lustre: lustre-OST0001-osc-ffff9247ca727000: disconnect after 23s idle [15409.071515] Lustre: lustre-OST0000-osc-ffff9247ca727000: Connection restored to 10.240.26.143@tcp (at 10.240.26.143@tcp) [15474.038827] LustreError: 11-0: lustre-OST0000-osc-ffff9247ca727000: operation ost_disconnect to node 10.240.26.143@tcp failed: rc = -107</pre> </div></div> <p> </p> <p>+3 on master, all on the same patch:<br/> <a href="https://testing.whamcloud.com/test_sessions/dbc4c026-0863-41b3-b17c-b695415fa8aa" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/dbc4c026-0863-41b3-b17c-b695415fa8aa</a><br/> <a href="https://testing.whamcloud.com/test_sessions/3db9b21f-fd9b-49a0-bbd7-5dda5aea1be9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/3db9b21f-fd9b-49a0-bbd7-5dda5aea1be9</a><br/> <a href="https://testing.whamcloud.com/test_sessions/e22114c8-b11d-4494-ae19-030d82261043" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/e22114c8-b11d-4494-ae19-030d82261043</a></p> <p>Still being hit on master, 14/310 runs in the past week.</p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/7cd51716-a038-4e24-a46b-4a14e83cc1a2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7cd51716-a038-4e24-a46b-4a14e83cc1a2</a></p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/db2d7940-4287-4ff0-9435-81c6520360b7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/db2d7940-4287-4ff0-9435-81c6520360b7</a> </p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/dc29fa4d-ef8c-4838-a182-7a544385f4cc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/dc29fa4d-ef8c-4838-a182-7a544385f4cc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/3d837ba0-73c7-4737-9894-5bbca2c9b479" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3d837ba0-73c7-4737-9894-5bbca2c9b479</a></p> <p>+1 on b2_15 branch: <a href="https://testing.whamcloud.com/test_sets/150ed79f-6d70-4048-b875-56a9bccc54cf" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/150ed79f-6d70-4048-b875-56a9bccc54cf</a></p> <blockquote></blockquote> <p><span class="error">[13964.128411]</span> Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180<br/> <span class="error">[13965.655492]</span> Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 5 clients reconnect<br/> ...<br/> <span class="error">[14061.885567]</span> Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-quota test_7a: @@@@@@ FAIL: reintegration failed <br/> <span class="error">[14067.469119]</span> Lustre: lustre-OST0000: recovery is timed out, evict stale exports<br/> <span class="error">[14067.470635]</span> Lustre: lustre-OST0000: disconnecting 1 stale clients<br/> <span class="error">[14067.787175]</span> Lustre: lustre-OST0000: Recovery over after 1:42, of 5 clients 4 recovered and 1 was evicted.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>reintegration starts only when recovery is over. in this case the recovery process was stuck due to a missing client (to be evicted in the end) and the recovery process took 102 seconds while test 7a waits 90s at most.</p> <p>"Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50688" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50688</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15123" title="sanity-quota: test_7a Error: 'reintegration failed'" class="issue-link" data-issue-key="LU-15123"><del>LU-15123</del></a> tests: quota reintegration starts after recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6a2cb8c1aa0a96ec1d6e4603132635459ac0615</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50688/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50688/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15123" title="sanity-quota: test_7a Error: 'reintegration failed'" class="issue-link" data-issue-key="LU-15123"><del>LU-15123</del></a> tests: check quota reintegration after recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4432b6e2824775e292f96e202d6fc0db231bc749</p> <p>Landed for 2.16</p> <p>"Xing Huang <hxing@ddn.com>" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51233" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51233</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15123" title="sanity-quota: test_7a Error: 'reintegration failed'" class="issue-link" data-issue-key="LU-15123"><del>LU-15123</del></a> tests: check quota reintegration after recovery<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 8002f7d111f817032ffe8ee3485abf5e4472a148</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51233/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51233/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15123" title="sanity-quota: test_7a Error: 'reintegration failed'" class="issue-link" data-issue-key="LU-15123"><del>LU-15123</del></a> tests: check quota reintegration after recovery<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 13805e3a2d4f520e297bc408d94b9971a6094f9a</p> Related LU-13795 Development Rank 1|i027n3: Rank (Obsolete) 9223372036854775807 Severity