https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17317 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/d7c41dec-3599-4738-b89d-03f240498d8c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d7c41dec-3599-4738-b89d-03f240498d8c</a></p> <p>test_16 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test all_off:60001:c0:60003:003, wanted 1 1, got 0 0 </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/15743" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/15743</a> - 4.18.0-477.27.1.el8_8.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/15743" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/15743</a> - 4.18.0-477.27.1.el8_lustre.ddn17.x86_64</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-sec test_16 - test all_off:60001:c0:60003:003, wanted 1 1, got 0 0</p> LU-17317

sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0

Bug Minor Resolved Fixed WC Triage Maloo Tue, 28 Nov 2023 07:57:57 +0000 Tue, 9 Jan 2024 10:38:14 +0000 Tue, 9 Jan 2024 10:38:14 +0000 Lustre 2.16.0 0 4 <p>For instance for this failure in sanity-sec test_17:<br/> <a href="https://testing.whamcloud.com/test_sets/1037004b-dbe8-44ef-a668-4eed6e6114eb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1037004b-dbe8-44ef-a668-4eed6e6114eb</a></p> <p>We have the following nodes:</p> <ul class="alternate" type="square"> <li>MDS 1, MDS 3 (trevis-94vm6)<br/> 10.240.43.205@tcp</li> <li>Client 1 (trevis-94vm1)<br/> 10.240.43.200@tcp</li> </ul> <p>The MDS wants a client to release a lock, by sending an LDLM_BL_CALLBACK request (104). But it gets an error from the client (called a 'reverse server' in the GSS exchanges).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:1.0:1701460059.716146:0:10439:0:(client.c:1758:ptlrpc_send_new_req()) Sending RPC req@0000000014e4d538 pname:cluuid:pid:xid:nid:opc:job mdt00_003:lustre-MDT0002_UUID:10439:1784106942685952:10.240.43.200@tcp:104: 00000100:00100000:1.0:1701460059.716168:0:10439:0:(client.c:2533:ptlrpc_set_wait()) set 00000000e75f5696 going to sleep for 11 seconds 02000000:00000400:1.0:1701460059.716391:0:10439:0:(sec_gss.c:685:gss_cli_ctx_handle_err_notify()) req x1784106942685952/t0, ctx 0000000082de400c idx 0x544590679e7e5f1c(0-&gt;c): reverse server respond (00080000/00000000) 00000100:00020000:1.0:1701460059.716395:0:10439:0:(client.c:1479:after_reply()) @@@ unwrap reply failed: rc = -22 req@0000000014e4d538 x1784106942685952/t0(0) o104-&gt;lustre-MDT0002@10.240.43.200@tcp:15/16 lens 328/224 e 0 to 0 dl 1701460070 ref 1 fl Rpc:RQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1701460059.716403:0:10439:0:(client.c:2239:ptlrpc_check_set()) Completed RPC req@0000000014e4d538 pname:cluuid:pid:xid:nid:opc:job mdt00_003:lustre-MDT0002_UUID:10439:1784106942685952:10.240.43.200@tcp:104: </pre> </div></div> <p>And indeed on the client, the GSS context id (544590679e7e5f1c) is considered invalid.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1701460059.716218:0:8154:0:(events.c:373:request_in_callback()) peer: 12345-10.240.43.205@tcp (source: 12345-10.240.43.205@tcp) 02000000:00000400:0.0:1701460059.716236:0:207145:0:(gss_svc_upcall.c:1619:gss_svc_upcall_get_ctx()) Invalid gss ctx idx 0x544590679e7e5f1c from 10.240.43.205@tcp 02000000:08000000:0.0:1701460059.716239:0:207145:0:(sec_gss.c:1969:gss_pack_err_notify()) prepare gss error notify(0x80000/0x0) to 10.240.43.205@tcp </pre> </div></div> <p>So this client ends up not refreshing its lock. This is a problem as file/dir access rights were changed from a different client.</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53377" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53377</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17317" title="sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0" class="issue-link" data-issue-key="LU-17317"><del>LU-17317</del></a> gss: no cache flush for rsi and rsc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a785f7c4c2a71064cb61023e479ded0347d16b72</p> <p>The problem described here stems from the fact that server side can use outdated gss contexts in ldlm callbacks. Apparently this was fine with previous gss code based on sunrpc cache, because the cache entries were removed (very) asynchronously. With the new implementation based on the upcall cache, the cache entries are removed as they are found expired. This explains why with this new code, the server gets <tt>GSS_S_NO_CONTEXT</tt> from an evicted client if the server has sent an outdated gss context.</p> <p>So patch "<a href="https://jira.whamcloud.com/browse/LU-17317" title="sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0" class="issue-link" data-issue-key="LU-17317"><del>LU-17317</del></a> gss: do not continue using expired reverse context" <a href="https://review.whamcloud.com/53375" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/53375</a> aims at surviving this situation. The server is still allowed to try to use an outdated gss context (this is important for inflight communications), but if it gets <tt>GSS_S_NO_CONTEXT</tt> from the client, then it marks this gss context as dead, and replaces it with a new one.</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53405" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53405</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17317" title="sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0" class="issue-link" data-issue-key="LU-17317"><del>LU-17317</del></a> dbg: investigate test failures - 1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4b9b24ab89f4992ba40272cb0fddda0636a0152e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53377/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53377/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17317" title="sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0" class="issue-link" data-issue-key="LU-17317"><del>LU-17317</del></a> gss: no cache flush for rsi and rsc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3615fa4a86be793652d53c94818c5aeb81e2257e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53375/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53375/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17317" title="sanity-sec test_16: test all_off:60001:c0:60003:003, wanted 1 1, got 0 0" class="issue-link" data-issue-key="LU-17317"><del>LU-17317</del></a> gss: do not continue using expired reverse context<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 67acf6047e343a0e35f077c6aed4483a14d2015c</p> Related LU-17228 LU-17286 Development Rank 1|i042yv: Rank (Obsolete) 9223372036854775807 Severity