https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13343 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/1b932583-bcd8-4933-9b6a-2d70a0b7aecc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1b932583-bcd8-4933-9b6a-2d70a0b7aecc</a></p> <p>test_140a failed with the following error on the client console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 245.535452] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock,skpath=/tmp/test-framework-keys trevis-3vm9@tcp:/lustre /mnt/lustre2 [ 245.868341] LustreError: 15982:0:(gss_keyring.c:862:gss_sec_lookup_ctx_kr()) failed request key: -126 [ 245.869838] LustreError: 15982:0:(sec.c:449:sptlrpc_req_get_ctx()) req ffff9fb5ed3c3600: fail to get context [ 245.871460] LustreError: 15982:0:(lmv_obd.c:306:lmv_connect_mdc()) target lustre-MDT0000_UUID connect error -111 [ 245.872973] LustreError: 15982:0:(llite_lib.c:320:client_common_fill_super()) cannot connect to lustre-clilmv-ffff9fb5f8a2d000: rc = -111 [ 245.882513] LustreError: 15982:0:(lov_obd.c:824:lov_cleanup()) lustre-clilov-ffff9fb5f8a2d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 245.891188] Lustre: Unmounted lustre-client [ 245.892122] LustreError: 15982:0:(obd_mount.c:1681:lustre_fill_super()) Unable to mount (-111) [ 246.144872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_140a: @@@@@@ FAIL: unable to mount \/mnt\/lustre2 on MDS [ 246.335604] Lustre: DEBUG MARKER: recovery-small test_140a: @@@@@@ FAIL: unable to mount /mnt/lustre2 on MDS </pre> </div></div> <p>It looks like the mount2 setup is missing the security setup.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> recovery-small test_140a - unable to mount /mnt/lustre2 on MDS</p> LU-13343

recovery-small test_140a: unable to mount /mnt/lustre2 on MDS

Bug Minor Resolved Fixed Sebastien Buisson Maloo Mon, 9 Mar 2020 10:30:57 +0000 Thu, 1 Feb 2024 14:35:09 +0000 Sun, 9 Jul 2023 14:14:36 +0000 Lustre 2.16.0 0 7 <p>It looks like this test has been failing since 2020-01-23, about 160 times, as many as 8 times a day.</p> <p>Per <a href="https://jira.whamcloud.com/browse/LU-13342" title="recovery-small test_140a and test_140b incompatible with SSK" class="issue-link" data-issue-key="LU-13342"><del>LU-13342</del></a>, Sebastien writes:</p> <blockquote> <p>recovery-small test_140a and test_140b are using a 'local client' ie a client mounted on a server node, which is not compatible with SSK keys installed by the test framework.</p> <p>So just skip these tests when running with SSK.</p></blockquote> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37832" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37832</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> tests: skip recovery-small test_140 with SSK<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4c37186c40eee52e8f25aec66c73c4ba5d39432f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37832/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37832/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> tests: skip recovery-small test_140 with SSK<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: df5b71a5e8031929d816b19a2f045bfa40388b5b</p> <p>I was going to resolve this issue as "fixed" rather than marking it "<tt>always_except</tt>" and keeping it open, because my understanding is that local mounts would/should never use SSK/Kerberos as there is no transport security needed for <tt>memcpy()</tt> within the node.</p> <p>That said, <em>should</em> there be an internal check for connections from the local NID to skip the <tt>gss_sec_lookup_ctx_kr()</tt> code/check so that this doesn't generate an error at all? Otherwise, it seems like if SSK is configured on a server it would prevent local mounts from working at all? Am I misunderstanding the issue here?</p> <p>The problem is just about the test framework. There is no restriction to use SSK with local clients (ie clients mounted on server nodes), it is just that it requires installing the client key as well on these nodes.<br/> ATM the test framework does not handle this situation: each node has a dedicated role, and is either an MDS or an OSS or a client.</p> <p>This is why I pushed this patch to skip the few tests that are using local clients, when SSK is enabled. If the number of cases increase, we would want to have SSK keys installed properly wherever they are needed.</p> <p>What I'm asking about is the reverse - if SSK is enabled on the servers, it seems to me that it <em>should</em> be possible to mount a local client <b>without</b> the need to configure SSK for that client. I can't see any benefit to SSK/KRB for a local client mount, since the data doesn't go over the network, and the server itself can verify that the NID of the client is local.</p> <p>On the one hand, you are raising an interesting point. There is no real added value of checking SSK/KRB credentials for a local node.</p> <p>On the other hand, one of the purposes of strong authentication is to define roles for nodes. When you install credentials on nodes (MDS, OSS, client), you explicitly assign them a role, meaning you want to prevent nodes from being re-purposed. So if we implement what you suggest, we would weaken this principle, with the initial intention of making local mounts easier.</p> <p>I am not against what you suggest, but we have to be aware if the implications.</p> <p>For clients mounting a local filesystem on the server for data movement or protocol re-export there is a need for the local client mount. I agree that the admin could configure the "client" for this local mount, but even if that was not inconvenient for the admin, it would still hurt performance due to encryption overhead for both the "client" and the "server" running on the same node for no real benefit, so we would likely recommend against using KRB/SSK for local connections.</p> <p>Understood. So I will work on a patch to add an internal check for connections from the local NID to skip the gss_sec_lookup_ctx_kr() code/check. Please leave this ticket open so that we keep in mind we already have recovery-small test_140a and test_140b that are making use of local clients.</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46704" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46704</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> gss: no sec flavor on loopback connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 635d0545b57e19c4139c48789d0d1cb009ab09a6</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46736" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46736</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> debug: debug recovery-small 140<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0332c6a72d770eabb136774b4da512ae2d75c231</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50941" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50941</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> dbg: print debug traces<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 77f3505e20acec948fccf24c4ee8ce9aae1a78b8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/46704/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/46704/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> gss: no sec flavor on loopback connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e3e91ea95fd96a5eafc598e3812390b4cbac05c3</p> <p>Landed for 2.16</p> <p>"Xing Huang &lt;hxing@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51616" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51616</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13343" title="recovery-small test_140a: unable to mount /mnt/lustre2 on MDS" class="issue-link" data-issue-key="LU-13343"><del>LU-13343</del></a> gss: no sec flavor on loopback connection<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 3947a3b87798dae5b225b8df2d8effaed0e9c933</p> Duplicate LU-13342 Related Development Rank 1|i00v4f: Rank (Obsolete) 9223372036854775807 Severity