https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7854 Lustre <p>As part of the Kerberos test plan, I ran the test-group/regression suite of tests including sanity-gss. I set up a Kerberos environment and ran the regression test group. sanity-gss tests 1 and 2 fail in the same way; with a connection refused error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-gss test 1: create file == 10:46:36 (1457001996) chmod: cannot access `/lustre/scratch': Connection refused sanity-gss test_1: @@@@@@ FAIL: chmod /lustre/scratch failed </pre> </div></div> <p>From the dmesg from the MDS for test 2, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: Setting sptlrpc rule: scratch.srpc.flavor.default=gssnull Lustre: 9164:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126 LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 1423 previous similar messages LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-lwp-MDT0000_UUID-&gt;scratch-MDT0000_UUID LustreError: 7734:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126 LustreError: 7734:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 203419 previous similar messages LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 209084 previous similar messages LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126 LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 461885 previous similar messages LustreError: 7730:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03980: fail to get context LustreError: 7730:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 467259 previous similar messages Lustre: DEBUG MARKER: == sanity-gss test 1: create file == 10:46:36 (1457001996) LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126 LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 950299 previous similar messages LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 958900 previous similar messages Lustre: DEBUG MARKER: sanity-gss test_1: @@@@@@ FAIL: chmod /lustre/scratch failed LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-mdtlov_UUID-&gt;scratch-OST0000_UUID Lustre: DEBUG MARKER: == sanity-gss test 2: lfs flushctx == 10:46:40 (1457002000) LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126 LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 1953286 previous similar messages LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 1963518 previous similar messages Lustre: DEBUG MARKER: sanity-gss test_2: @@@@@@ FAIL: chmod /lustre/scratch failed LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-mdtlov_UUID-&gt;scratch-OST0000_UUID LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) Skipped 4 previous similar messages </pre> </div></div> <p>Only three of 13 tests passed on this and previous runs of sanity-gss. The logs for the sanity-gss test suite are at <a href="https://testing.hpdd.intel.com/test_sets/09108032-e16b-11e5-8edf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/09108032-e16b-11e5-8edf-5254006e85c2</a></p> COmbined MGS/MDS with one MDT, two OSSs with two OSTs, a single client. Kerberos is setup LU-7854

sanity-gss test_1 fails with 'chmod /lustre/scratch failed'

Bug Minor Resolved Fixed Sebastien Buisson James Nunez Thu, 3 Mar 2016 22:38:03 +0000 Thu, 15 Mar 2018 14:09:11 +0000 Thu, 15 Mar 2018 14:09:11 +0000 Lustre 2.9.0 Lustre 2.11.0 0 7 <p>Is this a regression, or has this test not been run/passed recently?</p> <p>This is not a regression and it looks like this test suite is rarely executed. Looking at Maloo results for sanity-gss for the past year and a half, it looks like this test suite has not passed testing in that time frame.</p> <p>Jeremy, Sebastien, have you ever run the sanity-gss.sh script successfully? Are there any patches needed to get this working in our test environment?</p> <p>Sebastien, this is something that should be run as part of your patch <a href="http://review.whamcloud.com/18781" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18781</a> "<a href="https://jira.whamcloud.com/browse/LU-7845" title="Support namespace in credentials retrieval" class="issue-link" data-issue-key="LU-7845"><del>LU-7845</del></a> gss: support namespace in lgss_keyring" to ensure there are no regressions.</p> <p>I believe I've had it run through once, but so far I haven't added patches for SK testing. Looking at sanity-gss it's using gssnull which requires the lsvcgssd to be running but I don't see any start_gss_daemons called there. The -126 is -ENOKEY which would make sense given that gssnull requires lsvcgssd to be running with the -z flag (from my last patch set for shared key).</p> <p>Jeremy - Thanks for the comment. I have lsvcgssd running on all servers, but I didn't give it any flags.</p> <p>Thanks Andreas.<br/> I have never run sanity-gss in my environment, but I think Jeremy's suggestion could do it. When it is fixed, I will add sanity-gss to the Test-Parameters of my patch at <a href="http://review.whamcloud.com/18781" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18781</a>.</p> <p>Is this still a live issue or was it fixed under <a href="https://jira.whamcloud.com/browse/LU-7845" title="Support namespace in credentials retrieval" class="issue-link" data-issue-key="LU-7845"><del>LU-7845</del></a>?</p> <p>Hi,</p> <p>This issue is not directly related to <a href="https://jira.whamcloud.com/browse/LU-7845" title="Support namespace in credentials retrieval" class="issue-link" data-issue-key="LU-7845"><del>LU-7845</del></a>, although I once proposed to run sanity-gss while review of <a href="http://review.whamcloud.com/18781" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18781</a> was in progress. And it is not fixed now that patch from <a href="https://jira.whamcloud.com/browse/LU-7845" title="Support namespace in credentials retrieval" class="issue-link" data-issue-key="LU-7845"><del>LU-7845</del></a> is merged.</p> <p>I think the best way to tackle this is to push a new patch to modify sanity-gss according to Jeremy’s suggestion (i.e. starting lsvcgssd) and add sanity-gss to the Test-Parameters of the patch.</p> <p>Sebastien.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/27383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27383</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7854" title="sanity-gss test_1 fails with &#39;chmod /lustre/scratch failed&#39;" class="issue-link" data-issue-key="LU-7854"><del>LU-7854</del></a> tests: start gss daemons in sanity-gss<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 81d2e2f4f15b0cb71d44cafe95e7e0c17380ce29</p> <p>Hi,</p> <p>Following Jeremy's suggestion, I uploaded patch <a href="https://review.whamcloud.com/27383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27383</a> in order to start lsvcgssd deamons with '-z' flag for gssnull flavor.<br/> This enables test_1 to pass, but I found some instabilities with gssnull flavor while working on sanity-gss. See <a href="https://jira.whamcloud.com/browse/LU-9582" title="gssnull instability" class="issue-link" data-issue-key="LU-9582">LU-9582</a>. However, I think patch <a href="https://review.whamcloud.com/27383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27383</a> should be landed independently of those instabilities, that could be addressed separately.</p> <p>Sebastien.</p> <p>The tests cannot pass because of the problem described in <a href="https://jira.whamcloud.com/browse/LU-9567" title="sptlrpc rules are not being updated" class="issue-link" data-issue-key="LU-9567"><del>LU-9567</del></a>.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sbuisson" class="user-hover" rel="sbuisson">sbuisson</a> just to confirm - you are now able to move forward with this fix now, right?</p> <p>Sure, I have just refreshed patch at <a href="https://review.whamcloud.com/27383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27383</a>.</p> <p>FYI, work on GSS (or Shared Key or Kerberos) is currently blocked because of issue described in <a href="https://jira.whamcloud.com/browse/LU-10531" title="GSS, Shared Key and Kerberos support broken in master and lustre 2.10" class="issue-link" data-issue-key="LU-10531"><del>LU-10531</del></a>, due to the following patch recently landed to master branch:<br/> <a href="https://review.whamcloud.com/28590" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28590</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sbuisson" class="user-hover" rel="sbuisson">sbuisson</a> is work now able to proceed since the patches for <a href="https://jira.whamcloud.com/browse/LU-10531" title="GSS, Shared Key and Kerberos support broken in master and lustre 2.10" class="issue-link" data-issue-key="LU-10531"><del>LU-10531</del></a> have landed?</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/31317" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31317</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7854" title="sanity-gss test_1 fails with &#39;chmod /lustre/scratch failed&#39;" class="issue-link" data-issue-key="LU-7854"><del>LU-7854</del></a> gss: install lgssc.conf under /etc/request-key.d/<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ef1a4773d457ca9be475c77bf1f8632da777e634</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31317/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31317/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7854" title="sanity-gss test_1 fails with &#39;chmod /lustre/scratch failed&#39;" class="issue-link" data-issue-key="LU-7854"><del>LU-7854</del></a> gss: install lgssc.conf under /etc/request-key.d/<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e299df1e9eeae5d20f3b8a544a5f4be4fd30872c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27383/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27383/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7854" title="sanity-gss test_1 fails with &#39;chmod /lustre/scratch failed&#39;" class="issue-link" data-issue-key="LU-7854"><del>LU-7854</del></a> tests: start gss daemons in sanity-gss<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 69b2712e2ffe485ee21408b849040d42b5a9aa2a</p> <p>Landed for 2.11</p> Blocker LU-9567 Related LU-10531 Development Rank 1|hzy3lr: Rank (Obsolete) 9223372036854775807 Severity