[LU-7854] sanity-gss test_1 fails with 'chmod /lustre/scratch failed'

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0
• Affects Version/s: Lustre 2.9.0
• Environment: COmbined MGS/MDS with one MDT, two OSSs with two OSTs, a single client. Kerberos is setup
• Severity: 3
• Assignee: Sebastien Buisson (Inactive)
• Reporter: James Nunez (Inactive)

Description

As part of the Kerberos test plan, I ran the test-group/regression suite of tests including sanity-gss. I set up a Kerberos environment and ran the regression test group. sanity-gss tests 1 and 2 fail in the same way; with a connection refused error:

== sanity-gss test 1: create file == 10:46:36 (1457001996)
chmod: cannot access `/lustre/scratch': Connection refused
 sanity-gss test_1: @@@@@@ FAIL: chmod /lustre/scratch failed 

From the dmesg from the MDS for test 2, we see

Lustre: DEBUG MARKER: Setting sptlrpc rule: scratch.srpc.flavor.default=gssnull
Lustre: 9164:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 1423 previous similar messages
LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-lwp-MDT0000_UUID->scratch-MDT0000_UUID
LustreError: 7734:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126
LustreError: 7734:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 203419 previous similar messages
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 209084 previous similar messages
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 461885 previous similar messages
LustreError: 7730:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03980: fail to get context
LustreError: 7730:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 467259 previous similar messages
Lustre: DEBUG MARKER: == sanity-gss test 1: create file == 10:46:36 (1457001996)
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 950299 previous similar messages
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 958900 previous similar messages
Lustre: DEBUG MARKER: sanity-gss test_1: @@@@@@ FAIL: chmod /lustre/scratch failed
LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-mdtlov_UUID->scratch-OST0000_UUID
Lustre: DEBUG MARKER: == sanity-gss test 2: lfs flushctx == 10:46:40 (1457002000)
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) failed request key: -126
LustreError: 7732:0:(gss_keyring.c:791:gss_sec_lookup_ctx_kr()) Skipped 1953286 previous similar messages
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) req ffff880036c03c80: fail to get context
LustreError: 7736:0:(sec.c:440:sptlrpc_req_get_ctx()) Skipped 1963518 previous similar messages
Lustre: DEBUG MARKER: sanity-gss test_2: @@@@@@ FAIL: chmod /lustre/scratch failed
LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) OOM trying to ping scratch-MDT0000-mdtlov_UUID->scratch-OST0000_UUID
LustreError: 29799:0:(pinger.c:105:ptlrpc_ping()) Skipped 4 previous similar messages

Only three of 13 tests passed on this and previous runs of sanity-gss. The logs for the sanity-gss test suite are at https://testing.hpdd.intel.com/test_sets/09108032-e16b-11e5-8edf-5254006e85c2

Issue Links

• is blocked by: LU-9567 sptlrpc rules are not being updated
• is related to: LU-10531 GSS, Shared Key and Kerberos support broken in master and lustre 2.10