[LU-17852] Lots of "Invalid gss ctx"

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.16.0
• Labels: gss sec
• Severity: 3
• Assignee: Sebastien Buisson
• Reporter: Sebastien Buisson

Description

There are some Lustre warnings

(gss_svc_upcall.c:962:gss_svc_upcall_get_ctx()) Invalid gss ctx idx 0xa6a894a730abbb50 from 192.168.0.192@o2ib

which seems to match on server side:

Lustre: 1439318:0:(sec_gss.c:685:gss_cli_ctx_handle_err_notify()) testfs-MDT0001: req x1797233949251264/t0, ctx 00000000137475b7 idx 0xfa7d3292507e467a(0->c): reverse server res>
Lustre: 1439318:0:(sec_gss.c:723:gss_cli_ctx_handle_err_notify()) testfs-MDT0001: client might have lost the context (NO_CONTEXT), retrying

This seems related to LDLM BL AST. It comes from expired GSS contexts on server side, which seem to be too numerous, and stay there for hours or even days. For instance:

- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715685010, delta: -7197, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0fe2:0xd24bbd2980dc2c0f", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715602210, delta: -89997, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0e6b:0xd24bbd2980dc27d5", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715516218, delta: -175989, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0d11:0xd24bbd2980dc243c", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715432277, delta: -259930, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0ba8:0xd24bbd2980dc1f65", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715347795, delta: -344412, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0a3d:0xd24bbd2980dc1a68", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715261655, delta: -430552, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a08d5:0xd24bbd2980dc15d0", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715685010, delta: -7197, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0f7a:0xd24bbd2980dc2c0c", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715602210, delta: -89997, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0df1:0xd24bbd2980dc27d2", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715516218, delta: -175989, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0ca8:0xd24bbd2980dc2439", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715432277, delta: -259930, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0b30:0xd24bbd2980dc1f61", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715347795, delta: -344412, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a09cc:0xd24bbd2980dc1a65", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715261655, delta: -430552, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a085d:0xd24bbd2980dc15cd", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715685010, delta: -7197, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0f00:0xd24bbd2980dc2c09", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715602210, delta: -89997, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0edf:0xd24bbd2980dc27d8", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715516345, delta: -175862, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0c22:0xd24bbd2980dc2421", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715431937, delta: -260270, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0ab6:0xd24bbd2980dc1e80", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715346043, delta: -346164, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a0950:0xd24bbd2980dc1a62", mech: "krb5 (aes256-cts-hmac-sha1-96)" }
- { peer_nid: 192.168.0.8@o2ib, uid: 0, ctxref: 1, expire: 1715261655, delta: -430552, flags: [uptodate, cached], seq: 0, win: 2048, key: 00000000, keyref: 0, hdl: "0xd28d8bea236a07e5:0xd24bbd2980dc15ca", mech: "krb5 (aes256-cts-hmac-sha1-96)" }

Comments

[Peter Jones]

This work all merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55127/
Subject: LU-17852 gss: do not use expired reverse gss contexts
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3f6cf9107d8a3325d6337593f872977555d82c9f

[Gerrit Updater]

"Sebastien Buisson <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55358
Subject: LU-17852 gss: improve kernel key usage
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 89f14a6ecf7ce6eada52fc47a5e87a69e7b96172

[Gerrit Updater]

"Sebastien Buisson <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55127
Subject: LU-17852 gss: purge expired reverse gss contexts
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 53d13b5751dbf4f8255059255bdbe874dc21bdd5