https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13739 Lustre <p>Client mount fails with the below error message. This is with "lustre02.srpc.flavor.tcp.cli2mdt=skpi" and "lustre02.srpc.flavor.tcp.cli2ost=skpi"</p> <p>The server and clients have the correct SSK keys and keyring shows them as loaded. </p> <p>Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(lmv_obd.c:315:lmv_connect_mdc()) target lustre02-MDT0000_UUID connect error -1<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(lmv_obd.c:315:lmv_connect_mdc()) Skipped 1 previous similar message<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(llite_lib.c:292:client_common_fill_super()) cannot connect to lustre02-clilmv-ffff8964b4dea800: rc = -1<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(llite_lib.c:292:client_common_fill_super()) Skipped 1 previous similar message<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(lov_obd.c:839:lov_cleanup()) lustre02-clilov-ffff8964b4dea800: lov tgt 0 not cleaned! deathrow=0, lovrc=1<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(lov_obd.c:839:lov_cleanup()) Skipped 13 previous similar messages<br/> Jul 1 17:11:54 zabbix01 kernel: LustreError: 22972:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-1)</p> RHEL7 LU-13739

mount fails with SSK keys

Bug Major Closed Duplicate Jeremy Filizetti Joe Frith Wed, 1 Jul 2020 21:16:48 +0000 Wed, 8 Jul 2020 20:14:22 +0000 Wed, 8 Jul 2020 20:14:22 +0000 Lustre 2.12.5 0 2 <p>On the client  </p> <p> </p> <p><span class="error">&#91;root@ ~&#93;</span># keyctl show<br/> Session Keyring<br/> 422481900 --alswrv 0 0 keyring: _ses<br/> 51709085 --alswrv 0 65534 &#95; keyring: _uid.0<br/> 180533941 --alswrv 0 0 &#95; user: lustre:lustre02<br/> <span class="error">&#91;root@ ~&#93;</span># keyctl pipe 180533941 | lgss_sk -r -<br/> Version: 1<br/> Type: client<br/> HMAC alg: sha256<br/> Crypto alg: ctr(aes)<br/> Ctx Expiration: 604800 seconds<br/> Shared keylen: 256 bits<br/> Prime length: 2048 bits<br/> File system: lustre02<br/> MGS NIDs:<br/> Nodemap name: default<br/> Shared key:<br/> 0000: 5900 0856 8af8 5fbc 2549 168f 8a29 56a8 Y..V.._.%I...)V.<br/> 0010: 858e 3bd9 5fd1 23af 780f b92b 3bcc c406 ..;._.#.x..+;...<br/> Prime (p):<br/> 0000: f470 ae37 4a92 24bd d442 9456 2cf8 9cbd .p.7J.$..B.V,...<br/> 0010: 4dc6 400c 76b8 9edc c823 18e5 86f9 a0ba M.@.v....#......<br/> 0020: 70e2 72ba f5bf 4320 0386 a047 e772 2567 p.r...C ...G.r%g<br/> 0030: 8b65 10ef 758d e8a3 0441 bc0c 8b36 2be9 .e..u....A...6+.<br/> 0040: c38f cbdc 20ea 7461 890e c59b c948 f964 .... .ta.....H.d<br/> 0050: 8dd8 3891 4947 cb34 93d9 4150 1f4a 7eae ..8.IG.4..AP.J~.<br/> 0060: 65c1 d7b4 6e3b 274a 753f d0af 242a 8e10 e...n;'Ju?..$*..<br/> 0070: 9055 3ad5 b195 856c c7b8 b9f0 2b34 666a .U:....l....+4fj<br/> 0080: e8fc 2988 5f77 ced9 cbc0 2911 179b c1d9 ..)._w....).....<br/> 0090: 0717 e8d7 a14c 14f6 4907 fa0c 3de9 fffa .....L..I...=...<br/> 00a0: f524 623c 6664 fa20 4246 c1f3 1c06 27cc .$b&lt;fd. BF....'.<br/> 00b0: ea75 7d10 8804 3489 88fd 392f 5c89 284b .u}...4...9/\.(K<br/> 00c0: 0aee 6df5 5471 95a7 6e1d 669c 658f e848 ..m.Tq..n.f.e..H<br/> 00d0: f74b 15be 4a1e fbc1 8fcc 78ed 87c5 1abe .K..J.....x.....<br/> 00e0: 028a 66cd e230 d6f9 8e8b f9e9 9cfe 6013 ..f..0........`.<br/> 00f0: c6d5 fcae b2be d59f 9375 8beb 8564 ad63 .........u...d.c</p> <p> </p> <p>On the server side -</p> <p> <span class="error">&#91;root@ ~&#93;</span># keyctl show<br/> Session Keyring<br/> 13605391 --alswrv 0 0 keyring: _ses<br/> 861388744 --alswrv 0 65534 &#95; keyring: _uid.0<br/> 77605995 --alswrv 0 0 &#95; user: lustre:lustre02:default<br/> <span class="error">&#91;root@ ~&#93;</span># keyctl pipe 77605995 | lgss_sk -r -<br/> Version: 1<br/> Type: server<br/> HMAC alg: sha256<br/> Crypto alg: ctr(aes)<br/> Ctx Expiration: 604800 seconds<br/> Shared keylen: 256 bits<br/> Prime length: 2048 bits<br/> File system: lustre02<br/> MGS NIDs:<br/> Nodemap name: default<br/> Shared key:<br/> 0000: 5900 0856 8af8 5fbc 2549 168f 8a29 56a8 Y..V.._.%I...)V.<br/> 0010: 858e 3bd9 5fd1 23af 780f b92b 3bcc c406 ..;._.#.x..+;...<br/> <span class="error">&#91;root@ ~&#93;</span>#</p> <p>Do you have the MDS server logs as well?</p> <p>I see this on the MDS server. </p> <p> </p> <p>kernel: LustreError: 233128:0:(tgt_handler.c:921:tgt_connect_check_sptlrpc()) lustre02-MDT0000: unauthorized rpc flavor 0 from x.x.x.x@tcp, expect 22</p> <p>This seems to indicate the client is sending the wrong RPC flavor SPTLRPC_POLICY_NULL.  For some reason your client isn't pulling the correct info out of the MGS llog.  What does "lctl get_param mgs.MGS.live.*" say?</p> <p><span class="error">&#91;root@mds ~&#93;</span># lctl get_param mgs.MGS.live.*<br/> mgs.MGS.live.lustre02=<br/> fsname: lustre02<br/> flags: 0x20 gen: 58<br/> lustre02-MDT0000<br/> lustre02-OST0000<br/> lustre02-OST0001<br/> lustre02-OST0002<br/> lustre02-OST0003<br/> lustre02-OST0004<br/> lustre02-OST0005<br/> lustre02-OST0006</p> <p>Secure RPC Config Rules:<br/> lustre02.srpc.flavor.tcp.cli2mdt=skpi</p> <p>imperative_recovery_state:<br/> state: full<br/> nonir_clients: 0<br/> nidtbl_version: 37<br/> notify_duration_total: 0.000238135<br/> notify_duation_max: 0.000238135<br/> notify_count: 1<br/> mgs.MGS.live.params=<br/> fsname: params<br/> flags: 0x20 gen: 1</p> <p>Secure RPC Config Rules:</p> <p>imperative_recovery_state:<br/> state: full<br/> nonir_clients: 0<br/> nidtbl_version: 2<br/> notify_duration_total: 0.000000000<br/> notify_duation_max: 0.000000000<br/> notify_count: 0<br/> <span class="error">&#91;root@mds ~&#93;</span>#</p> <p>ptlrpc_gss kernel module was not loaded on the client. After fixing this issue, the mount still hangs, I am seeing the below in the client logs, nothing on the server side. </p> <p> </p> <p>Jul 6 22:06:31 zabbix01 kernel: Lustre: 22730:0:(gss_svc_upcall.c:1149:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active<br/> Jul 6 22:06:31 zabbix01 kernel: Key type lgssc registered<br/> Jul 6 22:06:42 zabbix01 kernel: Lustre: 22734:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1594087591/real 1594087591&#93;</span> req@ffff8af7b3270000 x1671521988773568/t0(0) o801-&gt;lustre02-MDT0000-mdc-ffff8af635af1000@10.42.41.33@tcp:12/10 lens 224/224 e 0 to 1 dl 1594087602 ref 2 fl Rpc:X/0/ffffffff rc 0/-1<br/> Jul 6 22:06:42 zabbix01 kernel: LustreError: 22734:0:(gss_keyring.c:1409:gss_kt_update()) negotiation: rpc err -85, gss err 0<br/> Jul 6 22:06:42 zabbix01 lgss_keyring: <span class="error">&#91;22734&#93;</span>:ERROR:do_nego_rpc(): status: -110 (Connection timed out)<br/> Jul 6 22:06:42 zabbix01 lgss_keyring: <span class="error">&#91;22734&#93;</span>:ERROR:lgssc_negotiation_manual(): negotiation rpc error -85<br/> Jul 6 22:06:42 zabbix01 lgss_keyring: <span class="error">&#91;22734&#93;</span>:ERROR:lgssc_kr_negotiate_manual(): key 18d782c4: failed to negotiate<br/> Jul 6 22:06:42 zabbix01 kernel: Lustre: 22734:0:(sec_gss.c:315:cli_ctx_expire()) ctx ffff8af7b4d19d00(0-&gt;lustre02-MDT0000_UUID) get expired: 1594087791(+189s)</p> <p>Can you confirm that lsvcgss service is running on the server side and that you have the "-s" included in /etc/sysconfig/lsvcgss?</p> <p><span class="error">&#91;root@starmds01 ~&#93;</span># cat /etc/sysconfig/lsvcgss</p> <ol> <li>Optional arguments passed to lsvcgssd.<br/> LSVCGSSDARGS='-s -m'</li> </ol> <p><span class="error">&#91;root@starmds01 ~&#93;</span># systemctl status lsvcgss.service<br/> ● lsvcgss.service - SYSV: start and stop the lsvcgssd daemon<br/> Loaded: loaded (/etc/rc.d/init.d/lsvcgss; bad; vendor preset: disabled)<br/> Active: active (running) since Mon 2020-07-06 22:53:25 EDT; 18h ago<br/> Docs: man:systemd-sysv-generator(8)<br/> Process: 204274 ExecStart=/etc/rc.d/init.d/lsvcgss start (code=exited, status=0/SUCCESS)<br/> CGroup: /system.slice/lsvcgss.service<br/> └─204278 /usr/sbin/lsvcgssd -s -m</p> <p>Jul 06 22:53:25 starmds01 systemd<span class="error">&#91;1&#93;</span>: Starting SYSV: start and stop the lsvcgssd daemon...<br/> Jul 06 22:53:25 starmds01 lsvcgss<span class="error">&#91;204274&#93;</span>: Starting lsvcgssd<br/> Jul 06 22:53:25 starmds01 systemd<span class="error">&#91;1&#93;</span>: Started SYSV: start and stop the lsvcgssd daemon.<br/> <span class="error">&#91;root@starmds01 ~&#93;</span>#</p> <p>Can you add a "-vv" as well to the LSVCGSSDARGS and then send the output from the syslog for the MDS.  If possible it'd be nice to have the output of "lctl dk" as well.</p> <p>lctl dk shows below when the client is connecting. </p> <p> </p> <p>00010000:00080000:2.0F:1594173014.717485:0:235886:0:(ldlm_lib.c:1227:target_handle_connect()) MGS: connection from 17390dcc-e9ea-2781-83dc-708eda5b1b60@130.199.148.189@tcp t0 exp (null) cur 539645 last 0<br/> 00000020:00000080:2.0:1594173014.717501:0:235886:0:(genops.c:1417:class_connect()) connect: client 17390dcc-e9ea-2781-83dc-708eda5b1b60, cookie 0x9c7d9c60f6ed33e9<br/> 00000020:01000000:2.0:1594173014.717507:0:235886:0:(lprocfs_status_server.c:491:lprocfs_exp_setup()) using hash ffff99b7cfe94c00<br/> 00000100:00080000:2.0:1594173014.717523:0:235886:0:(import.c:86:import_set_state_nolock()) ffff99e7d97b6000 : changing import state from RECOVER to FULL<br/> 00000100:02000000:2.0:1594173014.717527:0:235886:0:(import.c:1597:ptlrpc_import_recovery_state_machine()) MGS: Connection restored to e3959aaf-45a8-6dda-c117-ced5724d4e0b (at 130.199.148.189@tcp)<br/> 20000000:01000000:2.0:1594173014.776757:0:235886:0:(mgs_nids.c:632:mgs_get_ir_logs()) Reading IR log lustre02-cliir bufsize 1048576.<br/> 20000000:01000000:2.0:1594173014.776763:0:235886:0:(mgs_nids.c:192:mgs_nidtbl_read()) fsname lustre02, entry size 32, pages 4064/1/256/255.<br/> 20000000:01000000:2.0:1594173014.776766:0:235886:0:(mgs_nids.c:204:mgs_nidtbl_read()) Read IR logs lustre02 return with 32, version 64</p> <p> </p> <p>Even with the -vv option added, I see only the below line on the MDS syslog. </p> <p> </p> <p>Jul 7 21:49:20 starmds01 kernel: Lustre: MGS: Connection restored to e3959aaf-45a8-6dda-c117-ced5724d4e0b (at 130.199.148.189@tcp)<br/> Jul 7 21:49:20 starmds01 kernel: Lustre: Skipped 14 previous similar messages</p> <p>It looks like I can reproduce this with Centos 7.8  is that version you are using?  If you add security to the debug on the MDS do you see something like the following?</p> <p><tt><span class="error">&#91;root@tr-mds-1 ~&#93;</span># echo '+sec' &gt; /sys/kernel/debug/lnet/debug</tt><br/> <tt><span class="error">&#91;root@tr-mds-1 ~&#93;</span># lctl dk | grep gss</tt><br/> <tt>02000000:08000000:5.0:1594175878.715795:0:31012:0:(sec_gss.c:1991:gss_svc_handle_init()) processing gss init(1) request from 172.26.0.250@tcp</tt><br/> <tt>02000000:08000000:5.0:1594175878.715810:0:31012:0:(gss_svc_upcall.c:960:gss_svc_upcall_handle_init()) cache_check return ENOENT, drop</tt></p> <p>Yes, see below. I am using RHEL 7.8. Is there a workaround or quick fix?</p> <p> </p> <p><span class="error">&#91;root@starmds01 ~&#93;</span># lctl dk | grep gss<br/> 02000000:08000000:22.0F:1594176204.563936:0:236049:0:(sec_gss.c:1991:gss_svc_handle_init()) processing gss init(1) request from 130.199.148.189@tcp<br/> 02000000:08000000:22.0:1594176204.563953:0:236049:0:(gss_svc_upcall.c:960:gss_svc_upcall_handle_init()) cache_check return ENOENT, drop<br/> 02000000:08000000:8.0F:1594176215.608251:0:235969:0:(sec_gss.c:1991:gss_svc_handle_init()) processing gss init(1) request from 130.199.148.189@tcp<br/> 02000000:08000000:8.0:1594176215.608265:0:235969:0:(gss_svc_upcall.c:960:gss_svc_upcall_handle_init()) cache_check return ENOENT, drop<br/> <span class="error">&#91;root@starmds01 ~&#93;</span>#</p> <p>There was some additional changes to the sunrpc code incorporated into RHEL 7.8 from upstream that assumes the files for the cache get closed repeatedly. Since Lustre's lsvcgssd just opens the file one and keeps it open a last_close variable in the cache_detail struct (rsi_cache) is not being updated so this is why we are seeing the issue. I'll see if at some point today I can take a look at how the NFS stuff works to make sure my assumptions aren't wrong here on how it uses the cache and then create a patch for Lustre. </p> <p>There is a patch that Sebastien has already posted for <a href="https://jira.whamcloud.com/browse/LU-13754" title="GSS-based authentication fails on CentOS/RHEL 7.8" class="issue-link" data-issue-key="LU-13754"><del>LU-13754</del></a> that should work. I thought there was additional patches needed but it was just something messed up in my test environment. If you have any issues after testing the patch please file it under the other bug.</p> Related LU-13754 Development Rank 1|i0147r: Rank (Obsolete) 9223372036854775807 Severity