https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10593 Lustre <p>This is an IB specific issue, has been seen on 2.10.1 and master branch. Test with same config on TCP pass.</p> <p>This is what I got with IB network when trying to mount client. I have tried with different flavors (skpi, ski, gss null) and all failed with same error. The patch <a href="https://review.whamcloud.com/30937" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30937</a><br/> doesn't help with this issue.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[104174.396583] LustreError: 48209:0:(gss_keyring.c:1426:gss_kt_update()) negotiation: rpc err -13, gss err 0 [104174.410110] LustreError: 48209:0:(gss_keyring.c:1426:gss_kt_update()) Skipped 4 previous similar messages [104174.423357] Lustre: 48209:0:(sec_gss.c:316:cli_ctx_expire()) ctx ffff880824f1e540(0-&gt;lustre-MDT0000_UUID) get expired: 1517010602(+200s) [104174.441414] Lustre: 48209:0:(sec_gss.c:316:cli_ctx_expire()) Skipped 4 previous similar messages [104199.411600] Lustre: 45094:0:(sec_gss.c:1226:gss_cli_ctx_fini_common()) gss.keyring@ffff8807f2a62400: destroy ctx ffff88105aa0dec0(0-&gt;lustre-MDT0000_UUID) [104199.432714] Lustre: 45094:0:(sec_gss.c:1226:gss_cli_ctx_fini_common()) Skipped 5 previous similar messages [104249.405563] LustreError: 48222:0:(gss_keyring.c:1426:gss_kt_update()) negotiation: rpc err -13, gss err 0 [104249.418953] LustreError: 48222:0:(gss_keyring.c:1426:gss_kt_update()) Skipped 2 previous similar messages [104249.432234] Lustre: 48222:0:(sec_gss.c:316:cli_ctx_expire()) ctx ffff88105aa0c180(0-&gt;lustre-MDT0000_UUID) get expired: 1517010677(+200s) [104249.450371] Lustre: 48222:0:(sec_gss.c:316:cli_ctx_expire()) Skipped 2 previous similar messages [104299.408796] Lustre: 45094:0:(sec_gss.c:1226:gss_cli_ctx_fini_common()) gss.keyring@ffff8807f2a62400: destroy ctx ffff88105aa0cf00(0-&gt;lustre-MDT0000_UUID) [104299.429934] Lustre: 45094:0:(sec_gss.c:1226:gss_cli_ctx_fini_common()) Skipped 3 previous similar messages </pre> </div></div> LU-10593

Cannot mount client if SSK is setup over IB network

Bug Minor Open Unresolved James A Simmons Sarah Liu Wed, 31 Jan 2018 19:55:56 +0000 Fri, 15 Feb 2019 14:32:20 +0000 Lustre 2.11.0 1 9 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> and <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sbuisson" class="user-hover" rel="sbuisson">sbuisson</a> is this a known issue?</p> <p>This is exactly my problem I see.</p> <p>Hi Sarah, James,</p> <p>I am trying to setup an IB-based test cluster to investigate this issue. In the meantime, do you have the opportunity to try again with debug activated on all nodes?</p> <ul class="alternate" type="square"> <li>lsvcgssd daemon launched with '-vv' flag;</li> <li>request-key debug: move /usr/sbin/request-key to /usr/sbin/request-key.orig, and create a shell script named /usr/sbin/request-key with following content: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#!/bin/bash /usr/sbin/request-key.orig -d $* </pre> </div></div></li> <li>lctl set_param sptlrpc.gss.lgss_keyring.debug_level=4</li> <li>lctl set_param debug=+sec</li> </ul> <p>Some debug info will be available in dmesg and lustre debug logs, but for userspace debug info you will have to use journalctl.</p> <p>Thanks,<br/> Sebastien.</p> <p>client console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1060.602386] LustreError: 4000:0:(gss_keyring.c:849:gss_sec_lookup_ctx_kr()) failed request key: -13 [ 1060.612583] LustreError: 4000:0:(sec.c:448:sptlrpc_req_get_ctx()) req ffff880842b90300: fail to get context [ 1060.623549] LustreError: 4000:0:(lmv_obd.c:319:lmv_connect_mdc()) target lustre-MDT0000_UUID connect error -111 [ 1060.634916] LustreError: 4000:0:(llite_lib.c:284:client_common_fill_super()) cannot connect to lustre-clilmv-ffff8801738d6000: rc = -111 [ 1060.741909] LustreError: 4000:0:(lov_obd.c:876:lov_cleanup()) lustre-clilov-ffff8801738d6000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1060.757081] Lustre: Unmounted lustre-client [ 1060.762761] LustreError: 4000:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-111) mount.lustre: mount onyx-80-ib@o2ib:/lustre at /mnt/lustre failed: Connection refused [root@onyx-79 ~]# </pre> </div></div> <p>MDS dmesg. MDS debug please see attached</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 742.528583] alg: No test for adler32 (adler32-zlib) [ 742.534137] alg: No test for crc32 (crc32-table) [ 743.298513] Lustre: Lustre: Build Version: 2.10.57_57_g98ddc99 [ 743.369060] LNet: Using FMR for registration [ 743.404231] LNet: Added LNI 192.168.1.80@o2ib [8/256/0/180] [ 743.413664] LNet: Added LNI 10.2.2.52@tcp [8/256/0/180] [ 743.419594] LNet: Accept secure, port 988 [ 743.519797] LDISKFS-fs (sdb1): recovery complete [ 743.530730] LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.740193] Lustre: MGS: Connection restored to 9eed7cd8-6b21-4821-b7ce-181f90e7c677 (at 0@lo) [ 745.368317] LNet: 3884:0:(o2iblnd_cb.c:3294:kiblnd_check_conns()) Timed out tx for 192.168.1.82@o2ib: 4295411 seconds [ 745.406187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 300-900 [ 758.369573] LNet: 3884:0:(o2iblnd_cb.c:3294:kiblnd_check_conns()) Timed out tx for 192.168.1.82@o2ib: 4295424 seconds [ 770.387018] Lustre: lustre-MDT0000: Connection restored to 9eed7cd8-6b21-4821-b7ce-181f90e7c677 (at 0@lo) [ 771.370844] LNet: 3884:0:(o2iblnd_cb.c:3294:kiblnd_check_conns()) Timed out tx for 192.168.1.82@o2ib: 4295437 seconds [ 796.373327] LNet: 3884:0:(o2iblnd_cb.c:3294:kiblnd_check_conns()) Timed out tx for 192.168.1.82@o2ib: 4295462 seconds [ 821.375835] LNet: 3884:0:(o2iblnd_cb.c:3294:kiblnd_check_conns()) Timed out tx for 192.168.1.82@o2ib: 4295487 seconds [ 827.771270] Lustre: MGS: Connection restored to d1e5edfc-64c0-4708-b7f3-ab19e20a579e (at 192.168.1.82@o2ib) [ 835.247329] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.1.82@o2ib (at 192.168.1.82@o2ib) [ 835.258711] Lustre: Skipped 1 previous similar message [ 1039.470883] Lustre: MGS: Connection restored to 3bb9b7f5-bfdf-013f-4c84-dc859d55a7ec (at 192.168.1.79@o2ib) [ 1064.301828] Lustre: MGS: Connection restored to 3bb9b7f5-bfdf-013f-4c84-dc859d55a7ec (at 192.168.1.79@o2ib) [root@onyx-80 ~]# </pre> </div></div> <p>Did you add the debugging the Sebastien asked? Without that information from the client it will be hard to see what is wrong since it appears to be during the request-key portion. </p> <p>I have ran into an issue where the server side lsvcgss loses access to the key for some reason. This may require me to rework the key handling so that keys remain associated with lustre processes. However, this returns a GSS error to the client not an RPC error so it's a different issue.</p> <p>True Jeremy <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Hopefully I managed to reproduce the issue on a test system at DDN (negotiation: rpc err -13, gss err 0). I figured out how to make it work with Kerberos, but I guess it would be the same with Shared Key.</p> <p>The issue stems from the fact that context negotiation process needs to perform name resolution at some point:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lgssc_kr_negotiate_{krb,manual} lgss_get_service_str ipv4_nid2hostname lnet_nid2hostname </pre> </div></div> <p>So in order to make it work, your Lustre nodes' NIDs on IB must have an associated hostname that can be resolved by each other node. With Kerberos, the credentials must be created for these IB-based hostnames.</p> <p>It explains why it works out of the box when using a TCP based interconnect network. NIDs naturally match nodes' hostnames <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Do you agree to close this ticket with 'configuration issue' as the reason?</p> <p>Cheers,<br/> Sebastien.</p> <p>No I would consider this a real bug. Not all interconnects have IP address. Consider the Cray Gemini interconnects. Also discussion is under way about using the IB hardware address instead of an IP address as a lookup in the lnet layer. That change will then make IB totally unusable. We do need a proper solution.</p> <p>James,</p> <p>I would tend to consider what you suggest as an enhancement or feature request. Even in the case of using the IB hardware address instead of an IP address as a lookup in the lnet layer, Kerberos credentials must be associated to nodes. So probably that would mean adapting the way name resolution is carried out today, once this change for IB hardware address is done...</p> <p>Sebastien.</p> <p>Sebastien,</p> <p>1. the above logs was with debug enabled.<br/> 2. I used hostname when mount</p> <p>on MDS ping client. and vice versa</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-80 ~]# lctl ping onyx-79-ib@o2ib This command has been deprecated. Plesae use 'lnetctl ping' 12345-0@lo 12345-192.168.1.79@o2ib 12345-10.2.2.51@tcp [root@onyx-80 ~]# [root@onyx-79 ~]# lctl ping onyx-80-ib@o2ib This command has been deprecated. Plesae use 'lnetctl ping' 12345-0@lo 12345-192.168.1.80@o2ib 12345-10.2.2.52@tcp [root@onyx-79 ~] </pre> </div></div> <p>&gt; Not all interconnects have IP addresses.</p> <p>For this kind of interconnect there is already an existing mechanism in Lustre. You can put a script named /etc/lustre/nid2hostname on all your nodes, which takes 3 parameters:</p> <ul> <li>$lnd is a string identifying the LND, like "PTL4LND"</li> <li>$netid is the network identifier in hex format, like "0x12"</li> <li>$nid is the NID in hex format<br/> The script is supposed to output the corresponding hostname, or an error message starting with '@' for error logging.</li> </ul> <p>Note that at the moment this script is only called in the case of a PTL4LND interconnect type, as QSWLND or GMLND for instance were deprecated by patch <a href="https://review.whamcloud.com/23621" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23621</a> some time ago.</p> <p>Does SSK function available on Lustre 2.12 or 2.10.6  IB network ?</p> <p>I have test Lustre 2.12 or 2.10.6 (IB network) also got the same error.</p> <p>[ 877.959783] LustreError: 14203:0:(gss_keyring.c:1423:gss_kt_update()) negotiation: rpc err -13, gss err 0<br/> [ 877.959878] Lustre: 14203:0:(sec_gss.c:315:cli_ctx_expire()) ctx ffff8b207acc3200(0-&gt;testfs-MDT0000_UUID) get expired: 1549873937(+200s)<br/> [ 877.968204] Lustre: 13621:0:(sec_gss.c:1225:gss_cli_ctx_fini_common()) gss.keyring@ffff8b207c897300: destroy ctx ffff8b2077e8fec0(0-&gt;testfs-MDT0000_UUID)<br/> [ 877.968213] Lustre: 13621:0:(sec_gss.c:1225:gss_cli_ctx_fini_common()) Skipped 1 previous similar message<br/> [ 978.206863] LustreError: 14303:0:(gss_keyring.c:1423:gss_kt_update()) negotiation: rpc err -13, gss err 0</p> <p> </p> <p>The problem Sebastien pointed out with hostname resolution is probably your issue.  You will need to resolve the reverse lookup for your IB addresses.</p> <p> </p> <p>You can increase debugging on the client with:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@r01svr1 ~]# echo 3 &gt; /proc/fs/lustre/sptlrpc/gss/lgss_keyring/debug_level </pre> </div></div> <p> </p> <p>And look in you logs for something similar to:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Feb 12 14:48:45 r01svr1 lgss_keyring: [22410]:INFO:main(): key 441386067, desc 0@e, ugid 0:0, sring 279671070, coinfo 14:sk:0:0:r:n:1:0x500000a0a0a13:SiteA2-MDT0000-mdc-ffff964c12745800:0x500000a0a0601:1 Feb 12 14:48:45 r01svr1 lgss_keyring: [22410]:DEBUG:parse_callout_info(): parse call out info: secid 14, mech sk, ugid 0:0, is_root 1, is_mdt 0, is_ost 0, svc type n, svc 1, nid 0x500000a0a0a13, tgt SiteA2-MDT0000-mdc-ffff964c12745800, self nid 0x500000a0a0601, pid 1 Feb 12 14:48:45 r01svr1 lgss_keyring: [22410]:INFO:sk_create_cred(): Creating credentials <span class="code-keyword">for</span> target: SiteA2-MDT0000-mdc-ffff964c12745800 with nodemap: (<span class="code-keyword">null</span>) Feb 12 14:48:45 r01svr1 lgss_keyring: [22410]:INFO:sk_create_cred(): Searching <span class="code-keyword">for</span> key with description: lustre:SiteA2 Feb 12 14:48:45 r01svr1 lgss_keyring: [22411]:ERROR:ipv4_nid2hostname(): O2IBLND: can't resolve 0x130a0a0a Feb 12 14:48:45 r01svr1 lgss_keyring: [22411]:ERROR:lgss_get_service_str(): cannot resolve hostname from nid 500000a0a0a13 Feb 12 14:48:45 r01svr1 lgss_keyring: [22411]:ERROR:lgssc_kr_negotiate_manual(): key 1a4f0453: failed to construct service string</pre> </div></div> <p> </p> <p> </p> <p>The log <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31984/31984_ssk_20190213.log" title="ssk_20190213.log attached to LU-10593">ssk_20190213.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>seems got the same hostname resolution issue.</p> <p> </p> <p>You will need to resolve the reverse lookup for your IB addresses.</p> <p>=&gt;What should I  do anything for resolve  IB addresses ?</p> <p> </p> <p>Just make sure you when you do an "nslookup &lt;ipoib address&gt;" it resolves to a hostname and things should progress beyond that issue.</p> Related Development Rank 1|hzzrzj: Rank (Obsolete) 9223372036854775807 Severity