https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3550 Lustre <p>When attempting to mount NFS exported Lustre, the mount operation reports 'stale file handle' and fails to complete. This happens with 2.4 servers and a 2.4 client. It does NOT happen with a 2.4 client and 2.2 servers.</p> <p>Investigation of the NFS traffic between the NFS client and NFS server (Lustre client) shows the NFS client requesting the file handle for the mount, then receiving a file handle back from the server. There is a bit more chatter, then the client sends back the same file handle as part of an info request. Then the server responds with a stale file handle error.</p> <p>This is happening on both CentOS 6.4 and SLES11SP2 clients.</p> <p>I'm attaching a series of logs of this issue.<br/> Here's a description of what's in those logs:<br/> Lustre MDS (2.4). (Full DK logs provided)<br/> Lustre Client(2.4)/NFS Server <span class="error">&#91;The source of the NFS export&#93;</span> (Full DK logs &amp; /var/log/messages with nfsd debug on full (0x7FFF))<br/> NFS Client (/var/log/messages with nfs debug set to 1, and a tcpdump of all traffic)</p> <p>For analyzing the tcpdump (if you need it - I suspect the NFS debug logs will make it irrelevant), the IP addresses:<br/> NFS Server: 172.29.53.155<br/> NFS Client: 172.29.53.160</p> <p>The /var/log/messages logs are not trimmed, sorry. Look for the last debug markers from Lustre in those files and you can line them up with the rest of the logs.</p> LU-3550

Stale file handle on mount when mounting Lustre 2.4 via NFS

Bug Major Resolved Fixed nasf Patrick Farrell Tue, 2 Jul 2013 17:48:31 +0000 Wed, 20 Nov 2013 09:26:05 +0000 Mon, 18 Nov 2013 01:30:22 +0000 Lustre 2.4.0 Lustre 2.5.0 0 10 <p>The underlying issue is that NFS on Linux does not currently support 64 bit root inodes.</p> <p>This means that Lustre, with a 2.4 MDS, cannot be exported over NFS.</p> <p>After working this out by reading the NFS code, I discovered Andreas has a patch going upstream to the kernel for this:<br/> <a href="http://www.spinics.net/lists/linux-nfs/msg37384.html" class="external-link" target="_blank" rel="nofollow noopener">http://www.spinics.net/lists/linux-nfs/msg37384.html</a></p> <p>Is this documented somewhere as a known regression?<br/> Is there any plan to resolve this separately for Lustre clients?</p> <p>Excuse me - A closer look at the patch from Andreas suggests it's for a related issue but not exactly the one we're facing.</p> <p>The issue I'm looking at comes up in mk_fsid, called from fh_compose, which is called from exp_rootfh:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline void mk_fsid(<span class="code-object">int</span> vers, u32 *fsidv, dev_t dev, ino_t ino, u32 fsid, unsigned <span class="code-object">char</span> *uuid) { u32 *up; <span class="code-keyword">switch</span>(vers) { <span class="code-keyword">case</span> FSID_DEV: fsidv[0] = htonl((MAJOR(dev)&lt;&lt;16) | MINOR(dev)); fsidv[1] = ino_t_to_u32(ino); <span class="code-keyword">break</span>; } </pre> </div></div> <p>Where we see the inode being coerced to 32 bits. This is what goes out on the wire to the client, even though Lustre has 64 bit inodes.</p> <p>I will have to look more closely at Andreas's patch and the issue it's resolving, as well as the code I noted above, to understand fully what's going on.</p> <p>Andreas's patch is for an issue with parsing 64 bit inode numbers in NFS-utils, and so isn't involved here.</p> <p>The problem is this:<br/> In NFS, for mount requests, the root of the file system is identified by only the FSID. The FSID is defined as a 64 bit integer type in NFSv3 (See <a href="http://tools.ietf.org/html/rfc1813" class="external-link" target="_blank" rel="nofollow noopener">http://tools.ietf.org/html/rfc1813</a> for the NFSv3 RFC), but in the default case in Linux, it's built as two 32 bit integers, one of which is the inode. For NFSv4, it is two 64 bit types (<a href="http://www.ietf.org/rfc/rfc3530.txt" class="external-link" target="_blank" rel="nofollow noopener">http://www.ietf.org/rfc/rfc3530.txt</a>).</p> <p>Linux includes an option for a 64 bit inode type, in that case:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">case</span> FSID_UUID16_INUM: <span class="code-comment">/* 8 <span class="code-object">byte</span> inode and 16 <span class="code-object">byte</span> fsid */</span> *(u64*)fsidv = (u64)ino; memcpy(fsidv+2, uuid, 16); <span class="code-keyword">break</span>; </pre> </div></div> <p>&#8212;<br/> The vers value set here is a function of the export options and whether or not it's a root export.<br/> Here's the relevant code, from set_version_and_fsid_type:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (exp-&gt;ex_flags &amp; NFSEXP_FSID) { fsid_type = FSID_NUM; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (exp-&gt;ex_uuid) { <span class="code-keyword">if</span> (fhp-&gt;fh_maxsize &gt;= 64) { <span class="code-keyword">if</span> (is_root_export(exp)) fsid_type = FSID_UUID16; <span class="code-keyword">else</span> fsid_type = FSID_UUID16_INUM; } <span class="code-keyword">else</span> { <span class="code-keyword">if</span> (is_root_export(exp)) fsid_type = FSID_UUID8; <span class="code-keyword">else</span> fsid_type = FSID_UUID4_INUM; } } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (!old_valid_dev(exp_sb(exp)-&gt;s_dev)) <span class="code-comment">/* <span class="code-keyword">for</span> newer device numbers, we must use a newer fsid format */</span> fsid_type = FSID_ENCODE_DEV; <span class="code-keyword">else</span> fsid_type = FSID_DEV; </pre> </div></div> <p>The export option in question (ex_uuid) is one I can't quite figure out how to set in the export options.<br/> The code which parses the export options seems to be expecting the character sequence "uuid" (svc_export_parse), but when specifying -o uuid=<span class="error">&#91;some uuid&#93;</span>, I get an error.<br/> I can't find out how to actually set this flag.</p> <p>On the other hand, when I do -o fsid= on the export I can specify an integer or a UUID.<br/> This (either the integer or the UUID) allows me to mount the NFS export and do normal operations on it with NFSv4 or NFSv3 (at least client side. I will be testing with an NFSv3 only server today.).</p> <p>Presumably this is hitting this case in set_version_and_fsid:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">} <span class="code-keyword">else</span> <span class="code-keyword">if</span> (exp-&gt;ex_flags &amp; NFSEXP_FSID) { fsid_type = FSID_NUM; </pre> </div></div> <p>In any case, this appears to be a work around. Longer term, if we don't wish to have to specify -o fsid=, the NFS code in the kernel would need to change somehow to support 64 bit inodes in FSIDs.</p> <p>I've discussed this internally at Cray with someone with NFS expertise.</p> <p>He agrees that this work around (using the -o fsid= option to exportfs when exporting Lustre over NFS) is the appropriate solution, as the only other option is a fairly invasive patch to the NFS code in the Linux kernel. In light of that, WC may want to update documentation for exporting Lustre over NFS, but no code changes are necessary.</p> <p>There are two issues for this topic:</p> <p>1) Originally, Lustre did not return FSID via statfs() to nfs-utils. This issue has been resolved by the patch <a href="http://review.whamcloud.com/6493" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6493</a>, which has already been landed to master (Lustre-2.5)</p> <p>2) The nfs-utils defect of converting 64-bits ino# into 32-bits and causes information lost as to cannot locate the right root. It can be resolved by the patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/utils/mountd/cache.c b/utils/mountd/cache.c index 517aa62..a7212e7 100644 --- a/utils/mountd/cache.c +++ b/utils/mountd/cache.c @@ -388,7 +388,7 @@ struct parsed_fsid { <span class="code-object">int</span> fsidtype; /* We could use a union <span class="code-keyword">for</span> <span class="code-keyword">this</span>, but it would be more * complicated; why bother? */ - unsigned <span class="code-object">int</span> inode; + uint64_t inode; unsigned <span class="code-object">int</span> minor; unsigned <span class="code-object">int</span> major; unsigned <span class="code-object">int</span> fsidnum; -- 1.7.1 </pre> </div></div> <p>If you have chance, you can test above two patches together for verification. Thanks!</p> <p>nasf,</p> <p>I've been trying to build nfs-utils 1.2.3 <span class="error">&#91;default in CentOS 6.4&#93;</span> (without patches, just to verify I can) and I am stuck in a dependency hell, with it not finding various installed packages. A bit of searching shows that patching has been done to nfs-utils to clean up a lot of unnecessary dependencies, which include the ones I'm dealing with.<br/> (<a href="http://www.spinics.net/lists/linux-nfs/msg26388.html" class="external-link" target="_blank" rel="nofollow noopener">http://www.spinics.net/lists/linux-nfs/msg26388.html</a>)</p> <p>However, as I understand it, the kernel nfsd /proc interface has changed since CentOS 6.4 and SLES11SP2, so I can't just go grab the latest nfs-utils and expect it to work.</p> <p>Do you have a particular version you recommend building, or any tips on this?</p> <p>I may be able to land that linking patch by itself without problem and will try that next, but I thought I'd ask you as well.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Above patch is for the latest nfs-utils. If you want to use nfs-utils-1.2.3, then the following one:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">343,344c343,344 &lt; uint64_t inode=0; &lt; uint64_t inode64; --- &gt; unsigned <span class="code-object">int</span> inode=0; &gt; unsigned <span class="code-object">long</span> <span class="code-object">long</span> inode64; </pre> </div></div> <p>nasf,</p> <p>Has WC tested the latest nfs-utils with CentOS 6.4? I thought I saw a proc interface change between the CentOS 6.4 kernel and the kernels targeted by 1.2.8, but I could be wrong about that.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Hi Patrick,</p> <p>I downloaded the nfs-untils-1.2.3 source, and patched/compiled/tested on RHEL6 (2.6.32-358.6.1.el6). Not care the proc changes.</p> <p>I'm not sure what the long term plan is regarding this bug. The fundamental limitation isn't in Lustre, and we've got an acceptable workaround with setting FSID manually.</p> <p>Is further work planned on the Intel side, or should this bug be closed? Cray is getting along fine with the work around.</p> <p>We have submitted related patch to the kernel maintainer, and hope the issue can be resolved from root. From Intel side, we cannot do more but waiting for the respond. If you have got your things work, we can close this ticket, and reopen it in future when needed.</p> Related LU-2904 LU-4057 Development Rank 1|hzvujr: Rank (Obsolete) 8928 Severity