https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12209 Lustre <p>I'm facing a new issue on Oak (2.10.7 servers), tried with both 2.10 and 2.12 clients:</p> <p>As root:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cd /oak/stanford/groups/ # lfs mkdir -i 1 caiwei lfs mkdir: dirstripe error on 'caiwei': Stale file handle lfs setdirstripe: cannot create stripe dir 'caiwei': Stale file handle # lfs getdirstripe . lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none </pre> </div></div> <p>does that ring a bell? Oak is only using DNE v1 with statically striped directories. Never seen that before 2.10.7 (we recently upgraded Oak).</p> <p>a basic lctl dk doesn't show anything on the MDS but I may have to enable specific debug flags to see more. No other traces found so far.</p> <p>Tried with 2.10 and 2.12 clients, with or without idmap.</p> <p>Thanks!</p> CentOS 7.6, servers 2.10.7, clients 2.12 or 2.10 LU-12209

cannot create stripe dir: Stale file handle

Bug Critical Resolved Duplicate Lai Siyao Stephane Thiell Fri, 19 Apr 2019 15:26:54 +0000 Sun, 21 Apr 2019 13:38:17 +0000 Sat, 20 Apr 2019 14:04:15 +0000 Lustre 2.10.7 0 4 <p>Note that MDT0001 is still working fine within already-created directories:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@oak-rbh01 giocomo]# pwd /oak/stanford/groups/giocomo [root@oak-rbh01 giocomo]# lfs getdirstripe . lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none [root@oak-rbh01 giocomo]# mkdir .testdir [root@oak-rbh01 giocomo]# lfs mkdir -i 1 .testdir2 [root@oak-rbh01 giocomo]# lfs getdirstripe .testdir* lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none [root@oak-rbh01 giocomo]# rmdir .testdir* [root@oak-rbh01 giocomo]# </pre> </div></div> <p>If it is this repeatable, can you get -1 debug on the client and the server?  I know that may be a pain server side, but if possible it would be great.</p> <p>Hi Patrick,</p> <p>Ok, I will try (should be in an hour, I'm on my way to the office). But it looks like it is repeatable but only if doing lfs mkdir -i 1 in a parent directory striped on MDT0. See below.</p> <p>Creating a directory on MDT0 in a parent dir in MDT1 does work:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@oak-rbh01 giocomo]# lfs getdirstripe . lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none [root@oak-rbh01 giocomo]# lfs mkdir -i 0 .testdir_mdt0 [root@oak-rbh01 giocomo]# lfs getdirstripe .testdir_mdt0 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@oak-rbh01 giocomo]# rmdir .testdir_mdt0 </pre> </div></div> <p>But not the other way around:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@oak-rbh01 giocomo]# cd ../ruthm [root@oak-rbh01 ruthm]# lfs getdirstripe . lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@oak-rbh01 ruthm]# lfs mkdir -i 0 .testdir_mdt0 [root@oak-rbh01 ruthm]# [root@oak-rbh01 ruthm]# lfs mkdir -i 1 .testdir_mdt1 error on LL_IOC_LMV_SETSTRIPE '.testdir_mdt1' (3): Stale file handle error: mkdir: create stripe dir '.testdir_mdt1' failed </pre> </div></div> <p>This is done.<br/>  <br/> Command issued on client sh-101-60 (10.9.101.60@o2ib4) running 2.12 was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sh-101-60 ruthm]# lctl clear [root@sh-101-60 ruthm]# lfs mkdir -i 1 .testdir_mdt1 lfs mkdir: dirstripe error on '.testdir_mdt1': Stale file handle lfs setdirstripe: cannot create stripe dir '.testdir_mdt1': Stale file handle </pre> </div></div> <p>Nothing else was running on this client.</p> <p>Client logs attached as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32447/32447_sh-101-60.dk.gz" title="sh-101-60.dk.gz attached to LU-12209">sh-101-60.dk.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>MDT0 and 1 dk logs attached as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32449/32449_oak-md1-s2-MDT0.dk.gz" title="oak-md1-s2-MDT0.dk.gz attached to LU-12209">oak-md1-s2-MDT0.dk.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> and <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32448/32448_oak-md1-s1-MDT1.dk.gz" title="oak-md1-s1-MDT1.dk.gz attached to LU-12209">oak-md1-s1-MDT1.dk.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Stephane,</p> <p>Thanks for the more detailed logs.</p> <p>Here's the source of that ESTALE:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000040:00000001:18.0:1555692294.403870:0:22597:0:(llog_osd.c:322:llog_osd_declare_write_rec()) Process entered 00000040:00000001:18.0:1555692294.403870:0:22597:0:(llog_osd.c:340:llog_osd_declare_write_rec()) Process leaving (rc=18446744073709551500 : -116 : ffffffffffffff8c) 00000040:00000001:18.0:1555692294.403871:0:22597:0:(llog.c:960:llog_declare_write_rec()) Process leaving (rc=18446744073709551500 : -116 : ffffffffffffff8c) 00000040:00000001:18.0:1555692294.403871:0:22597:0:(llog_cat.c:141:llog_cat_new_log()) Process leaving via out (rc=18446744073709551500 : -116 : 0xffffffffffffff8c) </pre> </div></div> <p>Looks to be out of osp_md_declare_write:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (dt2osp_obj(dt)-&gt;opo_stale) return -ESTALE;</pre> </div></div> <p>But I'm not sure of much more.  I'm going to ask Lai to take a look at this - It's in the DNE area, as you noted.</p> <p> </p> <p>As for what would fix this...  A failover /failbackof MDT1 might do the trick.  It kind of looks like there's confusion over the state of an object in memory, and I think that might clear it up.</p> <p>Thanks Patrick for this analysis. I see that <tt>obj-&gt;opo_stale = 1;</tt> only in <tt>osp_invalidate()</tt>...</p> <p>Because it's not impacting production, but just new group creation, we won't failover the MDT today (new groups can wait a bit <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>). We have some interactive jobs running. But I'll try to find a good time during the weekend to do so. Let me know if you want me to grab more debug info before then.</p> <p>This looks to be the same issue which was fixed by <a href="https://review.whamcloud.com/#/c/33401/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33401/</a>, can you apply this patch on all MDS's and try again?</p> <p>Hi Lai,</p> <p>We restarted the servers with the patch this morning and the problem is now gone. Thanks!</p> <p>Nice! Thanks all. <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sthiell" class="user-hover" rel="sthiell">sthiell</a> note that this fix is included in the upcoming 2.12.1</p> <p>Peter, this patch (<a href="https://review.whamcloud.com/#/c/33401/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33401/</a> - <a href="https://jira.whamcloud.com/browse/LU-11418" title="hung threads on MDT and MDT won&#39;t umount" class="issue-link" data-issue-key="LU-11418"><del>LU-11418</del></a> llog: refresh remote llog upon -ESTALE) is already available in 2.12.0:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 71f409c9b31b90fa432f1f46ad4e612fb65c7fcc Author: Lai Siyao &lt;lai.siyao@intel.com&gt; Date: Wed Oct 17 13:29:53 2018 +0800 LU-11418 llog: refresh remote llog upon -ESTALE </pre> </div></div> <p>But it's not included in 2.10.7 (that we're running on our Oak servers).</p> <p>Stephane</p> <p>You are correct. Yet another illustration as to why it is confusing to having multiple patches tracked under the same Jira ticket spanning release boundaries</p> <p>Peter</p> Duplicate LU-11418 Development Rank 1|i00f53: Rank (Obsolete) 9223372036854775807 Severity