https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3421 Lustre <p>Several times during our test shot we would encounter a situation where OSTs would report ENOSPC even tho there was enough space and inodes available. In time the MDS would become pinned and it would have to be rebooted to get a working file system again. I have attached the console logs for the MDS and OSS.</p> RHEL6.4 running with Lutsre 2.4-rc2 and cray clients running [2.4.0-rc1 SLES11 SP1 / 2.4.0-rc2 SLES11 SP2] clients LU-3421

(ost_handler.c:1762:ost_blocking_ast()) Error -2 syncing data on lock cancel causes first ENOSPC client issues then MDS server locks up

Bug Minor Resolved Fixed Zhenyu Xu James A Simmons llnl Thu, 30 May 2013 16:50:26 +0000 Thu, 18 Sep 2014 21:00:51 +0000 Tue, 3 Sep 2013 13:11:54 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 0 7 <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>backtrace from the vmcore which will be uploaded soon:</p> <p>PID: 4814 TASK: ffff880e23333500 CPU: 15 COMMAND: "mdt03_140"<br/> #0 <span class="error">&#91;ffff8800607e7e90&#93;</span> crash_nmi_callback at ffffffff8102d316<br/> #1 <span class="error">&#91;ffff8800607e7ea0&#93;</span> notifier_call_chain at ffffffff81513a85<br/> #2 <span class="error">&#91;ffff8800607e7ee0&#93;</span> atomic_notifier_call_chain at ffffffff81513aea<br/> #3 <span class="error">&#91;ffff8800607e7ef0&#93;</span> notify_die at ffffffff8109cc1e<br/> #4 <span class="error">&#91;ffff8800607e7f20&#93;</span> do_nmi at ffffffff8151174b<br/> #5 <span class="error">&#91;ffff8800607e7f50&#93;</span> nmi at ffffffff81511010<br/> <span class="error">&#91;exception RIP: _spin_lock+30&#93;</span><br/> RIP: ffffffff8151087e RSP: ffff880e23335c40 RFLAGS: 00000293<br/> RAX: 00000000000014a6 RBX: ffff88099a82c200 RCX: ffff880e23335cf8<br/> RDX: 00000000000014a3 RSI: ffff880e23335cf0 RDI: ffff880ac63c35d8<br/> RBP: ffff880e23335c40 R8: 0000000000000000 R9: ffff8800b6c1dc00<br/> R10: 0000000000000009 R11: ffffffffa08dd5a0 R12: ffff88099a82c200<br/> R13: ffff880ac63c35c0 R14: ffff880e23335cf8 R15: 0000000000000000<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff880e23335c40&#93;</span> _spin_lock at ffffffff8151087e<br/> #7 <span class="error">&#91;ffff880e23335c48&#93;</span> lock_res_and_lock at ffffffffa07e3070 <span class="error">&#91;ptlrpc&#93;</span><br/> #8 <span class="error">&#91;ffff880e23335c68&#93;</span> ldlm_lock_enqueue at ffffffffa07e859d <span class="error">&#91;ptlrpc&#93;</span><br/> #9 <span class="error">&#91;ffff880e23335cc8&#93;</span> ldlm_handle_enqueue0 at ffffffffa080f1bf <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff880e23335d38&#93;</span> mdt_enqueue at ffffffffa0e6e3c6 <span class="error">&#91;mdt&#93;</span><br/> #11 [ffff8#12 <span class="error">&#91;ffff880e23335da8&#93;</span> mds_regular_handle at ffffffffa0eae165 <span class="error">&#91;mdt&#93;</span><br/> #13 <span class="error">&#91;ffff880e23335db8&#93;</span> ptlrpc_server_handle_request at ffffffffa0841388 <span class="error">&#91;ptlrpc&#93;</span><br/> #14 <span class="error">&#91;ffff880e23335eb8&#93;</span> ptlrpc_main at ffffffffa084271e <span class="error">&#91;ptlrpc&#93;</span><br/> #15 <span class="error">&#91;ffff880e23335f48&#93;</span> kernel_thread at ffffffff8100c0ca80e23335d58]</p> <p>It looks like there are a couple of different bugs being hit here:</p> <ul class="alternate" type="square"> <li>the OSTs are apparently reporting -ENOSPC when there is space available on them. Is this possibly a case where the OST is reporting ENOSPC during precreate when there are free inodes, but there are no free blocks? What is the <tt>lfs df</tt> and <tt>lfs df -i</tt> output from the filesystem? If this state hits again, please also collect the output from <tt>lctl get_param osp.<b>.sync_</b> osp.*.create_count</tt> on the MDS</li> <li>the MDS goes into a busy loop trying to create objects on the OSTs, rather than returning -ENOSPC to the client. While it is good to ensure that the OSTs have been tried for creates, it doesn't make sense to try precreate so often if the OST is already out of space.</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 29 08:18:40 widow-mds1 kernel: [76013.652071] Lustre: routed1-OST0015-osc-MDT0000: slow creates, last=[0x100150000:0x22041:0x0], next=[0x100150000:0x22041:0x0], reserved=0, syn_changes=2061, syn_rpc_in_progress=1, status=-19 May 29 08:19:32 widow-mds1 kernel: [76065.440057] Lustre: routed1-OST00f0-osc-MDT0000: slow creates, last=[0x100f00000:0x25e93:0x0], next=[0x100f00000:0x25e93:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=1, status=-19 May 29 08:23:37 widow-mds1 kernel: [76310.325026] LustreError: 6065:0:(osp_precreate.c:484:osp_precreate_send()) routed1-OST002a-osc-MDT0000: can't precreate: rc = -28 May 29 08:23:37 widow-mds1 kernel: [76310.367372] LustreError: 6065:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 70657 previous similar messages </pre> </div></div> <p>There was previously debugging code in osp_precreate_reserve() because I was worried about exactly this kind of situation, but my recent patch <a href="http://review.whamcloud.com/6219" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6219</a> (commit dc2bcafd2a0b) removed it before the 2.4.0 release because it would have LBUG'd in this case:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">@@ -1062,17 +1061,6 @@ <span class="code-object">int</span> osp_precreate_reserve(<span class="code-keyword">const</span> struct lu_env *env, struct osp_device *d) <span class="code-keyword">while</span> ((rc = d-&gt;opd_pre_status) == 0 || rc == -ENOSPC || rc == -ENODEV || rc == -EAGAIN) { -#<span class="code-keyword">if</span> LUSTRE_VERSION_CODE &lt; OBD_OCD_VERSION(2, 3, 90, 0) - /* - * to address Andreas's concern on possible busy-loop - * between <span class="code-keyword">this</span> thread and osp_precreate_send() - */ - <span class="code-keyword">if</span> (unlikely(count++ == 1000)) { - osp_precreate_timeout_condition(d); - LBUG(); - } -#endif - /* * increase number of precreations */ </pre> </div></div> <p>It probably makes sense to reinstate this if it loops only, say, twice for -ENOSPC, since there are other OSTs that could be used and it doesn't make sense to block the MDS thread for such a long time.</p> <p>It seems to me that the update of <tt>osp_pre_status</tt> at the start of <tt>osp_pre_update_status()</tt> is racy. If <tt>rc == 0</tt> (i.e. the statfs succeeded) then <tt>osp_pre_status = 0</tt>, even though it is set to <tt>-ENOSPC</tt> again later on. It would be better to have something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (rc != 0) { d-&gt;opd_pre_status = rc; <span class="code-keyword">goto</span> out; } </pre> </div></div> <p>Otherwise, <tt>opd_pre_status</tt> can be changed to 0 and break <tt>osp_precreate_ready_condition()</tt> checking of -ENOSPC.</p> <p>It also seems like the -ENOSPC hysteresis that is so well described in the comment in <tt>osp_pre_update_status()</tt> is not actually implemented. When the free space is &lt; 0.1% <tt>opd_pre_status = -ENOSPC</tt>, but it is immediately cleared if free space is &gt;= 0.1%. It also seems that there is a longstanding bug in the code, with min(used blocks vs. 1 GByte). It seems something like the following is needed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> * On very large disk, say 16TB 0.1% will be 16 GB. We don't want to * lose that amount of space so in those cases we report no space left * <span class="code-keyword">if</span> their is less than 1 GB left, and clear it at 2GB. <span class="code-keyword">if</span> (likely(msfs-&gt;os_type != 0)) { <span class="code-comment">/* msfs contains valid data */</span> used = min((msfs-&gt;os_blocks - msfs-&gt;os_bfree) &gt;&gt; 10, 1ULL &lt;&lt; (30 - msfs-&gt;os_bsize); <span class="code-keyword">if</span> (d-&gt;opd_pre_status == 0 &amp;&amp; msfs-&gt;os_ffree &lt; 32 || msfs-&gt;os_bavail &lt; used) { d-&gt;opd_pre_status = -ENOSPC; : : } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (d-&gt;opd_pre_status == -ENOSPC &amp;&amp; msgs-&gt;os_ffree &gt; 64 &amp;&amp; msfs-&gt;os_bavail &gt; used * 2) { d-&gt;opd_pre_status = 0; : : } </pre> </div></div> <p>The <tt>opd_pre_status = 0</tt> state should only be set if there is enough space, not only if the STATFS RPC succeeded.</p> <p>James, I expect the stack trace you pointed out is not a primary cause of this problem. That looks just to be an MDS thread that is blocked waiting for a DLM lock that is held by another thread. My guess is that there is something wrong in the precreate code at ENOSPC time that is keeping the MDS threads blocked for too long a time, and this is caused by either really being out of space on the OSTs (either inodes or blocks is the same, since there is no point allocating objects on an OST with no space) or maybe false ENOSPC caused by a problem with the space grants or similar.</p> <p>It <em>does</em> point out that we have something wrong with the usage of the DLM locks, or similar, because we should never be blocked for such a long time holding a spinlock.</p> <p>I uploaded the vmcore to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-3421" title="(ost_handler.c:1762:ost_blocking_ast()) Error -2 syncing data on lock cancel causes first ENOSPC client issues then MDS server locks up" class="issue-link" data-issue-key="LU-3421"><del>LU-3421</del></a></p> <p>Andreas the lfs df -i output was pretty much what you seen below. The lfs df results were basically the same. In the last test shot we saw issues with 4 ost becoming very full but this was not the case this time.</p> <p>/lustre/routed1-OST0130_UUID 3872000 86598 3785402 2% /lustre/routed1<span class="error">&#91;OST:304&#93;</span><br/> /lustre/routed1-OST00ac_UUID 3872000 86584 3785416 2% /lustre/routed1<span class="error">&#91;OST:172&#93;</span><br/> /lustre/routed1-OST0047_UUID 3872000 86539 3785461 2% /lustre/routed1<span class="error">&#91;OST:71&#93;</span><br/> /lustre/routed1-OST01c2_UUID 3872000 85067 3786933 2% /lustre/routed1<span class="error">&#91;OST:450&#93;</span><br/> UUID Inodes IUsed IFree IUse% Mounted on</p> <p>Talking with Oleg he didn't believe it was due to grants. Each OST is 250GB in size. I also expected the real issue is happening on the OSS rather than the MDS but the admins only gathered a vmcore from the MDS.</p> <p>I agree that osp_pre_update_status() should not be setting the status to 0 first. though I think that check to limit time we spent in osp_precreate_reserve() does not depend on the status, it's a hardlimit. and in the worst case when osp_precreate_ready_condition() missed -ENOSPC due to the race we still should leave osp_pre_update_status() up on expiration? notice the expiration point is not reset in the loop.</p> <p>James, </p> <p>Do you have OSS debug log? Want to check the reason why it reports ENOSPC while it seems there are plenty.</p> <p>The only OSS logs I have are like the one I posted here.</p> <p>James, how many clients were mounting this filesystem? If each OST is 250GB, and each client gets a 32MB grant, that means 32 clients/GB of free space, so 8000 clients would essentially pin all of the available space on each client. I see something a bit strange in the code that might be causing a problem here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ofd_statfs(...) { osfs-&gt;os_bavail -= min_t(obd_size, osfs-&gt;os_bavail, ((ofd-&gt;ofd_tot_dirty + ofd-&gt;ofd_tot_pending + osfs-&gt;os_bsize - 1) &gt;&gt; ofd-&gt;ofd_blockbits)); : : /* The QoS code on the MDS does not care about space reserved <span class="code-keyword">for</span> * precreate, so take it out. */ <span class="code-keyword">if</span> (exp_connect_flags(exp) &amp; OBD_CONNECT_MDS) { struct filter_export_data *fed; fed = &amp;obd-&gt;obd_self_export-&gt;exp_filter_data; osfs-&gt;os_bavail -= min_t(obd_size, osfs-&gt;os_bavail, fed-&gt;fed_grant &gt;&gt; ofd-&gt;ofd_blockbits); } </pre> </div></div> <p>This is <b>subtracting</b> the granted space from the available space returned to the MDS, but I <em>think</em> it should be <b>adding</b> the granted space back into os_bavail so that the MDS does not consider the grant space as "used". Otherwise, if the clients have reserved a lot of space on the OSTs they may not actually get to use it because the MDS will never allocate space there.</p> <p>A secondary issue is that there is no coordination between the space granted to a specific client and the objects that the MDS allocates to that client, which would become more important as the free space is running out. There <em>could</em> be some kind of (IMHO complex) coordination here between the MDS and clients/OSTs, but I think it would be easier if we just got the grant shrinking code to work again, as there is no guarantee that (a) clients doing IO will have any grant at all, and (b) the clients have grant on the OSTs for which they have been asked to write on. Returning unused grant to the OSTs as the free space shrinks is the best way to ensure that there is some left for the clients actually doing IO.</p> <blockquote> <p>This is subtracting the granted space from the available space returned to the MDS,</p></blockquote> <p>This is actually only subtracting the space reserved for pre-creation for which we use the self export.</p> <blockquote> <p> but I think it should be adding the granted space back into os_bavail so that the MDS does not consider the grant space as "used". Otherwise, if the clients have reserved a lot of space on the OSTs they may not actually get to use it because the MDS will never allocate space there.</p></blockquote> <p>I think there is nothing to add back, since only tot_dirty and tot_pending are taken into account here. Please note that tot_granted is not withdrawn anywhere.</p> <blockquote> <p>A secondary issue is that there is no coordination between the space granted to a specific client and the objects that the MDS allocates to that client, which would become more important as the free space is running out.</p></blockquote> <p>I would agree if we were taking tot_granted out in statfs reply, however i don't think this is the case.</p> <blockquote> <p>There could be some kind of (IMHO complex) coordination here between the MDS and clients/OSTs, but I think it would be easier if we just got the grant shrinking code to work again, as there is no guarantee that (a) clients doing IO will have any grant at all, and (b) the clients have grant on the OSTs for which they have been asked to write on. Returning unused grant to the OSTs as the free space shrinks is the best way to ensure that there is some left for the clients actually doing IO.</p></blockquote> <p>I am all for resurrecting grant shrinking, although i haven't had the opportunity to do it yet by lack of time. In fact, we might as well just disconnect (and therefore release grants) from OSTs when idle and when the replay list is empty. We could then reconnect on demand. IMHO, such a scheme would have other benefits: less clients to wait for during recovery and less PING requests.</p> <p>As for the original problem, it seems that precreate requests fails with ENOSPC on the OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 29 09:00:06 widow-oss11a4 kernel: [79146.403713] LustreError: 25344:0:(ofd_obd.c:1338:ofd_create()) routed1-OST016b: unable to precreate: rc = -28 </pre> </div></div> <p>James, could you please dump the grant state on the OST by running "lctl get_param obdfilter.<b>.tot</b> obdfilter.<b>.grant</b>"?</p> <p>While running some tests locally, i found out that the space reserved for precreation always decreases, eventually reaches 0 and stays there. It seems that we exit from ofd_grant() at line 641:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 637 <span class="code-comment">/* align grant on block size */</span> 638 grant &amp;= ~((1ULL &lt;&lt; ofd-&gt;ofd_blockbits) - 1); 639 640 <span class="code-keyword">if</span> (!grant) 641 RETURN(0); </pre> </div></div> <p>I think there are two issues:</p> <ul> <li>ofd_grant_create() is not aggressive enough in reserving space for precreation and ends up requesting an amount of grant space smaller than a block</li> <li>the rounding in ofd_grant() turns the &lt;4KB allocation into 0.</li> </ul> <p>I will provide a patch.</p> <p>I tested your patch at small scale and it works fine. I like to keep this ticket open until our next test shot to ensure this addresses the problem. Andreas point out other issues as well.</p> <p>ok James. Do you have a timeframe for your next testshot yet?</p> <p>Looks like the end of July for our next test shot. I will see if I can duplicate it at a smaller scale in the mean time.</p> <p>James</p> <p>This patch landed has landed both for 2.4.1 and 2.5. Has the issue reproduced in any test runs featuring this patch? If not, then perhaps we can close the ticket and reopen if it ever does reappear...</p> <p>Peter</p> <p>I haven't seen this bug in some time. You can close it.</p> <p>Thanks James!</p> Related LU-1947 Development Rank 1|hzvsbb: Rank (Obsolete) 8486 Severity