https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7404 Lustre <p>Running SWL, OSS has repeated timeouts</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 5 15:23:57 iws9 kernel: LNet: Service thread pid 23042 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Nov 5 15:23:57 iws9 kernel: Pid: 23042, comm: ll_ost00_004 Nov 5 15:23:57 iws9 kernel: Nov 5 15:23:57 iws9 kernel: Call Trace: Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa067c380&gt;] ? vdev_mirror_child_done+0x0/0x30 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff815395c3&gt;] io_schedule+0x73/0xc0 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa05b2f8f&gt;] cv_wait_common+0xaf/0x130 [spl] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff810a1460&gt;] ? autoremove_wake_function+0x0/0x40 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa05b3028&gt;] __cv_wait_io+0x18/0x20 [spl] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa06bd2eb&gt;] zio_wait+0x10b/0x1e0 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0614939&gt;] dbuf_read+0x439/0x850 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0614ef1&gt;] __dbuf_hold_impl+0x1a1/0x4f0 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa06152bd&gt;] dbuf_hold_impl+0x7d/0xb0 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0616790&gt;] dbuf_hold+0x20/0x30 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa061d0d7&gt;] dmu_buf_hold_noread+0x87/0x140 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa061d1cb&gt;] dmu_buf_hold+0x3b/0x90 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0612fb8&gt;] ? dbuf_rele_and_unlock+0x268/0x400 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0686e5a&gt;] zap_lockdir+0x5a/0x770 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff81178fcd&gt;] ? kmem_cache_alloc_node_trace+0x1cd/0x200 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa06889ca&gt;] zap_lookup_norm+0x4a/0x190 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0688ba3&gt;] zap_lookup+0x33/0x40 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa062cc76&gt;] dmu_tx_hold_zap+0x146/0x210 [zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa1034255&gt;] osd_declare_object_create+0x2a5/0x440 [osd_zfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa11738e4&gt;] ofd_precreate_objects+0x4e4/0x19d0 [ofd] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa04b4b61&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa1180a9b&gt;] ? ofd_grant_create+0x23b/0x3e0 [ofd] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa116384e&gt;] ofd_create_hdl+0x56e/0x2640 [ofd] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0c28e80&gt;] ? lustre_pack_reply_v2+0x220/0x280 [ptlrpc] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0c930ec&gt;] tgt_request_handle+0x8bc/0x12e0 [ptlrpc] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0c3a9e1&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffffa0c39ba0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff810a0fce&gt;] kthread+0x9e/0xc0 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff810a0f30&gt;] ? kthread+0x0/0xc0 Nov 5 15:23:57 iws9 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Lustre-log dump attached</p> Hyperion /SWL 2.7.61 review build 35536 (patch <a href="http://review.whamcloud.com/17053">http://review.whamcloud.com/17053</a> - Revert &quot;<a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><strike>LU-4865</strike></a> zfs: grow block size by write pattern&quot;) LU-7404

ZFS OSS - Numerous timeouts - SWL

Bug Major Resolved Fixed Jinshan Xiong Cliff White RZ_LS zfs Fri, 6 Nov 2015 16:46:22 +0000 Wed, 20 Mar 2019 18:43:46 +0000 Thu, 3 Nov 2016 19:10:20 +0000 Lustre 2.8.0 0 17 <p>Issues seen after reverting patch for <a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a>.</p> <p>Current testing with DNE+ZFS on Hyperion has shown that 2.7.56 does <b>not</b> have this timeout problem, while 2.7.62 <b>does</b> have the timeouts. Testing is underway with the 2.7.59 tag to see if the timeout problem is present there as well. The <a href="https://jira.whamcloud.com/browse/LU-6750" title="missing stop callback in osd-zfs" class="issue-link" data-issue-key="LU-6750"><del>LU-6750</del></a> patch was landed as <tt>v2_7_56_0-5-g27929cc</tt> (i.e. 5 patches past 2.7.56) so it wasn't present in the 2.7.56 testing that passed. The <a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a> patch was landed as <tt>v2_7_59_0-20-g3e43691</tt> (i.e. 20 patches past 2.7.59) so the 2.7.59 testing will give us a good half-way mark without being affected by <a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a>.</p> <p>I've pushed <a href="http://review.whamcloud.com/17112" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17112</a> to revert the <a href="https://jira.whamcloud.com/browse/LU-6750" title="missing stop callback in osd-zfs" class="issue-link" data-issue-key="LU-6750"><del>LU-6750</del></a> patch in addition to the <a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a> patch reversion, based on the current tip of master, for the next stage of testing after 2.7.59, depending on those results.</p> <p>This problem has been isolated to the update from ZFS 0.6.4.2 to 0.6.5.2, commit v2_7_61_0-39-ge94d375d8a, patch <a href="http://review.whamcloud.com/16399" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16399</a> "<a href="https://jira.whamcloud.com/browse/LU-7153" title="Update ZFS/SPL version to 0.6.5.2" class="issue-link" data-issue-key="LU-7153"><del>LU-7153</del></a> build: Update SPL/ZFS to 0.6.5.2".</p> <p>One option for debugging would be to bisect the ZFS code upstream to see which ZFS patch has introduced this.</p> <p>Here is the patch to reset ZFS baseline to version 0.6.4.2: <a href="http://review.whamcloud.com/17712" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17712</a></p> <p>This hang isn't the same as deadlock as <a href="https://github.com/zfsonlinux/spl/pull/484" class="external-link" target="_blank" rel="nofollow noopener">zfsonlinux/spl#484</a> "Disable dynamic taskqs by default to avoid deadlock" that was fixed in 0.6.5.3 is it? If it isn't has someone opened a bug upstream for this?</p> <p>Hi Nathaniel,</p> <p>Cliff has verified that the same issue can be seen on current master, where the baseline of ZFS is 0.6.5.3. I will investigate if this is a problem of OSD-ZFS, or a problem of ZFS baseline. We will file an issue upstream if it turns out to be a problem of ZFS.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17712/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17712/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7404" title="ZFS OSS - Numerous timeouts - SWL" class="issue-link" data-issue-key="LU-7404"><del>LU-7404</del></a> zfs: reset ZFS baseline to 0.6.4.2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 182b30b7699858c73a990c36c51b70c40858a1fe</p> <p>I'm dropping the priority of this issue because it's not blocking 2.8 release any more. I will keep this ticket open till I find the root cause.</p> <p>I filed a ticket on upstreaming zfs at: <a href="https://github.com/zfsonlinux/zfs/issues/4210" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/issues/4210</a></p> <p>Given the discussion on <a href="https://github.com/zfsonlinux/zfs/issues/4210" class="external-link" target="_blank" rel="nofollow noopener">zfs#4210</a> should I push a patch to move to 0.6.5.4?</p> <p>Hi Nathaniel,</p> <p>We've tried 0.6.5.4 before and it didn't help.</p> <p>Only ZFS Master includes the patches the upstream ZFS developer mentioned and we tried that on Hyperion yesterday, unfortunately it didn't help either. </p> <p>it turned out the root cause of this issue may be due to cache hit ratio dropped significantly introduced by the commit of multilisted ARC implementation.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ARC Total accesses: 863.95k Cache Hit Ratio: 99.68% 861.22k Cache Miss Ratio: 0.32% 2.73k Actual Hit Ratio: 99.68% 861.22k Data Demand Efficiency: 100.00% 369.92k CACHE HITS BY CACHE LIST: Most Recently Used: 30.96% 266.66k Most Frequently Used: 69.04% 594.56k Most Recently Used Ghost: 0.00% 0 Most Frequently Used Ghost: 0.00% 0 CACHE HITS BY DATA TYPE: Demand Data: 42.95% 369.92k Prefetch Data: 0.00% 0 Demand Metadata: 57.05% 491.30k Prefetch Metadata: 0.00% 0 CACHE MISSES BY DATA TYPE: Demand Data: 0.00% 0 Prefetch Data: 0.00% 0 Demand Metadata: 100.00% 2.73k Prefetch Metadata: 0.00% 0 </pre> </div></div> <p>This is the ARC stats before the patch is introduced, the cache hit ratio is close to 100%.</p> <p>While this is the ARC stats after the commit is landed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ARC Total accesses: 967.83k Cache Hit Ratio: 18.54% 179.46k Cache Miss Ratio: 81.46% 788.37k Actual Hit Ratio: 18.54% 179.46k Data Demand Efficiency: 10.47% 868.25k CACHE HITS BY CACHE LIST: Most Recently Used: 35.70% 64.07k Most Frequently Used: 64.30% 115.39k Most Recently Used Ghost: 3.77% 6.77k Most Frequently Used Ghost: 6.30% 11.30k CACHE HITS BY DATA TYPE: Demand Data: 50.64% 90.89k Prefetch Data: 0.00% 0 Demand Metadata: 49.36% 88.58k Prefetch Metadata: 0.00% 0 CACHE MISSES BY DATA TYPE: Demand Data: 98.60% 777.36k Prefetch Data: 0.00% 0 Demand Metadata: 1.40% 11.01k Prefetch Metadata: 0.00% 0</pre> </div></div> <p>Ran latest patch from Jinshan - furnished him with debug information from SWL run.</p> <p>From Brian @ LLNL - it looks like the underlying storage on Hyperion doesn't match TXG dirty throttle in ZFS. OSTs can only do ~50MB/s but zfs_dirty_data_max was set to 6G by default. After Brian reduced zfs_dirty_data_max to 1G, he didn't see I/O timeouts any more. This has been verified by Cliff.</p> <p>The reason we can't see this problem elsewhere is because it's unusual to have such a slow storage in production.</p> <p>My theory to explain this problem is that the ARC lock contention in old ZFS releases may throttle I/O by itself.</p> <p>Shouldn't ZFS limit the TXG size based on the speed of the underlying storage? I'd thought that was the main feature of the dynamic TXG sizing - record how quickly the data could be flushed to disk in the previous TXG and then use it to limit it the size of the next TXG based on the desired TXG commit interval.</p> <p>From what I have seen in the code, ZFS starts to throttle I/O when dirty data in pool reaches 60% of zfs_dirty_data_max(6GB by default on Hyperion); at the same time it will wake up quiescing thread to close the current open txg. But there is no mechanism to adjust TXG size from the time how long previous TXG was complete. This pushed the txg sync time to be about 100s on average.</p> <p>Andreas the ZFS IO throttled used to work exactly as you've described it. That implementation had some significant draw backs so it was reworked in 2013. Specifically, it often resulted in a nasty saw tooth behavior when processes hit the calculated limit (which was often wrong). See <a href="https://github.com/zfsonlinux/zfs/commit/e8b96c6007bf97cdf34869c1ffbd0ce753873a3d" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/commit/e8b96c6007bf97cdf34869c1ffbd0ce753873a3d</a> for all gory the details. Bottom line the code was updated so that `dmu_tx_assign()` now starts injecting tiny delays which gradually increase as you approach the dirty limit. This back pressure is supposed to keep the TXG from getting to large and allow the system to settle in to a steady equilibrium.</p> <p>Clearly that doesn't seem to be working properly for a Lustre workload even though `dmu_tx_assign()` was injecting large delays. I suspect it may be because of the large number of Lustre IO threads all attempting to dirty ARC buffers, that wasn't exactly the use case this was designed for. The code also makes sure to allow any caller which was delayed in to the next TXG to prevent starving them. How quickly these delays ramp up is controlled by the `zfs_delay_scale` module parameter which we should able to tune to control this. There's a great comment describing this in the zfs-module-parameters.5 man page which is worth reading. See <a href="https://github.com/zfsonlinux/zfs/blob/master/man/man5/zfs-module-parameters.5#L1768" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/blob/master/man/man5/zfs-module-parameters.5#L1768</a>.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19632" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19632</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7404" title="ZFS OSS - Numerous timeouts - SWL" class="issue-link" data-issue-key="LU-7404"><del>LU-7404</del></a> zfs: Reset default zfs version to 0.6.5.5<br/> Project: fs/lustre-release<br/> Branch: b2_8<br/> Current Patch Set: 1<br/> Commit: 947f37272b35ede3791616261e6a6929aebcac86</p> <p>AFAIK no action is required for 2.9 and master is now running against a more current 0.6.5.x ZFS release</p> Duplicate LU-7602 Related LU-7153 LU-4865 LU-6750 LU-7987 Development End date Mon, 28 Dec 2015 16:46:22 +0000 Rank 1|hzxshj: Rank (Obsolete) 9223372036854775807 Severity Start date Fri, 6 Nov 2015 16:46:22 +0000