https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4950 Lustre <p>While running sanity-benchmark test fsx with the following configuration, it timed out in 3600s.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSCOUNT=1 MDSSIZE=2097152 OSTCOUNT=2 OSTSIZE=8388608 FSTYPE=zfs SLOW=yes </pre> </div></div> <p>txg_sync was stuck on OSS node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>txg_sync D 0000000000000001 0 2665 2 0x00000080 ffff88006f4cbba0 0000000000000046 00000000ffffffff 000037007418a66d ffff88006f4cbb10 ffff88007cba1a60 0000000000bba084 ffffffffaca9e91e ffff880070bc05f8 ffff88006f4cbfd8 000000000000fbc8 ffff880070bc05f8 Call Trace: [&lt;ffffffff810a7091&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff81528823&gt;] io_schedule+0x73/0xc0 [&lt;ffffffffa0142e7c&gt;] cv_wait_common+0x8c/0x100 [spl] [&lt;ffffffff8109b290&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa0142f08&gt;] __cv_wait_io+0x18/0x20 [spl] [&lt;ffffffffa02864ab&gt;] zio_wait+0xfb/0x1b0 [zfs] [&lt;ffffffffa021e5a5&gt;] dsl_pool_sync+0x2f5/0x540 [zfs] [&lt;ffffffffa023725e&gt;] spa_sync+0x40e/0xa00 [zfs] [&lt;ffffffff8103ea79&gt;] ? kvm_clock_get_cycles+0x9/0x10 [&lt;ffffffffa024253a&gt;] txg_sync_thread+0x27a/0x4b0 [zfs] [&lt;ffffffff81059329&gt;] ? set_user_nice+0xc9/0x130 [&lt;ffffffffa02422c0&gt;] ? txg_sync_thread+0x0/0x4b0 [zfs] [&lt;ffffffffa013e9bf&gt;] thread_generic_wrapper+0x5f/0x70 [spl] [&lt;ffffffffa013e960&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/7ec85f26-cb6e-11e3-95c9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7ec85f26-cb6e-11e3-95c9-52540035b04c</a></p> <br/> Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_5/47/">http://build.whamcloud.com/job/lustre-b2_5/47/</a><br/> FSTYPE=zfs<br/> LU-4950

sanity-benchmark test fsx hung: txg_sync was stuck on OSS

Bug Critical Closed Duplicate Alex Zhuravlev Jian Yu zfs Thu, 24 Apr 2014 08:48:49 +0000 Thu, 16 Apr 2020 07:29:31 +0000 Thu, 16 Apr 2020 07:29:31 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.7.0 Lustre 2.4.3 Lustre 2.5.3 Lustre 2.8.0 Lustre 2.10.0 0 15 <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/74/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/74/</a><br/> FSTYPE=zfs</p> <p>sanity-benchmark test iozone also hit the same issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/41f6d9ec-bfa7-11e3-8176-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/41f6d9ec-bfa7-11e3-8176-52540035b04c</a></p> <p>Hello Alex,</p> <p>Could you please take a look whether this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-2476" title="poor OST file creation rate performance with zfs backend" class="issue-link" data-issue-key="LU-2476"><del>LU-2476</del></a>/<a href="https://jira.whamcloud.com/browse/LU-2600" title="lustre metadata performance is very slow on zfs" class="issue-link" data-issue-key="LU-2600"><del>LU-2600</del></a> ? Thanks.</p> <p>It looks more likely a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4716" title="replay-ost-single test_5: stuck in dbuf_read-&gt;zio_wait" class="issue-link" data-issue-key="LU-4716"><del>LU-4716</del></a> - stuck in zio_wait(), i.e. IO couldn't complete.</p> <p>What was the storage devices for the OST storage pools? At least we should make sure not to use certain IO schedulers known to cause ZFS IO starvation for those devices. </p> <p>Hi Isaac,</p> <p>The tests were performed on virtual machines by autotest. The storage devices were LVs on a /dev/vda partition. And the I/O scheduler was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /sys/block/vda/queue/scheduler noop anticipatory [deadline] cfq </pre> </div></div> <p>FYI, the same failure was reported in <a href="https://jira.whamcloud.com/browse/LU-2887" title="sanity-quota test_12a: slow due to ZFS VMs sharing single disk" class="issue-link" data-issue-key="LU-2887"><del>LU-2887</del></a> before. There were some discussions and experiment data in that ticket.</p> <p>Jian, no, this is definitely not 2476/2600. I hit this locally few times as well - basically everything got stuck. we discussed this a bit with Brian B., but the root cause isn't clear yet.</p> <p>Hi Jian, I'd suggest to set IO scheduler of /dev/vda to noop. In fact, ZFS automatically sets IO schedulers of storage pool devices to noop, as other schedulers could cause ZFS IO starvation. But since the pool uses LVs, ZFS can't see /dev/vda at all.</p> <p>Hi Isaac,</p> <p>I tried noop before, please refer to: <a href="https://jira.hpdd.intel.com/browse/LU-2887?focusedCommentId=67639&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-67639" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-2887?focusedCommentId=67639&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-67639</a></p> <p>This seems to be similar to the txg_sync issue described in <a href="https://jira.whamcloud.com/browse/LU-2887" title="sanity-quota test_12a: slow due to ZFS VMs sharing single disk" class="issue-link" data-issue-key="LU-2887"><del>LU-2887</del></a>. Possibly related to how autotest sets up disks.</p> <p>In fact, in <a href="https://jira.whamcloud.com/browse/LU-2887" title="sanity-quota test_12a: slow due to ZFS VMs sharing single disk" class="issue-link" data-issue-key="LU-2887"><del>LU-2887</del></a>, even with OSTCOUNT=2, the txg_sync issue still existed. Please refer to the last several comments.<br/> In the Maloo reports in this ticket, the OSTCOUNT was also 2.</p> <p>This may actually be a <a href="https://github.com/zfsonlinux/zfs/issues/1640" class="external-link" target="_blank" rel="nofollow noopener">ZFS issue</a> and should be retested with zfs 0.6.3 when it arrives.</p> <p>There was also a failure of this test with ldiskfs in <a href="https://jira.whamcloud.com/browse/LU-5214" title="Failure on test suite replay-ost-single test_5" class="issue-link" data-issue-key="LU-5214"><del>LU-5214</del></a>.</p> <p>Hit several timeout failures in lustre-b2_6-rc2 zfs testing, cannot find error messages</p> <p><a href="https://testing.hpdd.intel.com/test_sets/254eb710-0ecb-11e4-9f57-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/254eb710-0ecb-11e4-9f57-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/f69a0dd2-0ec7-11e4-9f57-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f69a0dd2-0ec7-11e4-9f57-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/75580c7e-0ec2-11e4-9f57-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/75580c7e-0ec2-11e4-9f57-5254006e85c2</a></p> <p>That rules out this being fixed in zfs 0.6.3</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/84/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/84/</a><br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>Patch <a href="http://review.whamcloud.com/11298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11298</a> was already in Lustre b2_5 build #84. However, the failure still occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/eee5bd1a-2b24-11e4-bb80-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/eee5bd1a-2b24-11e4-bb80-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/88e0db3a-2b24-11e4-bb80-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/88e0db3a-2b24-11e4-bb80-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/294bd846-2b24-11e4-bb80-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/294bd846-2b24-11e4-bb80-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/deb57516-2b24-11e4-bb80-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/deb57516-2b24-11e4-bb80-5254006e85c2</a></p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/86/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/86/</a> (2.5.3 RC1)<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/f444ed7c-311d-11e4-b503-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f444ed7c-311d-11e4-b503-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9210d3d4-311b-11e4-b503-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9210d3d4-311b-11e4-b503-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/c618d298-30ae-11e4-9e60-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c618d298-30ae-11e4-9e60-5254006e85c2</a></p> <p>I assume the latest reports were using ZFS 0.6.3 already.</p> <p>Can you elaborate on how the OSS storage are configured:</p> <ul class="alternate" type="square"> <li>Two OSTs, I assume one zpool for each OST</li> <li>For each OST, what are the disks in the zpool? Any RAIDZ configuration? How are the disks configured on the host of the OSS VM? Does each disk correspond to a dedicated physical disk or do they share physical disks? Anything else on the host competing for IO on those disks? What are the IO schedulers on the host disks?</li> </ul> <p>It appeared that the OSS had 2G memory. Any ZFS tunings? Can you include outputs of 'cat /sys/modules/zfs/parameters/*' i.e. run-time values of the tunings?</p> <p>The write throttle and IO scheduling have been completely rewritten in 0.6.3. The defaults may not be proper for a VM test system. So first I need to understand exactly how the OSS is configured.</p> <p>Hi Isaac,</p> <p>I just added Minh and Joshua to watch this ticket. The OSS and OSTs were configured by autotest system. To answer the above questions, I need look into the test nodes where a ZFS test session is being tested. If you would like, I can trigger a ZFS test session and tell you the test nodes, so that you can log into the nodes and investigate the parameters directly. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I checked stack dumps from the most recent three failures reported by Jian, and found the OSS z_ioctl_iss thread were all stuck in a same place:</p> <p>z_ioctl_iss/0 S 0000000000000001 0 3996 2 0x00000080<br/> ffff880079dc9e20 0000000000000046 ffff880079dc9da0 ffffffffa024aa46<br/> ffff880079dc9de0 ffffc900192865f8 ffff880079a76000 0000000000000200<br/> ffff8800730c8638 ffff880079dc9fd8 000000000000fbc8 ffff8800730c8638<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa024aa46&gt;&#93;</span> ? vdev_disk_io_start+0x156/0x1b0 <span class="error">&#91;zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa013f686&gt;&#93;</span> taskq_thread+0x3c6/0x3f0 <span class="error">&#91;spl&#93;</span></p> <p>Likely it's really stuck in vdev_disk_io_flush() which is probably inlined. It seemed like the syncing txg is about to complete and a WRITE_FLUSH_FUA is to be issued after the uberblocks have been updated. There's also other threads waiting for the txg to be synced.</p> <p>The only place it could get stuck in vdev_disk_io_flush() seemed to be in bio_alloc(), which indicated a deadlock under memory pressure.</p> <p>There's two things to help troubleshoot:<br/> 1. Set zfs_nocacheflush:<br/> options zfs zfs_nocacheflush=1<br/> This makes DKIOCFLUSHWRITECACHE a noop. Then the deadlock may move elsewhere or simply disappear.<br/> 2. Whether zfs_nocacheflush is set or not, once the timeout happens, it'd be useful to gather "zpool events -v" outputs on the OSS, which'd give more details on the state of the stuck zio.</p> <p>seen this in hard failover test for master branch zfs<br/> server and client: lustre-master build #2642<br/> <a href="https://testing.hpdd.intel.com/test_sets/63b3a69c-37c4-11e4-a2a6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/63b3a69c-37c4-11e4-a2a6-5254006e85c2</a></p> <blockquote> <p>There's two things to help troubleshoot:<br/> 1. Set zfs_nocacheflush:<br/> options zfs zfs_nocacheflush=1<br/> This makes DKIOCFLUSHWRITECACHE a noop. Then the deadlock may move elsewhere or simply disappear.<br/> 2. Whether zfs_nocacheflush is set or not, once the timeout happens, it'd be useful to gather "zpool events -v" outputs on the OSS, which'd give more details on the state of the stuck zio.</p></blockquote> <p>I pushed a for-test-only patch on Lustre b2_5 branch to run the previously timed out tests on ZFS with "options zfs zfs_nocacheflush=1": <a href="http://review.whamcloud.com/11906" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11906</a></p> <p>If the deadlock still occurred, I'll ask TEI team for help to implement 2.</p> <p>My previous analysis might be inaccurate as I neglected the '?' in the stack entry:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[&lt;ffffffffa024aa46&gt;] ? vdev_disk_io_start+0x156/0x1b0 [zfs] </pre> </div></div> <p>Which meant that the vdev_disk_io_start() function likely returned and the WRITE_FLUSH_FUA bio was already submitted - the z_ioctl_iss thread was waiting for more work to do. I checked the OSS stack dumps, and the z_* threads were all idle, so there was no more IO left to do. These led me to believe that:<br/> 1. IO for syncing the txg were all done, and WRITE_FLUSH_FUA was submitted as an IO barrier, i.e. to make sure all previous writes do hit persistent storage.<br/> 2. But the WRITE_FLUSH_FUA somehow didn't complete, causing txg_sync to wait and hang.</p> <p>Suggestions to troubleshoot stay the same: on the OSS, set zfs_nocacheflush, and gather "zpool events -v" outputs.</p> <p>I set up a test cluster on onyx-23vm[5-8] with the following configuration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>NAME=ncli RCLIENTS=onyx-23vm6 mds_HOST=onyx-23vm7 MDSCOUNT=1 MDSSIZE=3145728 MDSDEV1=/dev/vda5 ost_HOST=onyx-23vm8 OSTCOUNT=2 OSTSIZE=8388608 OSTDEV1=/dev/vda5 OSTDEV2=/dev/vda6 SLOW=yes ENABLE_QUOTA=yes FSTYPE=zfs </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pdsh -l root -S -w onyx-23vm[5-8] "export PATH=\$PATH:/sbin:/usr/sbin; cat /sys/block/vda/queue/scheduler" onyx-23vm6: [noop] anticipatory deadline cfq onyx-23vm5: [noop] anticipatory deadline cfq onyx-23vm7: [noop] anticipatory deadline cfq onyx-23vm8: [noop] anticipatory deadline cfq </pre> </div></div> <p>Lustre build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/88/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/88/</a><br/> sanity-benchmark passed in 16878s: <a href="https://testing.hpdd.intel.com/test_sets/6cb0f36a-4473-11e4-8e4d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6cb0f36a-4473-11e4-8e4d-5254006e85c2</a><br/> performance-sanity passed in 7864s: <a href="https://testing.hpdd.intel.com/test_sets/6d589494-4473-11e4-8e4d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6d589494-4473-11e4-8e4d-5254006e85c2</a><br/> parallel-scale passed in 6336s: <a href="https://testing.hpdd.intel.com/test_sets/6e4fdc22-4473-11e4-8e4d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6e4fdc22-4473-11e4-8e4d-5254006e85c2</a></p> <p>I'm re-running the tests with "deadline" scheduler, which is the default value after the test nodes are provisioned.</p> <blockquote><p>I'm re-running the tests with "deadline" scheduler, which is the default value after the test nodes are provisioned.</p></blockquote> <p>sanity-benchmark passed in 15688s. However, performance-sanity test 8 hung. The output of "zpool events -v" on OSS is attached. And here is the test report: <a href="https://testing.hpdd.intel.com/test_sets/810760ae-45c4-11e4-aeca-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/810760ae-45c4-11e4-aeca-5254006e85c2</a></p> <p>Hi Isaac, if you can access onyx cluster, please feel free to use onyx-23vm[5-8]. The testing environment is remained.</p> <p>Looks like the combination of LVM elimination and noop scheduler made a difference. Can you please repeat the same test again (i.e. noop, and no LVM)? Considering the low pass rate, we'd be able to isolate LVM+deadline as the cause of IO timeout if it passes again.</p> <blockquote><p>Can you please repeat the same test again (i.e. noop, and no LVM)? </p></blockquote> <p>Sure. Testing was started.</p> <p>Hi Isaac,</p> <p>I ran the same tests on the same test nodes with the same configuration as that of the noop scheduler in the previous comments for two times. Both hung at performance-sanity. Here is the test report of the latest run: <a href="https://testing.hpdd.intel.com/test_sessions/b9ebf592-4803-11e4-a4e8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/b9ebf592-4803-11e4-a4e8-5254006e85c2</a></p> <p>So, noop scheduler does not resolve the issue.</p> <p>The test failure above appeared to be a different issue <a href="https://jira.whamcloud.com/browse/LU-3786" title="performance-sanity test_6: mkdir hung (sys_mkdir)" class="issue-link" data-issue-key="LU-3786"><del>LU-3786</del></a>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 7840 files. INFO: task mkdir:23576 blocked for more than 120 seconds. Not tainted 2.6.32-431.29.2.el6.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mkdir D 0000000000000000 0 23576 23568 0x00000080 ffff88003ca05db8 0000000000000086 ffff880067cb2040 ffff880067cb2040 ffff880067cb2040 ffff880067cb2040 000000103ca05e38 ffff88005a5f9000 ffff880067cb25f8 ffff88003ca05fd8 000000000000fbc8 ffff880067cb25f8 Call Trace: [&lt;ffffffff811ab390&gt;] ? mntput_no_expire+0x30/0x110 [&lt;ffffffff8152a5be&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8152a45b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff81196c00&gt;] lookup_create+0x30/0xd0 [&lt;ffffffff8119ae63&gt;] sys_mkdirat+0x83/0x1b0 [&lt;ffffffff811ab390&gt;] ? mntput_no_expire+0x30/0x110 [&lt;ffffffff810e1e07&gt;] ? audit_syscall_entry+0x1d7/0x200 [&lt;ffffffff8119afa8&gt;] sys_mkdir+0x18/0x20 </pre> </div></div> <p>So it still looks positive that the combination of LVM elimination and noop scheduler made a difference.</p> <p>sanity-hsm/24a on review-zfs on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a14816a6-5753-11e4-86e0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a14816a6-5753-11e4-86e0-5254006e85c2</a></p> <p>Likely it can be fixed by this ZFS patch:<br/> <a href="https://github.com/zfsonlinux/zfs/pull/2828" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/2828</a></p> <p>I'll test it soon.</p> <p>I know that we prefer to use tagged upstream ZFS releases, but if this allows us to pass tests I'd prefer that it be landed into our repository to use it. In that case, it is worthwhile to post in the upstream GitHub ticket that this is needed for Lustre to pass and hopefully Brian will land it more quickly.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12810" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12810</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4950" title="sanity-benchmark test fsx hung: txg_sync was stuck on OSS" class="issue-link" data-issue-key="LU-4950"><del>LU-4950</del></a> build: Use new version of ZFS/SPL<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 89bdb0a802a5c6701b4e7c76f5caa5cdd2276568</p> <p>Build ZFS with this change:</p> <p><a href="http://review.whamcloud.com/12810" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12810</a></p> <p>Isaac, can you look at this issue again? It seems the upstream Illumos patch that you were looking at didn't fix this problem in ZFS. We are still trying to land that patch for fixing <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a> via <a href="http://review.whamcloud.com/13050" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13050</a> but it doesn't help this problem.</p> <p>Nathaniel is working on updating our ZFS/SPL version to 0.6.3-1.2 in <a href="https://jira.whamcloud.com/browse/LU-6105" title="Update ZFS/SPL version to 0.6.3-1.2" class="issue-link" data-issue-key="LU-6105"><del>LU-6105</del></a>, which includes the following SPL fix:<br/> <a href="https://github.com/zfsonlinux/spl/commit/a3c1eb77721a0d511b4fe7111bb2314686570c4b" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/spl/commit/a3c1eb77721a0d511b4fe7111bb2314686570c4b</a></p> <p>It looks positive that the SPL fix would fix many of the ZIO timeouts we've been seeing, including this one and the <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a>.</p> <p><a href="https://jira.whamcloud.com/browse/LU-6105" title="Update ZFS/SPL version to 0.6.3-1.2" class="issue-link" data-issue-key="LU-6105"><del>LU-6105</del></a> landed to master some time back. Are we now able to close this as a duplicate?</p> <p>Patch <a href="http://review.whamcloud.com/13322" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13322</a> for <a href="https://jira.whamcloud.com/browse/LU-6105" title="Update ZFS/SPL version to 0.6.3-1.2" class="issue-link" data-issue-key="LU-6105"><del>LU-6105</del></a> landed on master branch, however, the issue in this ticket was not resolved:<br/> <a href="https://testing.hpdd.intel.com/test_sets/07180fae-c7cf-11e4-92dc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/07180fae-c7cf-11e4-92dc-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/d41fb90c-c7cf-11e4-92dc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d41fb90c-c7cf-11e4-92dc-5254006e85c2</a></p> <p>Console log on OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10:51:22:txg_sync D 0000000000000001 0 2753 2 0x00000080 10:51:22: ffff8800652c9b90 0000000000000046 00000000ffffffff 00001708d56216b3 10:51:22: 0000000000000000 ffff88006c6eeb40 000000000038fc98 ffffffffab01512e 10:51:22: ffff880063d04638 ffff8800652c9fd8 000000000000fbc8 ffff880063d04638 10:51:22:Call Trace: 10:51:22: [&lt;ffffffff810a6d31&gt;] ? ktime_get_ts+0xb1/0xf0 10:51:22: [&lt;ffffffff81529e83&gt;] io_schedule+0x73/0xc0 10:51:22: [&lt;ffffffffa0143596&gt;] cv_wait_common+0xa6/0x120 [spl] 10:51:22: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 10:51:22: [&lt;ffffffffa0143628&gt;] __cv_wait_io+0x18/0x20 [spl] 10:51:22: [&lt;ffffffffa028d81b&gt;] zio_wait+0xfb/0x1c0 [zfs] 10:51:22: [&lt;ffffffffa0220993&gt;] dsl_pool_sync+0xb3/0x3f0 [zfs] 10:51:22: [&lt;ffffffffa02398bf&gt;] spa_sync+0x40f/0xa70 [zfs] 10:51:22: [&lt;ffffffffa0243771&gt;] ? spa_txg_history_set+0xc1/0xf0 [zfs] 10:51:22: [&lt;ffffffffa0246c7d&gt;] txg_sync_thread+0x30d/0x520 [zfs] 10:51:22: [&lt;ffffffff810591a9&gt;] ? set_user_nice+0xc9/0x130 10:51:22: [&lt;ffffffffa0246970&gt;] ? txg_sync_thread+0x0/0x520 [zfs] 10:51:22: [&lt;ffffffffa013ec3f&gt;] thread_generic_wrapper+0x5f/0x70 [spl] 10:51:22: [&lt;ffffffffa013ebe0&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] 10:51:22: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 10:51:22: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 10:51:22: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 10:51:22: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>server and client:lustre-master build # 3094 RHEL6.6 zfs</p> <p><a href="https://testing.hpdd.intel.com/test_sets/a8e1a402-2865-11e5-8329-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a8e1a402-2865-11e5-8329-5254006e85c2</a></p> <p>Another instance for EL6.7 Server/EL6.7 Client - ZFS<br/> Master, build# 3270<br/> <a href="https://testing.hpdd.intel.com/test_sets/92039c10-a275-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/92039c10-a275-11e5-bdef-5254006e85c2</a><br/> Another instance forEL7.1 Server/EL7.1 Client - ZFS<br/> Master, build# 3264<br/> <a href="https://testing.hpdd.intel.com/test_sets/4296dba6-a135-11e5-83b8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4296dba6-a135-11e5-83b8-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>txg_sync D 0000000000000000 0 23282 2 0x00000080 ffff880039e73b60 0000000000000046 0000000000000000 0000a3cd3e7c281e ffff88007cd68ae8 ffff880037d9e600 00003b391acd9c96 ffffffffa9941bdc 0000000006f6b1a7 0000000103dd04a2 ffff88007cd69068 ffff880039e73fd8 Call Trace: [&lt;ffffffff815395d3&gt;] io_schedule+0x73/0xc0 [&lt;ffffffffa01a6eaf&gt;] cv_wait_common+0xaf/0x130 [spl] [&lt;ffffffff810a1460&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8106ee1a&gt;] ? __cond_resched+0x2a/0x40 [&lt;ffffffffa01a6f48&gt;] __cv_wait_io+0x18/0x20 [spl] [&lt;ffffffffa02b129b&gt;] zio_wait+0x10b/0x1e0 [zfs] [&lt;ffffffffa0238b1b&gt;] dsl_pool_sync+0x2bb/0x440 [zfs] [&lt;ffffffffa024fb3e&gt;] spa_sync+0x35e/0xb10 [zfs] [&lt;ffffffffa02661f8&gt;] txg_sync_thread+0x3d8/0x670 [zfs] [&lt;ffffffffa0265e20&gt;] ? txg_sync_thread+0x0/0x670 [zfs] [&lt;ffffffffa0265e20&gt;] ? txg_sync_thread+0x0/0x670 [zfs] [&lt;ffffffffa01a2258&gt;] thread_generic_wrapper+0x68/0x80 [spl] [&lt;ffffffffa01a21f0&gt;] ? thread_generic_wrapper+0x0/0x80 [spl] [&lt;ffffffff810a0fce&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a0f30&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>another instance on master build#3305 RHEL6.7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/6472141a-bc84-11e5-b3b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6472141a-bc84-11e5-b3b7-5254006e85c2</a></p> <p>Another instance on master for FULL - EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b5ff5e4c-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b5ff5e4c-cb88-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL6.7 Server/Client - ZFS, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/766ea3ec-cb55-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/766ea3ec-cb55-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/Client - ZFS, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/f0dd9616-ca6e-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f0dd9616-ca6e-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/98eb99ce-cb47-11e5-a59a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/98eb99ce-cb47-11e5-a59a-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/ddc75dc6-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ddc75dc6-cb88-11e5-b49e-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b5ff5e4c-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b5ff5e4c-cb88-11e5-b49e-5254006e85c2</a></p> <p>it would be interesting to try with <a href="http://review.whamcloud.com/#/c/18341/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18341/</a></p> <p>Another instance found on b2_8 for failover testing , build# 6.<br/> <a href="https://testing.hpdd.intel.com/test_sessions/54ec62da-d99d-11e5-9ebe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/54ec62da-d99d-11e5-9ebe-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2</a></p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a></p> <p>In master branch, tag-2.9.53 el7 zfs is still experiencing the same error, reopen it and see if anything else is needed<br/> <a href="https://testing.hpdd.intel.com/test_sets/19e62a78-f8f9-11e6-aac4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/19e62a78-f8f9-11e6-aac4-5254006e85c2</a></p> <p>oss console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>07:34:00:[ 3545.939658] Lustre: DEBUG MARKER: == obdfilter-survey test 1a: Object Storage Targets survey =========================================== 07:23:02 (1487748182) 07:34:00:[ 3546.225025] Lustre: DEBUG MARKER: lctl dl | grep obdfilter 07:34:00:[ 3546.764980] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' 07:34:00:[ 3547.846310] Lustre: Echo OBD driver; http://www.lustre.org/ 07:34:00:[ 4200.108290] INFO: task txg_sync:22316 blocked for more than 120 seconds. 07:34:00:[ 4200.110327] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 07:34:00: 07:34:00:[ 4200.112402] txg_sync D 0000000000000001 0 22316 2 0x00000080 07:34:00:[ 4200.116024] ffff88003b79fac0 0000000000000046 ffff88003a438fb0 ffff88003b79ffd8 07:34:00:[ 4200.118769] ffff88003b79ffd8 ffff88003b79ffd8 ffff88003a438fb0 ffff88007fd16c40 07:34:00:[ 4200.120870] 0000000000000000 7fffffffffffffff ffff88003bd09908 0000000000000001 07:34:00:[ 4200.123100] Call Trace: 07:34:00:[ 4200.124786] [&lt;ffffffff8168bac9&gt;] schedule+0x29/0x70 07:34:00:[ 4200.126894] [&lt;ffffffff81689519&gt;] schedule_timeout+0x239/0x2d0 07:34:00:[ 4200.128916] [&lt;ffffffff810c4fe2&gt;] ? default_wake_function+0x12/0x20 07:34:00:[ 4200.130962] [&lt;ffffffff810ba238&gt;] ? __wake_up_common+0x58/0x90 07:34:00:[ 4200.133032] [&lt;ffffffff81060c1f&gt;] ? kvm_clock_get_cycles+0x1f/0x30 07:34:00:[ 4200.135083] [&lt;ffffffff8168b06e&gt;] io_schedule_timeout+0xae/0x130 07:34:00:[ 4200.137129] [&lt;ffffffff810b1416&gt;] ? prepare_to_wait_exclusive+0x56/0x90 07:34:00:[ 4200.139259] [&lt;ffffffff8168b108&gt;] io_schedule+0x18/0x20 07:34:00:[ 4200.141213] [&lt;ffffffffa0679617&gt;] cv_wait_common+0xa7/0x130 [spl] 07:34:00:[ 4200.143303] [&lt;ffffffff810b1720&gt;] ? wake_up_atomic_t+0x30/0x30 07:34:00:[ 4200.145316] [&lt;ffffffffa06796f8&gt;] __cv_wait_io+0x18/0x20 [spl] 07:34:00:[ 4200.147464] [&lt;ffffffffa07d351b&gt;] zio_wait+0x10b/0x1f0 [zfs] 07:34:00:[ 4200.149508] [&lt;ffffffffa075ccdf&gt;] dsl_pool_sync+0xbf/0x440 [zfs] 07:34:00:[ 4200.151553] [&lt;ffffffffa0777868&gt;] spa_sync+0x388/0xb50 [zfs] 07:34:00:[ 4200.153602] [&lt;ffffffff810b174b&gt;] ? autoremove_wake_function+0x2b/0x40 07:34:00:[ 4200.155664] [&lt;ffffffff81689c72&gt;] ? mutex_lock+0x12/0x2f 07:34:00:[ 4200.157616] [&lt;ffffffffa07894e5&gt;] txg_sync_thread+0x3c5/0x620 [zfs] 07:34:00:[ 4200.159653] [&lt;ffffffffa0789120&gt;] ? txg_init+0x280/0x280 [zfs] 07:34:00:[ 4200.161605] [&lt;ffffffffa0674851&gt;] thread_generic_wrapper+0x71/0x80 [spl] 07:34:00:[ 4200.163625] [&lt;ffffffffa06747e0&gt;] ? __thread_exit+0x20/0x20 [spl] 07:34:00:[ 4200.165586] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 07:34:00:[ 4200.167428] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 07:34:00:[ 4200.169377] [&lt;ffffffff81696958&gt;] ret_from_fork+0x58/0x90 07:34:00:[ 4200.171244] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 07:34:00:[ 4200.173334] INFO: task lctl:3231 blocked for more than 120 seconds. 07:34:00:[ 4200.175136] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 07:34:00:[ 4200.177040] lctl D 0000000000000000 0 3231 3197 0x00000080 07:34:00:[ 4200.178923] ffff88003d3db948 0000000000000082 ffff88001d2f9f60 ffff88003d3dbfd8 07:34:00:[ 4200.181132] ffff88003d3dbfd8 ffff88003d3dbfd8 ffff88001d2f9f60 ffff88003b9c5b50 07:34:00:[ 4200.183052] ffff88003b9c5a10 ffff88003b9c5b58 ffff88003b9c5a38 0000000000000000 07:34:00:[ 4200.185136] Call Trace: 07:34:00:[ 4200.186747] [&lt;ffffffff8168bac9&gt;] schedule+0x29/0x70 07:34:00:[ 4200.188372] [&lt;ffffffffa067967d&gt;] cv_wait_common+0x10d/0x130 [spl] 07:34:00:[ 4200.190210] [&lt;ffffffff810b1720&gt;] ? wake_up_atomic_t+0x30/0x30 07:34:00:[ 4200.191823] [&lt;ffffffffa06796b5&gt;] __cv_wait+0x15/0x20 [spl] 07:34:00:[ 4200.193657] [&lt;ffffffffa07886a3&gt;] txg_wait_open+0xb3/0xf0 [zfs] 07:34:00:[ 4200.195240] [&lt;ffffffffa0743b0d&gt;] dmu_tx_wait+0x34d/0x360 [zfs] 07:34:00:[ 4200.196968] [&lt;ffffffffa0743bb1&gt;] dmu_tx_assign+0x91/0x4b0 [zfs] 07:34:00:[ 4200.198521] [&lt;ffffffffa0f8c05a&gt;] osd_trans_start+0xaa/0x3c0 [osd_zfs] 07:34:00:[ 4200.200290] [&lt;ffffffffa108e2db&gt;] ofd_trans_start+0x6b/0xe0 [ofd] 07:34:00:[ 4200.202029] [&lt;ffffffffa109460b&gt;] ofd_commitrw_write+0x94b/0x1c30 [ofd] 07:34:00:[ 4200.203800] [&lt;ffffffffa109850d&gt;] ofd_commitrw+0x56d/0xa30 [ofd] 07:34:00:[ 4200.205564] [&lt;ffffffffa0b30679&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] 07:34:00:[ 4200.207167] [&lt;ffffffffa11895ea&gt;] echo_client_prep_commit.isra.49+0x69a/0xc30 [obdecho] 07:34:00:[ 4200.209038] [&lt;ffffffffa11930cf&gt;] echo_client_iocontrol+0x9cf/0x1c50 [obdecho] 07:34:00:[ 4200.210635] [&lt;ffffffffa0b30679&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] 07:34:00:[ 4200.212403] [&lt;ffffffffa0b1bb1e&gt;] class_handle_ioctl+0x19de/0x2150 [obdclass] 07:34:00:[ 4200.213966] [&lt;ffffffff812a8558&gt;] ? security_capable+0x18/0x20 07:34:00:[ 4200.215613] [&lt;ffffffffa0b005d2&gt;] obd_class_ioctl+0xd2/0x170 [obdclass] 07:34:00:[ 4200.217176] [&lt;ffffffff81212025&gt;] do_vfs_ioctl+0x2d5/0x4b0 07:34:00:[ 4200.218778] [&lt;ffffffff81691ef1&gt;] ? __do_page_fault+0x171/0x450 07:34:00:[ 4200.220285] [&lt;ffffffffa0b1b258&gt;] ? class_handle_ioctl+0x1118/0x2150 [obdclass] 07:34:00:[ 4200.222049] [&lt;ffffffff812122a1&gt;] SyS_ioctl+0xa1/0xc0 07:34:00:[ 4200.223471] [&lt;ffffffff81696a09&gt;] system_call_fastpath+0x16/0x1b 07:34:00:[ 4200.225197] INFO: task lctl:3232 blocked for more than 120 seconds. </pre> </div></div> <p>Sarah, I can't find ofd_destroy() in the traces, I doubt this is <a href="https://jira.whamcloud.com/browse/LU-4950" title="sanity-benchmark test fsx hung: txg_sync was stuck on OSS" class="issue-link" data-issue-key="LU-4950"><del>LU-4950</del></a>..</p> <p>a duplicate of <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a></p> Duplicate LU-5575 LU-6812 LU-7374 Related LU-5242 LU-2887 LU-5214 LU-6105 LU-12234 LU-9247 LU-5575 Development Rank 1|hzwkwn: Rank (Obsolete) 13698 Severity