https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2887 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c</a>.</p> <p>The sub-test test_12a failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: sanity-quota 12a</p> <p>Looking through test 12a, things seem to have hung up on the runas dd (with oflag=sync) at the end of the test.</p> <p>OST has threads that are blocked on disk I/O (oss dmesg):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">txg_sync D 0000000000000000 0 24236 2 0x00000080 ffff88005027dbc0 0000000000000046 ffff88004b906ec0 0000000000000086 ffff88005027db70 ffff88007c7d4408 0000000000000001 ffff88007c7d4420 ffff880052101058 ffff88005027dfd8 000000000000fb88 ffff880052101058 Call Trace: [&lt;ffffffff81090b9e&gt;] ? prepare_to_wait_exclusive+0x4e/0x80 [&lt;ffffffffa016b5ac&gt;] cv_wait_common+0x9c/0x1a0 [spl] [&lt;ffffffffa02d5160&gt;] ? zio_execute+0x0/0xf0 [zfs] [&lt;ffffffff81090990&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa016b6e3&gt;] __cv_wait+0x13/0x20 [spl] [&lt;ffffffffa02d533b&gt;] zio_wait+0xeb/0x160 [zfs] [&lt;ffffffffa026b807&gt;] dsl_pool_sync+0x2a7/0x480 [zfs] [&lt;ffffffffa027e147&gt;] spa_sync+0x397/0x9a0 [zfs] [&lt;ffffffffa028fd41&gt;] txg_sync_thread+0x2c1/0x490 [zfs] [&lt;ffffffff810527f9&gt;] ? set_user_nice+0xc9/0x130 [&lt;ffffffffa028fa80&gt;] ? txg_sync_thread+0x0/0x490 [zfs] [&lt;ffffffffa0164668&gt;] thread_generic_wrapper+0x68/0x80 [spl] [&lt;ffffffffa0164600&gt;] ? thread_generic_wrapper+0x0/0x80 [spl] [&lt;ffffffff81090626&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81090590&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 ll_ost_io00_0 D 0000000000000000 0 18170 2 0x00000080 ffff8800427b9820 0000000000000046 0000000000000046 0000000000000001 ffff8800427b98b0 0000000000000086 ffff8800427b97e0 ffff88005027dd60 ffff8800427b7ab8 ffff8800427b9fd8 000000000000fb88 ffff8800427b7ab8 Call Trace: [&lt;ffffffff81090b9e&gt;] ? prepare_to_wait_exclusive+0x4e/0x80 [&lt;ffffffffa016b5ac&gt;] cv_wait_common+0x9c/0x1a0 [spl] [&lt;ffffffff81090990&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa016b6e3&gt;] __cv_wait+0x13/0x20 [spl] [&lt;ffffffffa028f573&gt;] txg_wait_synced+0xb3/0x190 [zfs] [&lt;ffffffffa0c71015&gt;] osd_trans_stop+0x365/0x420 [osd_zfs] [&lt;ffffffffa0cb9062&gt;] ofd_trans_stop+0x22/0x60 [ofd] [&lt;ffffffffa0cbdf06&gt;] ofd_commitrw_write+0x406/0x11b0 [ofd] [&lt;ffffffffa0cbf13d&gt;] ofd_commitrw+0x48d/0x920 [ofd] [&lt;ffffffffa085b708&gt;] obd_commitrw+0x128/0x3d0 [ost] [&lt;ffffffffa0862599&gt;] ost_brw_write+0xe49/0x14d0 [ost] [&lt;ffffffff812739b6&gt;] ? vsnprintf+0x2b6/0x5f0 [&lt;ffffffffa088c1f0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa08680e3&gt;] ost_handle+0x31e3/0x46f0 [ost] [&lt;ffffffffa05ca154&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [&lt;ffffffffa08dc02c&gt;] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc] [&lt;ffffffffa05be5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa08d3759&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [&lt;ffffffff81052223&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa08dd576&gt;] ptlrpc_main+0xb76/0x1870 [ptlrpc] [&lt;ffffffffa08dca00&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa08dca00&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] [&lt;ffffffffa08dca00&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> LU-2887

sanity-quota test_12a: slow due to ZFS VMs sharing single disk

Bug Minor Resolved Fixed Alex Zhuravlev Maloo MB performance zfs Wed, 27 Feb 2013 21:34:34 +0000 Mon, 29 May 2017 06:03:58 +0000 Sat, 3 Dec 2016 01:25:49 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 14 <p>i've been this in the local testing as well.</p> <p>Compairing results with ldiskfs via maloo:</p> <p>Some zfs runs complete in ~4K seconds (possibly SLOW=yes).<br/> ldisksfs runs complete in ~300 to 900 seconds.</p> <p>Possible fix is to mark this as SLOW only or SLOW for zfs?</p> <p>Will you work on this, Alex? Seems it's a generic zfs sync problem?</p> <p>sorry, not anytime soon</p> <p>Tag should be "sometimes_except" since it's now a SLOW test for zfs</p> <p>Alex, do you think we can learn something with some profiling ??</p> <p>Whole logs analysis does not indicate threads (at least ll_ost_io00_0 with pid 18170, which could be traced in Lustre debug-log) were hung. So we may only face a very slow test execution due to some external cause, the perfs were horribly slow :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:41:50: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d12/f.sanity-quota.12a-0] [count=17] [oflag=sync] 10:18:20:17+0 records in 10:18:23:17+0 records out 10:18:24:17825792 bytes (18 MB) copied, 2288.82 s, 7.8 kB/s 10:18:24:Write to ost1... 10:18:25:running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 10:18:26: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d12/f.sanity-quota.12a-1] [count=17] [oflag=sync] 10:26:34:dd: writing `/mnt/lustre/d0.sanity-quota/d12/f.sanity-quota.12a-1': Disk quota exceeded 10:26:37:5+0 records in 10:26:38:4+0 records out 10:26:39:4194304 bytes (4.2 MB) copied, 490.729 s, 8.5 kB/s 10:26:39:Free space from ost0... 10:26:40:CMD: wtm-24vm7 lctl set_param -n osd*.*MD*.force_sync 1 10:26:42:CMD: wtm-24vm7 lctl get_param -n osc.*MDT*.sync_* 10:26:43:CMD: wtm-24vm7 lctl get_param -n osc.*MDT*.sync_* 10:26:44:CMD: wtm-24vm7 lctl get_param -n osc.*MDT*.sync_* 10:26:45:CMD: wtm-24vm7 lctl get_param -n osc.*MDT*.sync_* 10:26:47:Waiting for local destroys to complete 10:26:51:CMD: wtm-24vm8 lctl set_param -n osd*.*OS*.force_sync 1 10:26:52:Write to ost1 after space freed from ost0... 10:26:54:running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 10:26:55: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d12/f.sanity-quota.12a-1] [count=17] [oflag=sync] </pre> </div></div> <p>BTW, my own runs of test_12a on zfs show much better perfs/stats. </p> <p>Nathaniel, did you hit this problem/time-out multiple times ?? Also, you have already set test_12a as ZFS_SLOW in change 5553 for <a href="https://jira.whamcloud.com/browse/LU-2874" title="Test timeout failure on test suite replay-ost-single test_8a: timeout on wait for dd" class="issue-link" data-issue-key="LU-2874"><del>LU-2874</del></a> !</p> <p>I will try to re-run the same test suite for your original <a href="https://jira.whamcloud.com/browse/LU-2829" title="Timeout on sanityn test_33a: zfs slow when commit_on_sharing enabled" class="issue-link" data-issue-key="LU-2829"><del>LU-2829</del></a>/change 5530/patch #2/build 13602 where these <a href="https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c</a> tests failed.</p> <p>Humm too bad/late, Build 13602 has been removed ...</p> <p>So back in trying to reproduce the very slow runs of test_12a.</p> <p>Most recent failure <a href="https://maloo.whamcloud.com/test_sets/bb4b4890-8895-11e2-b643-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bb4b4890-8895-11e2-b643-52540035b04c</a></p> <blockquote><p>Failure Rate: 7.00% of last 100 executions <span class="error">&#91;all branches&#93;</span></p></blockquote> <p>it might make sense to add some stats to osd-zfs/ so we know for sure how long txg_wait_synced() takes ?</p> <p>Lowering priority per discussion with Oleg.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/33/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/33/</a><br/> FSTYPE=zfs</p> <p>sanity-quota test_7a hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d7/f.sanity-quota.7a] [count=21] [oflag=sync] </pre> </div></div> <p>Syslog on OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 18 23:22:00 wtm-15vm4 kernel: txg_sync D 0000000000000000 0 15003 2 0x00000080 Aug 18 23:22:00 wtm-15vm4 kernel: ffff880052733ba0 0000000000000046 ffff8800ffffffff 0000333ae2e38992 Aug 18 23:22:00 wtm-15vm4 kernel: 00000000a0529170 ffff88006a5fce50 0000000000386bda ffffffffadee9654 Aug 18 23:22:00 wtm-15vm4 kernel: ffff8800523f3ab8 ffff880052733fd8 000000000000fb88 ffff8800523f3ab8 Aug 18 23:22:00 wtm-15vm4 kernel: Call Trace: Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff810a2351&gt;] ? ktime_get_ts+0xb1/0xf0 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff8150e913&gt;] io_schedule+0x73/0xc0 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa03e6d4c&gt;] cv_wait_common+0x8c/0x100 [spl] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff81096cc0&gt;] ? autoremove_wake_function+0x0/0x40 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa03e3717&gt;] ? taskq_dispatch_ent+0x57/0x110 [spl] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa03e6dd8&gt;] __cv_wait_io+0x18/0x20 [spl] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa052939b&gt;] zio_wait+0xfb/0x190 [zfs] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa04c2135&gt;] dsl_pool_sync+0x2f5/0x540 [zfs] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa04da82e&gt;] spa_sync+0x39e/0x970 [zfs] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff8103b8d9&gt;] ? kvm_clock_get_cycles+0x9/0x10 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa04e582a&gt;] txg_sync_thread+0x27a/0x4b0 [zfs] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff810560a9&gt;] ? set_user_nice+0xc9/0x130 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa04e55b0&gt;] ? txg_sync_thread+0x0/0x4b0 [zfs] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa03e2a3f&gt;] thread_generic_wrapper+0x5f/0x70 [spl] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffffa03e29e0&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff81096956&gt;] kthread+0x96/0xa0 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff810968c0&gt;] ? kthread+0x0/0xa0 Aug 18 23:22:00 wtm-15vm4 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/760750e4-095f-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/760750e4-095f-11e3-ad8a-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/33/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/33/</a><br/> FSTYPE=zfs</p> <p>replay-ost-single test_5 hung, and syslog on OSS showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 18 17:17:57 wtm-15vm4 kernel: txg_sync D 0000000000000000 0 29194 2 0x00000080 Aug 18 17:17:57 wtm-15vm4 kernel: ffff8800683c5ba0 0000000000000046 ffff8800ffffffff 00004af162da63c4 Aug 18 17:17:57 wtm-15vm4 kernel: 000000004a452700 ffff88004515a750 000000000070a032 ffffffffadef130d Aug 18 17:17:57 wtm-15vm4 kernel: ffff88006c9c1058 ffff8800683c5fd8 000000000000fb88 ffff88006c9c1058 Aug 18 17:17:57 wtm-15vm4 kernel: Call Trace: Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff810a2351&gt;] ? ktime_get_ts+0xb1/0xf0 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff8150e913&gt;] io_schedule+0x73/0xc0 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa03e6d4c&gt;] cv_wait_common+0x8c/0x100 [spl] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff81096cc0&gt;] ? autoremove_wake_function+0x0/0x40 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa03e3717&gt;] ? taskq_dispatch_ent+0x57/0x110 [spl] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa03e6dd8&gt;] __cv_wait_io+0x18/0x20 [spl] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa052939b&gt;] zio_wait+0xfb/0x190 [zfs] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa04c1f2c&gt;] dsl_pool_sync+0xec/0x540 [zfs] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa04da82e&gt;] spa_sync+0x39e/0x970 [zfs] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff8103b8d9&gt;] ? kvm_clock_get_cycles+0x9/0x10 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa04e582a&gt;] txg_sync_thread+0x27a/0x4b0 [zfs] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff810560a9&gt;] ? set_user_nice+0xc9/0x130 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa04e55b0&gt;] ? txg_sync_thread+0x0/0x4b0 [zfs] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa03e2a3f&gt;] thread_generic_wrapper+0x5f/0x70 [spl] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffffa03e29e0&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff81096956&gt;] kthread+0x96/0xa0 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff810968c0&gt;] ? kthread+0x0/0xa0 Aug 18 17:17:57 wtm-15vm4 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/15fe8b0a-095e-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/15fe8b0a-095e-11e3-ad8a-52540035b04c</a></p> <p>The following sub-tests also hung at "txg_sync":<br/> large-scale test_3a: <a href="https://maloo.whamcloud.com/test_sets/9ed19422-0963-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9ed19422-0963-11e3-ad8a-52540035b04c</a><br/> obdfilter-survey test_1a: <a href="https://maloo.whamcloud.com/test_sets/653914a0-0964-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/653914a0-0964-11e3-ad8a-52540035b04c</a><br/> parallel-scale test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/18f7c150-0963-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/18f7c150-0963-11e3-ad8a-52540035b04c</a><br/> parallel-scale-nfsv3 test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/bd745a9e-0964-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bd745a9e-0964-11e3-ad8a-52540035b04c</a><br/> parallel-scale-nfsv4 test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/17b573b2-0965-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/17b573b2-0965-11e3-ad8a-52540035b04c</a><br/> posix test_1: <a href="https://maloo.whamcloud.com/test_sets/73bee652-0965-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/73bee652-0965-11e3-ad8a-52540035b04c</a></p> <p>The issue in this ticket is blocking the zfs test session on Lustre b2_4 branch:<br/> <a href="https://maloo.whamcloud.com/test_sessions/0343b17e-095a-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/0343b17e-095a-11e3-ad8a-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>sanity-quota test 12a hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/15cc22b2-1559-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/15cc22b2-1559-11e3-8938-52540035b04c</a></p> <p>parallel-scale test simul: <a href="https://maloo.whamcloud.com/test_sets/4e4fc084-155a-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4e4fc084-155a-11e3-8938-52540035b04c</a></p> <p>is there a way to generate crash dump for the case? I'm still scratching my head around the issue. talked to Brian B., but he doesn't see the root cause as well.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a> (2.4.1 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>MDSCOUNT=1<br/> OSTCOUNT=7<br/> MDSSIZE=2097152<br/> OSTSIZE=2097152</p> <p>In <b>autotest</b> run, parallel-scale test simul timed out in <b>3600s</b>: <a href="https://maloo.whamcloud.com/test_sets/7a8c3714-18ab-11e3-aa54-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7a8c3714-18ab-11e3-aa54-52540035b04c</a></p> <p>Dmesg on OSS node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>txg_sync D 0000000000000000 0 30848 2 0x00000080 ffff880038d8bba0 0000000000000046 ffff8800ffffffff 0000c4392b16901e 000000001a86df70 ffff88006f691f30 00000000013837fa ffffffffadd58c90 ffff88002319d098 ffff880038d8bfd8 000000000000fb88 ffff88002319d098 Call Trace: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff8150ed03&gt;] io_schedule+0x73/0xc0 [&lt;ffffffffa03e6d4c&gt;] cv_wait_common+0x8c/0x100 [spl] [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa03e3717&gt;] ? taskq_dispatch_ent+0x57/0x110 [spl] [&lt;ffffffffa03e6dd8&gt;] __cv_wait_io+0x18/0x20 [spl] [&lt;ffffffffa052939b&gt;] zio_wait+0xfb/0x190 [zfs] [&lt;ffffffffa04c1f2c&gt;] dsl_pool_sync+0xec/0x540 [zfs] [&lt;ffffffffa04da82e&gt;] spa_sync+0x39e/0x970 [zfs] [&lt;ffffffff8103b8d9&gt;] ? kvm_clock_get_cycles+0x9/0x10 [&lt;ffffffffa04e582a&gt;] txg_sync_thread+0x27a/0x4b0 [zfs] [&lt;ffffffff810560a9&gt;] ? set_user_nice+0xc9/0x130 [&lt;ffffffffa04e55b0&gt;] ? txg_sync_thread+0x0/0x4b0 [zfs] [&lt;ffffffffa03e2a3f&gt;] thread_generic_wrapper+0x5f/0x70 [spl] [&lt;ffffffffa03e29e0&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>In <b>manual</b> test run, parallel-scale test simul passed in <b>8727s</b>: <a href="https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c</a></p> <p>In the above manual test report, we can see that although all of the sub-tests passed, many of them took very long time to be finished:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>simul 8727s connectathon 6880s iorssf 5570s iorfpp 5960s write_append_truncate 13940s write_disjoint 29272s </pre> </div></div> <p>I'll set up kdump on the test node and run the test manually again to try to get the vmcore dump file.</p> <p>Is this ticket related to <a href="https://jira.whamcloud.com/browse/LU-2476" title="poor OST file creation rate performance with zfs backend" class="issue-link" data-issue-key="LU-2476"><del>LU-2476</del></a>/<a href="https://jira.whamcloud.com/browse/LU-2600" title="lustre metadata performance is very slow on zfs" class="issue-link" data-issue-key="LU-2600"><del>LU-2600</del></a> which is about the performance issue on ZFS?</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a> (2.4.1 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>MDSCOUNT=1<br/> OSTCOUNT=7<br/> MDSSIZE=2097152<br/> OSTSIZE=2097152</p> <p>In <b>manual</b> test runs, sanity-quota did not hang, it finally passed in <b>27153s</b>: <a href="https://maloo.whamcloud.com/test_sets/529eda72-21ac-11e3-8292-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/529eda72-21ac-11e3-8292-52540035b04c</a><br/> Among the sub-tests, test 2 and 12a took very long time to be finished:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_2 16566s test_12a 3196s </pre> </div></div> <p>could some one run some single test (like simul) with ofd.*.sync_on_lock_cancel set to never please?</p> <p>I did submit <a href="http://review.whamcloud.com/#/c/7711/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7711/</a> few days ago, but Jira hasn't started even ..</p> <blockquote><p>could some one run some single test (like simul) with ofd.*.sync_on_lock_cancel set to never please?</p></blockquote> <p>Patch for Lustre b2_4 branch is in <a href="http://review.whamcloud.com/7725" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7725</a>.</p> <p>A manual test run is ongoing on wtm-<span class="error">&#91;82-85&#93;</span>, which are the test nodes where the parallel-scale test was performed against Lustre 2.4.1 RC2 previously. We'll compare the test results.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a> (2.4.1 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>MDSCOUNT=1<br/> OSTCOUNT=7<br/> MDSSIZE=2097152<br/> OSTSIZE=2097152</p> <p>Without the patch, parallel-scale test simul passed in <b>8727s</b>: <a href="https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c</a><br/> With the patch, parallel-scale test simul passed in <b>189s</b>: <a href="https://maloo.whamcloud.com/test_sets/a0eb119e-25af-11e3-aeff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a0eb119e-25af-11e3-aeff-52540035b04c</a></p> <p>ahaha</p> <p>Jian, can you tell which devices were used on OST in that 189s run? thanks.</p> <p>yet another question: if we're using raw devices for ZFS pools (not /dev/loop*), then what I/O scheduler is used?</p> <p>Hi Alex,</p> <p>The devices are SCSI disk partitions.</p> <p>For I/O scheduler:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /sys/block/sdb/queue/scheduler noop anticipatory [deadline] cfq </pre> </div></div> <p>thanks. no luck then..</p> <p>Jian, would you mind to try with noop please? (on the clean master branch)</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-master/1689/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1689/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs<br/> Test nodes: wtm-<span class="error">&#91;82-85&#93;</span></p> <p>With the default <b>"deadline"</b> I/O scheduler:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pdsh -l root -S -w wtm-[84,85] "cat /sys/block/sdb/queue/scheduler" wtm-84: noop anticipatory [deadline] cfq wtm-85: noop anticipatory [deadline] cfq </pre> </div></div> <p>parallel-scale test simul passed in <b>13457s</b>: <a href="https://maloo.whamcloud.com/test_sets/a154ee76-2680-11e3-b741-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a154ee76-2680-11e3-b741-52540035b04c</a></p> <p>With the <b>"noop"</b> I/O scheduler:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pdsh -l root -S -w wtm-[84,85] "cat /sys/block/sdb/queue/scheduler" wtm-84: [noop] anticipatory deadline cfq wtm-85: [noop] anticipatory deadline cfq </pre> </div></div> <p>parallel-scale test simul passed in <b>13589s</b>: <a href="https://maloo.whamcloud.com/test_sets/dae61290-26a4-11e3-94b1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dae61290-26a4-11e3-94b1-52540035b04c</a></p> <p>given no quick reply with the results, it's going to take another zillion seconds <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>would you mind to run manually simul and grab /proc/spl/kstat/zfs/txgs-* few times over the run?</p> <p>I guess it makes sense to specify zfs_txg_history=500 (or 1000) when you load zfs module.</p> <blockquote><p>would you mind to run manually simul and grab /proc/spl/kstat/zfs/txgs-* few times over the run?</p></blockquote> <blockquote><p>I guess it makes sense to specify zfs_txg_history=500 (or 1000) when you load zfs module.</p></blockquote> <p>OK, let me do more experiments.</p> <p>Hi Alex,<br/> Since the simul test is still running with noop scheduler, I did not stop it and gathered the /proc/spl/kstat/zfs/txgs-* data from both MDS (wtm-84) and OSS (wtm-85) for three times. Please refer to the attachments.</p> <p>thanks! learning ...</p> <p>do I understand correctly that all OSTs are using the same physical device?</p> <blockquote><p>do I understand correctly that all OSTs are using the same physical device?</p></blockquote> <p>Yes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ost_HOST=wtm-85 OSTCOUNT=7 OSTSIZE=2097152 OSTDEV1=/dev/sdb5 OSTDEV2=/dev/sdb6 OSTDEV3=/dev/sdb7 OSTDEV4=/dev/sdb8 OSTDEV5=/dev/sdb9 OSTDEV6=/dev/sdb10 OSTDEV7=/dev/sdb11 </pre> </div></div> <p>would it be possible to run simul using single OST on a non-shared phys.device?</p> <blockquote><p>would it be possible to run simul using single OST on a non-shared phys.device?</p></blockquote> <p>Sure. I can start a new test run now because the previous simul test run was just finished. I'm uploading the Maloo report.</p> <p>So, the new test run will be performed with single OST, deadline I/O scheduler, and zfs_txg_history=1000 while loading zfs module.</p> <p>please, make sure that single OST uses a physical device nobody else uses (including MDT). so the idea is have one ZFS pool in use on that device and zero concurrency between different ZFS pools. thanks in advance.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-master/1689/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1689/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>Test configuration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mds_HOST=wtm-84 MDSCOUNT=1 MDSSIZE=2097152 MDSDEV1=/dev/sdc ost_HOST=wtm-85 OSTCOUNT=1 OSTSIZE=15000000 OSTDEV1=/dev/sdd </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pdsh -l root -S -w wtm-84 "cat /sys/block/sdc/queue/scheduler" wtm-84: [noop] anticipatory deadline cfq # pdsh -l root -S -w wtm-85 "cat /sys/block/sdd/queue/scheduler" wtm-85: [noop] anticipatory deadline cfq </pre> </div></div> <p>parallel-scale test simul passed in <b>1950s</b>: <a href="https://maloo.whamcloud.com/test_sets/55b02a2c-26b1-11e3-9d3a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/55b02a2c-26b1-11e3-9d3a-52540035b04c</a></p> <p>hmm, sounds interesting.. any idea whether simul's workset depends on the space available? or probably it makes sense to return with OSTSIZE=$((6*2097152)) ?</p> <p>I think the performance of simul test does not depend on the available space on OST because no parameter is calculated from the available space on OST, and the simul test mainly does the open/close/unlink/rename/creat/truncate, etc. operations.<br/> In the above test run, I used OSTSIZE=15000000 (about 14G). Just re-ran the test with OSTSIZE=$((6*2097152))=12582912 (about 12G) and it passed in <b>2095s</b>: <a href="https://maloo.whamcloud.com/test_sets/e089858a-26c0-11e3-83d1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e089858a-26c0-11e3-83d1-52540035b04c</a></p> <p>thanks a lot for help. it seems the issue is due to two factors: 1) expensive sync (to be learned yet) 2) high cost of competing of few pools on a same phys.device. I don't think anybody will be running few pools on a same device in wild..</p> <p>You're welcome, Alex.</p> <p>I just checked the autotest configuration and found the OST devices used by it were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OSTDEV1=/dev/lvm-OSS/P1 OSTDEV2=/dev/lvm-OSS/P2 OSTDEV3=/dev/lvm-OSS/P3 OSTDEV4=/dev/lvm-OSS/P4 OSTDEV5=/dev/lvm-OSS/P5 OSTDEV6=/dev/lvm-OSS/P6 OSTDEV7=/dev/lvm-OSS/P7 </pre> </div></div> <p>It was likely that all of the LVs were created from the same PV, especially on virtual machines. Since most of the test nodes used by autotest system are VMs, and ZFS full test runs are performed by autotest regularly, it's hard to bypass the above factor 2).</p> <p>frankly I don't think we should be trying to improve ZFS for such an environment - it's not realistic to have many pools on a same device. solution to this isn't obvious either.</p> <p><a href="http://review.whamcloud.com/7778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7778</a></p> <p>Skip simul test for ZFS, since it takes too long.</p> <p><a href="http://review.whamcloud.com/7780" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7780</a> - FOR TEST ONLY patch to run parallel-scale with only 2 OSTs</p> <p>Hi Alex and Nathaniel,</p> <p>FYI, all of the following sub-tests run slowly on ZFS under the current test environment (7 OSTs share the same physical device):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sanityn: test 33a 13949s sanity-quota: test 2 16905s test 12a 2556s replay-ost-single: test 8a 2326s test 8b 2290s parallel-scale: test compilebench 6913s test simul 8727s test connectathon 6880s test iorssf 5570s test iorfpp 5960s test write_append_truncate 13940s test write_disjoint 29272s </pre> </div></div> <p>We'd better not to skip those tests for ZFS. We can improve the test scripts to reduce file sizes/counts, iteration times, etc. to make the tests still run for ZFS within a reasonable time (less than 1800s with SLOW=yes).</p> <p>can we reconfigure with OSTCOUNT=1 for parallel-scale specifically ?</p> <blockquote><p>can we reconfigure with OSTCOUNT=1 for parallel-scale specifically ?</p></blockquote> <p>This needs changes on autotest system. We can get the result of parallel-scale test with SLOW=no first to see whether those sub-tests can pass within a reasonable time or not.</p> <p>In manual test run with SLOW=no and OSTCOUNT=7, parallel-scale test simul passed in <b>112s</b>: <a href="https://maloo.whamcloud.com/test_sets/4b220e1e-28c7-11e3-8951-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4b220e1e-28c7-11e3-8951-52540035b04c</a><br/> However, the following sub-tests still took very long time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>compilebench 7760s iorssf 5061s iorfpp 5748s </pre> </div></div> <p>Let's wait for the autotest test result in <a href="http://review.whamcloud.com/7778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7778</a> to do a comparison.</p> <blockquote><p>Let's wait for the autotest test result in <a href="http://review.whamcloud.com/7778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7778</a> to do a comparison.</p></blockquote> <p>In autotest test run with SLOW=no and OSTCOUNT=7, parallel-scale timed out: <a href="https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>compilebench 6005s metabench 14400s (TIMEOUT) </pre> </div></div> <p>We still have to reduce the number of cbench_IDIRS, cbench_RUNS, mbench_NFILES, etc. for ZFS.</p> <p>From 7778 <a href="https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c</a><br/> the metabench result seems very strange.</p> <p>metabench normally runs fairly quickly (~200-500) judging by the results in maloo for ZFS runs.</p> <p>It looks like the whole system (client-30) just went out to lunch for 4 hours.</p> <p>Reduce performance expectation for ZFS in sanity-quota/0, lowest observed over last 4 weeks is ~150.</p> <p><a href="http://review.whamcloud.com/7848" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7848</a></p> <p>I would prefer the approach of setting OSTCOUNT=2 for ZFS-backed test filesystems. This is more in line with real systems, since we will typically only have a single OST per OSS with ZFS instead of 4 or more OSTs per OSS with ldiskfs. I think a lot of tests depend on having at least two OSTs, so OSTCOUNT=1 will probably cause some tests to be skipped.</p> <p>Another complimentary approach would be to format a single ZFS pool across a few LVs and then have the different OST/MDT targets in their own datasets in the shared pool. That would avoid the extra commits caused by having separate pools. The drawback is that all of the datasets would store their files into the same space, so some of the Lustre tests would be broken if we don't add ZFS reservations for the minimum size of the datasets (e.g. fill one OST and allocate objects to another OST would break).</p> <blockquote><p>I would prefer the approach of setting OSTCOUNT=2 for ZFS-backed test filesystems.</p></blockquote> <p>I just created TEI-790 to ask TEI team for help on this change.</p> <blockquote><p>Another complimentary approach would be to format a single ZFS pool across a few LVs and then have the different OST/MDT targets in their own datasets in the shared pool.</p></blockquote> <p>This needs change mdsdevname(), ostdevname() and the failover testing support codes for ZFS (<a href="http://review.whamcloud.com/6429" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6429</a>) in test-framework.sh.</p> <p>Since TEI-790 was fixed, I've triggered a full group test session on ZFS against Lustre b2_4 build #47. I'll vet the test results to see whether the timed-out tests can pass or not with OSTCOUNT=2 and SLOW=yes.</p> <p>Here is the test result on Lustre b2_4 build #47 with FSTYPE=zfs, OSTCOUNT=2 and SLOW=yes:<br/> <a href="https://maloo.whamcloud.com/test_sessions/05b82736-444d-11e3-8472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/05b82736-444d-11e3-8472-52540035b04c</a></p> <p>Timeout failures with OSTCOUNT=7 on the following sub-tests disappeared with OSTCOUNT=2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sanity-benchmark test bonnie LU-1960 replay-ost-single test 5 LU-2887 sanity-quota test 7a and 12a LU-2887 large-scale test 3a LU-2887 obdfilter-survey test 1a LU-2124 </pre> </div></div> <p>However, timeout failure on sanityn test 33a (<a href="https://jira.whamcloud.com/browse/LU-2829" title="Timeout on sanityn test_33a: zfs slow when commit_on_sharing enabled" class="issue-link" data-issue-key="LU-2829"><del>LU-2829</del></a>) still occurred.</p> <p>And because of <a href="https://jira.whamcloud.com/browse/LU-3906" title="Failure on test suite parallel-scale test_compilebench: IOError, No space left on device" class="issue-link" data-issue-key="LU-3906"><del>LU-3906</del></a> (out of space issue), parallel-scale, parallel-scale-nfsv3 and parallel-scale-nfsv4 tests were not really run, then we do not know whether the timeout failures on them disappear or not with OSTCOUNT=2.</p> <blockquote><p><a href="http://review.whamcloud.com/7778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7778</a></p></blockquote> <p>The above patch and <a href="http://review.whamcloud.com/8234" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8234</a> have landed on master branch. They are also needed on Lustre b2_4 and b2_5 branches.</p> <p>The two patches are combined and back-ported to Lustre b2_4 branch: <a href="http://review.whamcloud.com/8284" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8284</a></p> <p>Patch <a href="http://review.whamcloud.com/8284" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8284</a> landed on Lustre b2_4 branch. I'll trigger ZFS full group test session on Lustre b2_4 build #57 to check the test results. </p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/58/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/58/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=2097152</p> <p>replay-ost-single test 5 still timed out: <a href="https://maloo.whamcloud.com/test_sets/a76a6b78-5606-11e3-8e94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a76a6b78-5606-11e3-8e94-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>txg_sync D 0000000000000000 0 32309 2 0x00000080 ffff88004925dba0 0000000000000046 ffff8800ffffffff 00001e6701abaaf4 000000005cd72ac8 ffff88007126d2f0 00000000001ec6e4 ffffffffad6d9703 ffff880072f07ab8 ffff88004925dfd8 000000000000fb88 ffff880072f07ab8 Call Trace: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff8150ed93&gt;] io_schedule+0x73/0xc0 [&lt;ffffffffa03e6d4c&gt;] cv_wait_common+0x8c/0x100 [spl] [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa03e3717&gt;] ? taskq_dispatch_ent+0x57/0x110 [spl] [&lt;ffffffffa03e6dd8&gt;] __cv_wait_io+0x18/0x20 [spl] [&lt;ffffffffa052939b&gt;] zio_wait+0xfb/0x190 [zfs] [&lt;ffffffffa04c1f2c&gt;] dsl_pool_sync+0xec/0x540 [zfs] [&lt;ffffffffa04da82e&gt;] spa_sync+0x39e/0x970 [zfs] [&lt;ffffffff8103b8d9&gt;] ? kvm_clock_get_cycles+0x9/0x10 [&lt;ffffffffa04e582a&gt;] txg_sync_thread+0x27a/0x4b0 [zfs] [&lt;ffffffff810560a9&gt;] ? set_user_nice+0xc9/0x130 [&lt;ffffffffa04e55b0&gt;] ? txg_sync_thread+0x0/0x4b0 [zfs] [&lt;ffffffffa03e2a3f&gt;] thread_generic_wrapper+0x5f/0x70 [spl] [&lt;ffffffffa03e29e0&gt;] ? thread_generic_wrapper+0x0/0x70 [spl] [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/67/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/67/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=8388608</p> <p>performance-sanity test 4 timed out in 28800s:<br/> <a href="https://maloo.whamcloud.com/test_sets/68896e68-6579-11e3-8518-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/68896e68-6579-11e3-8518-52540035b04c</a></p> <p>parallel-scale test metabench timed out in 14400s:<br/> <a href="https://maloo.whamcloud.com/test_sets/bdff6c44-6579-11e3-8518-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bdff6c44-6579-11e3-8518-52540035b04c</a></p> <p>Landed for 2.4.2 and 2.6. Will land for 2.5.1 shortly.</p> <p>sanity-quota.sh test_12a is still being skipped on ZFS due to this bug, reopen until problem is actually fixed. At a minimum, the skip should only be done in the case of VM nodes, or ZFS-on-LVM or something.</p> <p>ZIL will improve this</p> Blocker LU-4009 Duplicate LU-4108 Related LU-2176 LU-2872 LU-3089 LU-3225 LU-2085 LU-1960 LU-2829 LU-2874 LU-3109 LU-2836 LU-5148 LU-2891 LU-2955 LU-4072 LU-4444 LU-2124 LU-4950 Development Rank 1|hzvk0v: Rank (Obsolete) 6968 Severity