[LU-2887] sanity-quota test_12a: slow due to ZFS VMs sharing single disk

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1
• Labels: MB performance zfs
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c.

The sub-test test_12a failed with the following error:

test failed to respond and timed out

Info required for matching: sanity-quota 12a

Looking through test 12a, things seem to have hung up on the runas dd (with oflag=sync) at the end of the test.

OST has threads that are blocked on disk I/O (oss dmesg):

txg_sync      D 0000000000000000     0 24236      2 0x00000080
 ffff88005027dbc0 0000000000000046 ffff88004b906ec0 0000000000000086
 ffff88005027db70 ffff88007c7d4408 0000000000000001 ffff88007c7d4420
 ffff880052101058 ffff88005027dfd8 000000000000fb88 ffff880052101058
Call Trace:
 [<ffffffff81090b9e>] ? prepare_to_wait_exclusive+0x4e/0x80
 [<ffffffffa016b5ac>] cv_wait_common+0x9c/0x1a0 [spl]
 [<ffffffffa02d5160>] ? zio_execute+0x0/0xf0 [zfs]
 [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa016b6e3>] __cv_wait+0x13/0x20 [spl]
 [<ffffffffa02d533b>] zio_wait+0xeb/0x160 [zfs]
 [<ffffffffa026b807>] dsl_pool_sync+0x2a7/0x480 [zfs]
 [<ffffffffa027e147>] spa_sync+0x397/0x9a0 [zfs]
 [<ffffffffa028fd41>] txg_sync_thread+0x2c1/0x490 [zfs]
 [<ffffffff810527f9>] ? set_user_nice+0xc9/0x130
 [<ffffffffa028fa80>] ? txg_sync_thread+0x0/0x490 [zfs]
 [<ffffffffa0164668>] thread_generic_wrapper+0x68/0x80 [spl]
 [<ffffffffa0164600>] ? thread_generic_wrapper+0x0/0x80 [spl]
 [<ffffffff81090626>] kthread+0x96/0xa0
 [<ffffffff8100c0ca>] child_rip+0xa/0x20
 [<ffffffff81090590>] ? kthread+0x0/0xa0
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
ll_ost_io00_0 D 0000000000000000     0 18170      2 0x00000080
 ffff8800427b9820 0000000000000046 0000000000000046 0000000000000001
 ffff8800427b98b0 0000000000000086 ffff8800427b97e0 ffff88005027dd60
 ffff8800427b7ab8 ffff8800427b9fd8 000000000000fb88 ffff8800427b7ab8
Call Trace:
 [<ffffffff81090b9e>] ? prepare_to_wait_exclusive+0x4e/0x80
 [<ffffffffa016b5ac>] cv_wait_common+0x9c/0x1a0 [spl]
 [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa016b6e3>] __cv_wait+0x13/0x20 [spl]
 [<ffffffffa028f573>] txg_wait_synced+0xb3/0x190 [zfs]
 [<ffffffffa0c71015>] osd_trans_stop+0x365/0x420 [osd_zfs]
 [<ffffffffa0cb9062>] ofd_trans_stop+0x22/0x60 [ofd]
 [<ffffffffa0cbdf06>] ofd_commitrw_write+0x406/0x11b0 [ofd]
 [<ffffffffa0cbf13d>] ofd_commitrw+0x48d/0x920 [ofd]
 [<ffffffffa085b708>] obd_commitrw+0x128/0x3d0 [ost]
 [<ffffffffa0862599>] ost_brw_write+0xe49/0x14d0 [ost]
 [<ffffffff812739b6>] ? vsnprintf+0x2b6/0x5f0
 [<ffffffffa088c1f0>] ? target_bulk_timeout+0x0/0xc0 [ptlrpc]
 [<ffffffffa08680e3>] ost_handle+0x31e3/0x46f0 [ost]
 [<ffffffffa05ca154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
 [<ffffffffa08dc02c>] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
 [<ffffffffa05be5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
 [<ffffffffa08d3759>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
 [<ffffffff81052223>] ? __wake_up+0x53/0x70
 [<ffffffffa08dd576>] ptlrpc_main+0xb76/0x1870 [ptlrpc]
 [<ffffffffa08dca00>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
 [<ffffffff8100c0ca>] child_rip+0xa/0x20
 [<ffffffffa08dca00>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
 [<ffffffffa08dca00>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

Issue Links

• is blocked by: LU-4009 Add ZIL support to osd-zfs
• is duplicated by: LU-4108 Failure on test suite performance-sanity test_4
• is related to: LU-2829 Timeout on sanityn test_33a: zfs slow when commit_on_sharing enabled
• is related to: LU-2874 Test timeout failure on test suite replay-ost-single test_8a: timeout on wait for dd
• is related to: LU-3109 ZFS - very slow reads, OST watchdogs
• is related to: LU-2836 Test failure on test suite sanity-quota, subtest test_3
• is related to: LU-5148 OSTs won't mount following upgrade to 2.4.2
• is related to: LU-2891 Test failure on test suite sanity-quota, subtest test_0: slow zfs dd
• is related to: LU-2955 replay-ost-single 8b: Hung until Autotest timed out
• is related to: LU-4072 sanity, subtest test_24v takes a VERY LONG TIME on ZFS
• is related to: LU-4444 conf-sanity test_69: ZFS took too long to create 100k files
• is related to: LU-2124 Test failure on test suite obdfilter-survey, subtest test_1a
• is related to: LU-4950 sanity-benchmark test fsx hung: txg_sync was stuck on OSS
• is related to: LU-2176 ZFS: running racer grounds everything to a standstill
• is related to: LU-2872 Test timeout failure on test suite sanity-quota test_1
• is related to: LU-3089 Test failure recovery-small test_55: dd should be finished!
• is related to: LU-3225 Timeout failure on test suite sanity-quota, subtest test_19
• is related to: LU-2085 sanityn test_16 (fsx) ran over its Autotest time
• is related to: LU-1960 Test failure on test suite sanity-benchmark, subtest test_bonnie