[LU-5775] obdfilter-survey test 1c: lctl hung on OSS

• Type: Bug
• Resolution: Duplicate
• Priority: Major
• Labels: zfs
• Environment: Lustre build: https://build.hpdd.intel.com/job/lustre-b2_5/96/; Distro/Arch: RHEL6.5/x86_64; FSTYPE=zfs
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Jian Yu

Description

While running obdfilter-survey test 1c with FSTYPE=zfs, lctl hung on OSS node as follows:

18:33:37:Lustre: DEBUG MARKER: == obdfilter-survey test 1c: Object Storage Targets survey, big batch == 22:47:03 (1413240423)
18:33:37:Lustre: DEBUG MARKER: lctl dl | grep obdfilter
18:33:37:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@'
18:33:37:Lustre: Echo OBD driver; http://www.lustre.org/
18:33:37:INFO: task lctl:2685 blocked for more than 120 seconds.
18:33:37:      Tainted: P           ---------------    2.6.32-431.29.2.el6_lustre.g9835a2a.x86_64 #1
18:33:37:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
18:33:37:lctl          D 0000000000000001     0  2685   2683 0x00000080
18:33:37: ffff880067553758 0000000000000086 ffff8800ffffffff 00002610ffda172f
18:33:37: ffff880067553778 ffff88006e7b4670 000000000057a0a4 ffffffffabc3c59b
18:33:37: ffff88007db06638 ffff880067553fd8 000000000000fbc8 ffff88007db06638
18:33:37:Call Trace:
18:33:37: [<ffffffff810a6d31>] ? ktime_get_ts+0xb1/0xf0
18:33:37: [<ffffffff81529ea3>] io_schedule+0x73/0xc0
18:33:37: [<ffffffffa014341c>] cv_wait_common+0x8c/0x100 [spl]
18:33:37: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
18:33:37: [<ffffffffa01434a8>] __cv_wait_io+0x18/0x20 [spl]
18:33:37: [<ffffffffa02890ab>] zio_wait+0xfb/0x1b0 [zfs]
18:33:37: [<ffffffffa01ff03d>] dmu_buf_hold_array_by_dnode+0x19d/0x4c0 [zfs]
18:33:37: [<ffffffffa01ffe68>] dmu_buf_hold_array_by_bonus+0x68/0x90 [zfs]
18:33:37: [<ffffffffa0df4b63>] osd_bufs_get+0x493/0xb00 [osd_zfs]
18:33:37: [<ffffffffa0eb5bdb>] ofd_preprw_read+0x14b/0x7f0 [ofd]
18:33:37: [<ffffffffa0eb69fa>] ofd_preprw+0x77a/0x1480 [ofd]
18:33:37: [<ffffffffa0fe1b42>] ? echo_client_iocontrol+0x1922/0x39d0 [obdecho]
18:33:37: [<ffffffff8116febc>] ? __kmalloc+0x20c/0x220
18:33:37: [<ffffffffa0fe22d0>] echo_client_iocontrol+0x20b0/0x39d0 [obdecho]
18:33:37: [<ffffffff8128d776>] ? vsnprintf+0x336/0x5e0
18:33:37: [<ffffffff8118d495>] ? chrdev_open+0x125/0x230
18:33:37: [<ffffffff811ab840>] ? mntput_no_expire+0x30/0x110
18:33:37: [<ffffffff8116febc>] ? __kmalloc+0x20c/0x220
18:33:37: [<ffffffffa071fb54>] class_handle_ioctl+0x12c4/0x1e50 [obdclass]
18:33:37: [<ffffffffa070d2ab>] obd_class_ioctl+0x4b/0x190 [obdclass]
18:33:37: [<ffffffff8119e992>] vfs_ioctl+0x22/0xa0
18:33:37: [<ffffffff8119eb34>] do_vfs_ioctl+0x84/0x580
18:33:37: [<ffffffff8119f0b1>] sys_ioctl+0x81/0xa0
18:33:37: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Maloo report: https://testing.hpdd.intel.com/test_sets/9cad37c4-5367-11e4-a9db-5254006e85c2

Issue Links

• duplicates: LU-5278 ZFS - many OST watchdogs with IOR
• duplicates: LU-3395 Test timeout on test suite obdfilter-survey, subtest test_1c