Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-2124] Test failure on test suite obdfilter-survey, subtest test_1a https://jira.whamcloud.com/browse/LU-2124 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/584999d6-1207-11e2-a663-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/584999d6-1207-11e2-a663-52540035b04c</a>.</p> <p>The sub-test test_1a failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: obdfilter-survey 1a</p> LU-2124 Test failure on test suite obdfilter-survey, subtest test_1a Bug Blocker Closed Fixed WC Triage Maloo Tue, 9 Oct 2012 10:26:19 +0000 Mon, 4 Nov 2013 10:42:50 +0000 Mon, 4 Nov 2013 10:42:50 +0000 Lustre 2.4.0 Lustre 2.4.1 0 6 <p>OST console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>21:57:00:Lustre: DEBUG MARKER: == obdfilter-survey test 1a: Object Storage Targets survey =========================================== 21:56:49 (1349758609) 21:57:00:Lustre: DEBUG MARKER: lctl dl | grep obdfilter 21:57:00:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' 22:48:59:hrtimer: interrupt took 55369 ns </pre> </div></div> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a> (2.4.1 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>obdfilter-survey test 1a hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== obdfilter-survey test 1a: Object Storage Targets survey == 23:44:44 (1378622684) CMD: client-24vm4 lctl dl | grep obdfilter CMD: client-24vm4 /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' + NETTYPE=tcp thrlo=8 nobjhi=1 thrhi=16 size=1024 case=disk rslt_loc=/tmp targets="10.10.4.119:lustre-OST0000 10.10.4.119:lustre-OST0001 10.10.4.119:lustre-OST0002 10.10.4.119:lustre-OST0003 10.10.4.119:lustre-OST0004 10.10.4.119:lustre-OST0005 10.10.4.119:lustre-OST0006" /usr/bin/obdfilter-survey Warning: Permanently added '10.10.4.119' (RSA) to the list of known hosts. Sat Sep 7 23:44:51 PDT 2013 Obdfilter-survey for case=disk from client-24vm2.lab.whamcloud.com </pre> </div></div> <p>Dmesg on OSS node client-24vm4 showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl D 0000000000000000 0 19552 19496 0x00000080 ffff88001bf65748 0000000000000086 ffff8800ffffffff 0000126bad99a78e ffff880061356070 ffff8800618efec0 00000000003e8684 ffffffffadd3ec96 ffff88001fefdaf8 ffff88001bf65fd8 000000000000fb88 ffff88001fefdaf8 Call Trace: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff8150ed03&gt;] io_schedule+0x73/0xc0 [&lt;ffffffffa03e6d4c&gt;] cv_wait_common+0x8c/0x100 [spl] [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa03e6dd8&gt;] __cv_wait_io+0x18/0x20 [spl] [&lt;ffffffffa052939b&gt;] zio_wait+0xfb/0x190 [zfs] [&lt;ffffffffa049f07d&gt;] dmu_buf_hold_array_by_dnode+0x1dd/0x560 [zfs] [&lt;ffffffffa049ff88&gt;] dmu_buf_hold_array_by_bonus+0x68/0x90 [zfs] [&lt;ffffffffa0dc1b33&gt;] osd_bufs_get+0x493/0xa30 [osd_zfs] [&lt;ffffffffa0e609cb&gt;] ofd_preprw_read+0x14b/0x7f0 [ofd] [&lt;ffffffffa0e617ea&gt;] ofd_preprw+0x77a/0x1480 [ofd] [&lt;ffffffffa05a7473&gt;] echo_client_iocontrol+0x2003/0x3b40 [obdecho] [&lt;ffffffff81281826&gt;] ? vsnprintf+0x336/0x5e0 [&lt;ffffffffa071049f&gt;] class_handle_ioctl+0x12ff/0x1ec0 [obdclass] [&lt;ffffffffa06f82ab&gt;] obd_class_ioctl+0x4b/0x190 [obdclass] [&lt;ffffffff81195352&gt;] vfs_ioctl+0x22/0xa0 [&lt;ffffffff8103c7d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 [&lt;ffffffff811954f4&gt;] do_vfs_ioctl+0x84/0x580 [&lt;ffffffff81195a71&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/f9d6f946-18ab-11e3-aa54-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f9d6f946-18ab-11e3-aa54-52540035b04c</a></p> <p>The same failure also occurred on previous Lustre b2_4 builds:<br/> <a href="https://maloo.whamcloud.com/test_sets/0ce085aa-169c-11e3-aa2a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0ce085aa-169c-11e3-aa2a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/92b17690-16b4-11e3-8c83-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/92b17690-16b4-11e3-8c83-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f1befe08-1657-11e3-aa2a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f1befe08-1657-11e3-aa2a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/ecb6f352-1409-11e3-980d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ecb6f352-1409-11e3-980d-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/e10c9e46-13f3-11e3-9e61-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e10c9e46-13f3-11e3-9e61-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/46/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/46/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/42ba0f84-3064-11e3-b28a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/42ba0f84-3064-11e3-b28a-52540035b04c</a></p> <p>We'll see whether the timeout failure can disappear after TEI-790 is resolved.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> FSTYPE=zfs</p> <p>With OSTCOUNT=2, obdfilter-survey test 1a passed:<br/> <a href="https://maloo.whamcloud.com/test_sets/a488f632-4453-11e3-8472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a488f632-4453-11e3-8472-52540035b04c</a></p> <p>Let's close this ticket.</p> Related LU-2887 Development Rank 1|hzv9tj: Rank (Obsolete) 5125 Severity