https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9777 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/a9e1439e-689b-11e7-9230-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a9e1439e-689b-11e7-9230-5254006e85c2</a>.</p> <p>The sub-test test_70f failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: replay-single 70f</p> LU-9777

replay-single test_70f: test failed to respond and timed out

Bug Minor Open Unresolved WC Triage Maloo Fri, 14 Jul 2017 14:05:02 +0000 Fri, 13 Oct 2017 17:32:19 +0000 Lustre 2.10.1 0 4 <p>another one on 2.10.1 <a href="https://testing.hpdd.intel.com/test_sets/4e4e2c74-9b4d-11e7-ba27-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4e4e2c74-9b4d-11e7-ba27-5254006e85c2</a><br/> client console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10:01:57:[38268.794445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 5 times 10:01:57:[38268.956844] Lustre: DEBUG MARKER: test_70f failing OST 5 times 10:01:57:[38280.356600] INFO: task tee:16642 blocked for more than 120 seconds. 10:01:57:[38280.357275] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 10:01:57:[38280.358017] tee D ffffffff816a7610 0 16642 1505 0x00000080 10:01:57:[38280.358735] ffff88006740b9b0 0000000000000082 ffff880039d1cf10 ffff88006740bfd8 10:01:57:[38280.359563] ffff88006740bfd8 ffff88006740bfd8 ffff880039d1cf10 ffff88007fd16cc0 10:01:57:[38280.360369] 0000000000000000 7fffffffffffffff ffff88007ff5d770 ffffffff816a7610 10:01:57:[38280.361164] Call Trace: 10:01:57:[38280.361414] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:01:57:[38280.361903] [&lt;ffffffff816a94e9&gt;] schedule+0x29/0x70 10:01:57:[38280.362481] [&lt;ffffffff816a6ff9&gt;] schedule_timeout+0x239/0x2c0 10:01:57:[38280.363037] [&lt;ffffffff81062efe&gt;] ? kvm_clock_get_cycles+0x1e/0x20 10:01:57:[38280.363671] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:01:57:[38280.364184] [&lt;ffffffff816a8b6d&gt;] io_schedule_timeout+0xad/0x130 10:01:57:[38280.364760] [&lt;ffffffff816a8c08&gt;] io_schedule+0x18/0x20 10:01:57:[38280.365299] [&lt;ffffffff816a7621&gt;] bit_wait_io+0x11/0x50 10:01:57:[38280.365795] [&lt;ffffffff816a7145&gt;] __wait_on_bit+0x65/0x90 10:01:57:[38280.366349] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:01:57:[38280.366840] [&lt;ffffffff816a71f1&gt;] out_of_line_wait_on_bit+0x81/0xb0 10:01:57:[38280.367470] [&lt;ffffffff810b19d0&gt;] ? wake_bit_function+0x40/0x40 10:01:57:[38280.368041] [&lt;ffffffffc04f5db3&gt;] nfs_wait_on_request+0x33/0x40 [nfs] 10:01:57:[38280.368720] [&lt;ffffffffc04faeb3&gt;] nfs_updatepage+0x153/0x8d0 [nfs] 10:01:57:[38280.369338] [&lt;ffffffffc04ea371&gt;] nfs_write_end+0x141/0x350 [nfs] 10:01:57:[38280.369918] [&lt;ffffffff81182539&gt;] generic_file_buffered_write+0x189/0x2a0 10:01:57:[38280.370607] [&lt;ffffffff811848f2&gt;] __generic_file_aio_write+0x1e2/0x400 10:01:57:[38280.371266] [&lt;ffffffff8121cf41&gt;] ? update_time+0x81/0xd0 10:01:57:[38280.371776] [&lt;ffffffff81184b69&gt;] generic_file_aio_write+0x59/0xa0 10:01:57:[38280.372407] [&lt;ffffffffc04e97db&gt;] nfs_file_write+0xbb/0x1e0 [nfs] 10:01:57:[38280.372985] [&lt;ffffffff8120027d&gt;] do_sync_write+0x8d/0xd0 10:01:57:[38280.373548] [&lt;ffffffff81200d3d&gt;] vfs_write+0xbd/0x1e0 10:01:57:[38280.374036] [&lt;ffffffff81201b4f&gt;] SyS_write+0x7f/0xe0 10:01:57:[38280.374567] [&lt;ffffffff816b5009&gt;] system_call_fastpath+0x16/0x1b 10:01:57:[38285.330324] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u 10:01:57:[38285.797355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-49vm3.onyx.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid 10:01:57:[38285.989862] Lustre: DEBUG MARKER: onyx-49vm3.onyx.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid 10:01:57:[38286.082508] Lustre: DEBUG MARKER: lctl get_param -n at_max 10:01:57:[38287.271192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec 10:01:57:[38287.462276] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec 10:01:57:[38289.182270] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear 10:01:57:[38289.906118] Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.onyx-49vm4 10:01:57:[38290.090446] Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.onyx-49vm3.onyx.hpdd.intel.com 10:01:57:[38290.404005] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear 10:01:57:[38291.165204] Lustre: DEBUG MARKER: rm -f /tmp/f70f.replay-single 10:01:57:[38291.487042] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null 10:01:57:[38291.966735] Lustre: DEBUG MARKER: rc=0; val=$(/usr/sbin/lctl get_param -n catastrophe 2&gt;&amp;1); if [[ $? -eq 0 &amp;&amp; $val -ne 0 ]]; then echo $(hostname -s): $val; rc=$val; fi; exit $rc 10:01:57:[38292.648060] Lustre: DEBUG MARKER: dmesg 10:01:57:[38400.374599] INFO: task tee:16642 blocked for more than 120 seconds. 10:01:57:[38400.375292] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 10:01:57:[38400.376074] tee D ffffffff816a7610 0 16642 1505 0x00000080 10:01:57:[38400.376907] ffff88006740b9b0 0000000000000082 ffff880039d1cf10 ffff88006740bfd8 10:01:57:[38400.377727] ffff88006740bfd8 ffff88006740bfd8 ffff880039d1cf10 ffff88007fd16cc0 10:01:57:[38400.378543] 0000000000000000 7fffffffffffffff ffff88007ff5d770 ffffffff816a7610 10:09:48:[38400.379382] Call Trace: 10:09:48:[38400.379649] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:09:48:[38400.380240] [&lt;ffffffff816a94e9&gt;] schedule+0x29/0x70 10:09:48:[38400.380731] [&lt;ffffffff816a6ff9&gt;] schedule_timeout+0x239/0x2c0 10:09:48:[38400.381355] [&lt;ffffffff81062efe&gt;] ? kvm_clock_get_cycles+0x1e/0x20 10:09:48:[38400.381962] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:09:48:[38400.382518] [&lt;ffffffff816a8b6d&gt;] io_schedule_timeout+0xad/0x130 10:09:48:[38400.383104] [&lt;ffffffff816a8c08&gt;] io_schedule+0x18/0x20 10:09:48:[38400.383706] [&lt;ffffffff816a7621&gt;] bit_wait_io+0x11/0x50 10:09:48:[38400.384256] [&lt;ffffffff816a7145&gt;] __wait_on_bit+0x65/0x90 10:09:48:[38400.384785] [&lt;ffffffff816a7610&gt;] ? bit_wait+0x50/0x50 10:09:48:[38400.385332] [&lt;ffffffff816a71f1&gt;] out_of_line_wait_on_bit+0x81/0xb0 10:09:48:[38400.385951] [&lt;ffffffff810b19d0&gt;] ? wake_bit_function+0x40/0x40 10:09:48:[38400.386617] [&lt;ffffffffc04f5db3&gt;] nfs_wait_on_request+0x33/0x40 [nfs] 10:09:48:[38400.387252] [&lt;ffffffffc04faeb3&gt;] nfs_updatepage+0x153/0x8d0 [nfs] 10:09:48:[38400.387865] [&lt;ffffffffc04ea371&gt;] nfs_write_end+0x141/0x350 [nfs] 10:09:48:[38400.388544] [&lt;ffffffff81182539&gt;] generic_file_buffered_write+0x189/0x2a0 10:09:48:[38400.389209] [&lt;ffffffff811848f2&gt;] __generic_file_aio_write+0x1e2/0x400 10:09:48:[38400.389897] [&lt;ffffffff8121cf41&gt;] ? update_time+0x81/0xd0 10:09:48:[38400.390467] [&lt;ffffffff81184b69&gt;] generic_file_aio_write+0x59/0xa0 10:09:48:[38400.391071] [&lt;ffffffffc04e97db&gt;] nfs_file_write+0xbb/0x1e0 [nfs] 10:09:48:[38400.391756] [&lt;ffffffff8120027d&gt;] do_sync_write+0x8d/0xd0 10:09:48:[38400.392323] [&lt;ffffffff81200d3d&gt;] vfs_write+0xbd/0x1e0 10:09:48:[38400.392825] [&lt;ffffffff81201b4f&gt;] SyS_write+0x7f/0xe0 10:09:48:[38400.393368] [&lt;ffffffff816b5009&gt;] system_call_fastpath+0x16/0x1b 10:09:48:[38520.393616] INFO: task tee:16642 blocked for more than 120 seconds. 10:09:48:[38520.394291] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. </pre> </div></div> <p>+1 on master: <a href="https://testing.hpdd.intel.com/test_sets/141d3872-a018-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/141d3872-a018-11e7-b778-5254006e85c2</a></p> Related LU-10008 Development Rank 1|hzzgp3: Rank (Obsolete) 9223372036854775807 Severity