https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-802 Lustre <p>This issue was created by maloo for Chris Gearing &lt;chris@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/0219bc62-00f6-11e1-bb4f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0219bc62-00f6-11e1-bb4f-52540025f9af</a>.</p> <p>See dmesg client vm1</p> <p>The sub-test test_write_append_truncate failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: parallel-scale write_append_truncate</p> LU-802

Test failure on test suite parallel-scale, subtest test_write_append_truncate

Bug Blocker Closed Fixed Chris Gearing Maloo Fri, 28 Oct 2011 04:01:50 +0000 Thu, 31 Oct 2013 09:33:31 +0000 Thu, 31 Oct 2013 09:33:31 +0000 Lustre 2.2.0 Lustre 2.1.1 Lustre 2.1.2 Lustre 1.8.8 Lustre 2.4.1 0 5 <p>Lustre Tag: v2_1_1_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/41/</a><br/> Distro/Arch: RHEL6/x86_64(server), RHEL6/i686(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/e04b00fe-55d3-11e1-9aa8-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e04b00fe-55d3-11e1-9aa8-5254004bbbd3</a></p> <p>The stack trace on Client 1 (client-11vm1) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:26:40:write_append_ S 0002b90d 0 23901 23898 0x00000080 12:26:40: f63c5570 00000086 4f381feb 0002b90d 00000000 000003c8 00005d5d 0000a53a 12:26:40: 00000000 c3318200 00002643 00000000 00002643 c0b0f0c0 c0b0f0c0 f63c5818 12:26:40: c0b0f0c0 c0b0aaa4 c0b0f0c0 f63c5818 c69b4000 f98351ed f982663f 0000008f 12:26:40:Call Trace: 12:26:40: [&lt;f980de2e&gt;] ? cl_lock_mutex_put+0x3e/0x80 [obdclass] 12:26:40: [&lt;f980e328&gt;] ? cl_lock_state_wait+0x1d8/0x340 [obdclass] 12:26:40: [&lt;f980b8f5&gt;] ? cl_lock_used_mod+0x25/0x60 [obdclass] 12:26:40: [&lt;f980bbad&gt;] ? cl_lock_user_add+0x4d/0x200 [obdclass] 12:26:40: [&lt;c044d4d0&gt;] ? default_wake_function+0x0/0x10 12:26:40: [&lt;f9813969&gt;] ? cl_enqueue_locked+0x1a9/0x2d0 [obdclass] 12:26:40: [&lt;f93faee6&gt;] ? lov_io_call+0x76/0x250 [lov] 12:26:40: [&lt;f9813d69&gt;] ? cl_lock_request+0x89/0x2d0 [obdclass] 12:26:40: [&lt;f93fccc2&gt;] ? lov_io_lock+0x72/0x260 [lov] 12:26:40: [&lt;f89ec572&gt;] ? vvp_io_init+0xe2/0x380 [lustre] 12:26:40: [&lt;f84c4eaf&gt;] ? libcfs_debug_vmsg2+0x34f/0x820 [libcfs] 12:26:40: [&lt;f981bbe0&gt;] ? cl_io_lock+0x4e0/0xa10 [obdclass] 12:26:40: [&lt;f9814fee&gt;] ? cl_io_iter_init+0x6e/0x240 [obdclass] 12:26:41: [&lt;f981c232&gt;] ? cl_io_loop+0x122/0x2a0 [obdclass] 12:26:41: [&lt;f896497a&gt;] ? ll_file_io_generic+0x41a/0x6c0 [lustre] 12:26:41: [&lt;f9801af6&gt;] ? cl_env_get+0x16/0x480 [obdclass] 12:26:41: [&lt;f84c8ce2&gt;] ? cfs_hash_dual_bd_unlock+0x22/0x50 [libcfs] 12:26:41: [&lt;f84ccc4e&gt;] ? cfs_hash_find_or_add+0x7e/0x160 [libcfs] 12:26:41: [&lt;f8964d41&gt;] ? ll_file_aio_read+0x121/0x4d0 [lustre] 12:26:41: [&lt;c052ee2e&gt;] ? cp_new_stat64+0xee/0x100 12:26:41: [&lt;f8970c65&gt;] ? ll_file_read+0x165/0x440 [lustre] 12:26:41: [&lt;c05a24fc&gt;] ? security_file_permission+0xc/0x10 12:26:41: [&lt;c052a656&gt;] ? rw_verify_area+0x66/0xe0 12:26:41: [&lt;f8970b00&gt;] ? ll_file_read+0x0/0x440 [lustre] 12:26:41: [&lt;c052b01d&gt;] ? vfs_read+0x9d/0x190 12:26:41: [&lt;c04afccc&gt;] ? audit_syscall_entry+0x21c/0x240 12:26:41: [&lt;c052b151&gt;] ? sys_read+0x41/0x70 12:26:41: [&lt;c0409a9f&gt;] ? sysenter_do_call+0x12/0x28 </pre> </div></div> <p>Jinshan</p> <p>Could you please look at this one?</p> <p>Thanks</p> <p>Peter</p> <p>From the debug log, it seems everything was fine. How much time was set for this test? Let's try longer time before timing it out.</p> <p>&gt; From the debug log, it seems everything was fine. How much time was set for this test? Let's try longer time before timing it out.</p> <p>It seems the TIMEOUT value set by the autotest system is 3600.</p> <p>Per the following search result on Maloo:<br/> <a href="https://maloo.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;test_set[test_set_script_id]=b10ed7ea-55b4-11e0-bb3d-52540025f9af&amp;sub_test[sub_test_script_id]=6c0ca7dc-55d0-11e0-bb3d-52540025f9af&amp;sub_test[status]=&amp;sub_test[query_bugs]=&amp;test_session[test_host]=&amp;test_session[test_group]=&amp;test_session[user_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=1209600&amp;test_node[os_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[architecture_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node_network[network_type_id]=&amp;commit=Update+results" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;test_set[test_set_script_id]=b10ed7ea-55b4-11e0-bb3d-52540025f9af&amp;sub_test[sub_test_script_id]=6c0ca7dc-55d0-11e0-bb3d-52540025f9af&amp;sub_test[status]=&amp;sub_test[query_bugs]=&amp;test_session[test_host]=&amp;test_session[test_group]=&amp;test_session[user_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=1209600&amp;test_node[os_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[architecture_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node_network[network_type_id]=&amp;commit=Update+results</a></p> <p>The write_append_truncate test took very long time (large than 2000s) to run on the VMs, but took less than 10mins to run on the physical nodes with real devices:<br/> <a href="https://maloo.whamcloud.com/sub_tests/e59823a4-584c-11e1-9df1-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/e59823a4-584c-11e1-9df1-5254004bbbd3</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/4f4f93d0-579a-11e1-99fa-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/4f4f93d0-579a-11e1-99fa-5254004bbbd3</a></p> <p>Per the parallel-scale test script:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># # write_append_truncate # # threads per client write_THREADS=${write_THREADS:-8} write_REP=${write_REP:-10000} [ "$SLOW" = "no" ] &amp;&amp; write_REP=100 </pre> </div></div> <p>I think we could specify small values to write_REP/write_THREADS to reduce the run time on VMs by the autotest system.</p> <p>Thank you, yujian. Please let me know if you meet this problem again.</p> <p>got the similar issue: <a href="https://maloo.whamcloud.com/test_sets/9ff40214-6701-11e1-a9a4-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9ff40214-6701-11e1-a9a4-5254004bbbd3</a></p> <p>build lustre-master/493/RHEL6-x86_64</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/194/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/194/</a><br/> Distro/Arch: RHEL5.8/x86_64(server), RHEL6.2/x86_64(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>The similar issue occurred: <a href="https://maloo.whamcloud.com/test_sets/3c4bbb96-9b3e-11e1-a0a0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3c4bbb96-9b3e-11e1-a0a0-52540035b04c</a></p> <p>Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/86/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/86/</a><br/> Distro/Arch: RHEL6.2/x86_64(server), RHEL6.2/i686(client)<br/> Network: TCP (1GigE) <br/> ENABLE_QUOTA=yes</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/19c1ae10-aac9-11e1-bd84-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/19c1ae10-aac9-11e1-bd84-52540035b04c</a></p> <p>Lustre Client: 2.3.0<br/> Lustre Server: v2_1_4_RC1<br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/4f1c6314-4864-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4f1c6314-4864-11e2-8cdc-52540035b04c</a></p> <p>The timeout issue keeps occurring while testing Lustre 2.4.1 RC2:<br/> <a href="https://maloo.whamcloud.com/test_sets/8e327ac8-1796-11e3-8f44-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8e327ac8-1796-11e3-8f44-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/6edeb926-17b3-11e3-8f44-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6edeb926-17b3-11e3-8f44-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/da608a9e-1830-11e3-b39a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/da608a9e-1830-11e3-b39a-52540035b04c</a></p> <p>The test timed out in 3600s. However, the same test passed on Lustre 2.4.1 RC1 in about 5570s several days ago.</p> <p>Chris, could you please check whether the timeout mechanism in autotest was changed or not? Thanks.</p> <p>The timeout failure is preventing the remaining sub-tests in parallel-scale.sh from running. So, I flagged this ticket as a blocker.</p> <p>The parallel-scale write_append_truncate test passed in the latest full group test sessions without timeout issue. Let's close this ticket. If it hit timeout failure again, please feel free to reopen the ticket.</p> Development Rank 1|hzv3qv: Rank (Obsolete) 4130 Severity