https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2550 Lustre <p>sanity test 122 hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 122: fail client bulk callback (shouldn't LBUG) ========= 23:10:22 (1356678622) fail_loc=0x508 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00131226 s, 390 kB/s </pre> </div></div> <p>Console log on the client node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:13:02:INFO: task sync:19343 blocked for more than 120 seconds. 23:13:02:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 23:13:02:sync D 0000000000000002 0 19343 19221 0x00000080 23:13:02: ffff880302affc98 0000000000000082 0000000000000000 0000000000000000 23:13:02: 0000000000000000 0000000000000000 ffff880302affc68 ffffffff810097cc 23:13:02: ffff88031fb71af8 ffff880302afffd8 000000000000fb88 ffff88031fb71af8 23:13:02:Call Trace: 23:13:02: [&lt;ffffffff810097cc&gt;] ? __switch_to+0x1ac/0x320 23:13:02: [&lt;ffffffff811141f0&gt;] ? sync_page+0x0/0x50 23:13:02: [&lt;ffffffff814fe0f3&gt;] io_schedule+0x73/0xc0 23:13:02: [&lt;ffffffff8111422d&gt;] sync_page+0x3d/0x50 23:13:02: [&lt;ffffffff814feaaf&gt;] __wait_on_bit+0x5f/0x90 23:13:02: [&lt;ffffffff81114463&gt;] wait_on_page_bit+0x73/0x80 23:13:02: [&lt;ffffffff81092110&gt;] ? wake_bit_function+0x0/0x50 23:13:02: [&lt;ffffffff8112a965&gt;] ? pagevec_lookup_tag+0x25/0x40 23:13:02: [&lt;ffffffff811148db&gt;] wait_on_page_writeback_range+0xfb/0x190 23:13:02: [&lt;ffffffff814fe47c&gt;] ? wait_for_common+0x14c/0x180 23:13:02: [&lt;ffffffff81060250&gt;] ? default_wake_function+0x0/0x20 23:13:02: [&lt;ffffffff8111499f&gt;] filemap_fdatawait+0x2f/0x40 23:13:02: [&lt;ffffffff811a4874&gt;] sync_inodes_sb+0x114/0x190 23:13:02: [&lt;ffffffff811aa312&gt;] __sync_filesystem+0x82/0x90 23:13:02: [&lt;ffffffff811aa418&gt;] sync_filesystems+0xf8/0x130 23:13:02: [&lt;ffffffff811aa4b1&gt;] sys_sync+0x21/0x40 23:13:02: [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/154e712e-514f-11e2-b56e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/154e712e-514f-11e2-b56e-52540035b04c</a></p> Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/236/">http://build.whamcloud.com/job/lustre-b1_8/236/</a><br/> Distro/Arch: RHEL5.8/x86_64(server), RHEL6.3/x86_64(client)<br/> Network: IB (in-kernel OFED)<br/> LU-2550

sanity test 122 hung

Bug Blocker Resolved Fixed Niu Yawei Jian Yu LB Sun, 30 Dec 2012 21:51:19 +0000 Sun, 17 Feb 2013 11:04:28 +0000 Sun, 17 Feb 2013 01:36:54 +0000 Lustre 2.4.0 Lustre 2.1.4 Lustre 2.1.5 Lustre 1.8.8 Lustre 2.4.0 Lustre 2.1.5 Lustre 1.8.9 0 5 <p>This is a regression issue on Lustre b1_8 branch.</p> <p>The issue can be reproduced on Lustre b1_8 build #236:<br/> <a href="https://maloo.whamcloud.com/test_sessions/c8649180-57a3-11e2-8b17-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/c8649180-57a3-11e2-8b17-52540035b04c</a></p> <p>Another instance on Lustre b1_8 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/00323a48-5774-11e2-8772-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/00323a48-5774-11e2-8772-52540035b04c</a></p> <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Looks there is something wrong in the resend: in osc_brw_redo_request(), each time we rebuild a request, but never increase the aa_resends for the new request, so the io request will be resent infinitely if it always get -EIO (in this test). I'm wondering why it happened only on b_8? b2_1 &amp; master should have the same problem.</p> <p>The test in master(&amp; b2_1) doesn't have 'sync' after dd, I think that's why this bug isn't triggered in master.</p> <p>patch for b1_8: <a href="http://review.whamcloud.com/4964" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4964</a></p> <p>patch for master: <a href="http://review.whamcloud.com/5012" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5012</a></p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/242" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/242</a></p> <p>The issue still occurred: <a href="https://maloo.whamcloud.com/test_sets/327f6338-5ffb-11e2-84d4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/327f6338-5ffb-11e2-84d4-52540035b04c</a></p> <p>The above build contains patch <a href="http://review.whamcloud.com/4964" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4964</a>.</p> <p>The OBD_FAIL_PTLRPC_CLIENT_BULK_CB will result in rq_net_error set on the request, and trigger reconnect immediately... I think the test script needs be improved as well.</p> <p><a href="http://review.whamcloud.com/5050" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5050</a></p> <p>patch for b2_1: <a href="http://review.whamcloud.com/5184" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5184</a></p> <p>patches landed.</p> Development Rank 1|hzveef: Rank (Obsolete) 5974 Severity