https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12397 Lustre <p>There is a race around opd_new_connection causing osp_precreate_thread to infinite wait :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> l_wait_event(d-&gt;opd_pre_waitq, !osp_precreate_running(d) || d-&gt;opd_new_connection, &amp;lwi); </pre> </div></div> LU-12397

osp: race around opd_new_connection

Bug Major Resolved Fixed Sergey Cheremencev Sergey Cheremencev osp Thu, 6 Jun 2019 11:17:49 +0000 Thu, 23 Mar 2023 18:25:54 +0000 Fri, 25 Sep 2020 03:57:54 +0000 Lustre 2.14.0 0 5 <p>Below is scenario how this race can happen.</p> <p>Disk error caused to hung one of OSTs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 11 19:59:39 lstrn09 kernel: Call Trace: Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff8163a089&gt;] schedule+0x29/0x70 Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff814ba2a5&gt;] md_make_request+0xb5/0x370 Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff812c7342&gt;] generic_make_request+0xe2/0x130 Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff812c7401&gt;] submit_bio+0x71/0x150 Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa157612c&gt;] osd_submit_bio+0x1c/0x60 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa1578577&gt;] osd_do_bio.isra.26+0x3d7/0x830 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa15790f2&gt;] osd_write_commit+0x322/0x900 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa15f1ec0&gt;] ofd_commitrw_write.isra.32+0xc60/0x1c20 [ofd] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa15f5d42&gt;] ofd_commitrw+0x512/0xac0 [ofd] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa0ef3a9b&gt;] obd_commitrw.constprop.39+0x2f8/0x33b [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa0ee0530&gt;] tgt_brw_write+0x1010/0x1720 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa0edc3db&gt;] tgt_request_handle+0x8fb/0x11f0 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa0e7edab&gt;] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffffa0e826db&gt;] ptlrpc_main+0xc0b/0x2060 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff810a5b8f&gt;] kthread+0xcf/0xe0 Mar 11 19:59:39 lstrn09 kernel: [&lt;ffffffff81644fd8&gt;] ret_from_fork+0x58/0x90 </pre> </div></div> <p>Write and read requests couldn't be finished after disk failure.</p> <p>At the same time MDT000 and MDT0001 started reporting hung tasks traces like below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 11 20:11:36 lstrn02 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 11 20:11:36 lstrn02 kernel: mdt00_001 D ffff880f765211d8 0 24438 2 0x00000000 Mar 11 20:11:36 lstrn02 kernel: ffff880f77b635a8 0000000000000046 ffff88102b0d6780 ffff880f77b63fd8 Mar 11 20:11:36 lstrn02 kernel: ffff880f77b63fd8 ffff880f77b63fd8 ffff88102b0d6780 ffff88102b0d6780 Mar 11 20:11:36 lstrn02 kernel: ffff880f765211c8 ffff880f765211d0 ffffffff00000000 ffff880f765211d8 Mar 11 20:11:36 lstrn02 kernel: Call Trace: Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffff8163a089&gt;] schedule+0x29/0x70 Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffff8163b845&gt;] rwsem_down_write_failed+0x115/0x220 Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffff813019f3&gt;] call_rwsem_down_write_failed+0x13/0x20 Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa15d9d1f&gt;] lod_qos_prep_create+0x111f/0x1fbc [lod] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa15d35dc&gt;] lod_declare_striped_object+0x1ec/0x790 [lod] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa15d4bf1&gt;] lod_declare_object_create+0x231/0x4b0 [lod] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa13372ff&gt;] mdd_declare_object_create_internal+0xdf/0x2f0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa132b328&gt;] mdd_declare_create+0x48/0xef0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa132c959&gt;] mdd_create+0x789/0x12a0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa13b1dd2&gt;] mdt_reint_open+0x1f92/0x2e00 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa13a4e60&gt;] mdt_reint_rec+0x80/0x210 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa1385281&gt;] mdt_reint_internal+0x5e1/0xb40 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa1385942&gt;] mdt_intent_reint+0x162/0x430 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa1389455&gt;] mdt_intent_opc+0x215/0xb50 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa13911a8&gt;] mdt_intent_policy+0x138/0x320 [mdt] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0df955a&gt;] ldlm_lock_enqueue+0x35a/0x9c0 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0e241b3&gt;] ldlm_handle_enqueue0+0x9c3/0x1830 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0eb1712&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0eb63db&gt;] tgt_request_handle+0x8fb/0x11f0 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0e58dab&gt;] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffffa0e5c6db&gt;] ptlrpc_main+0xc0b/0x2060 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffff810a5b8f&gt;] kthread+0xcf/0xe0 Mar 11 20:11:36 lstrn02 kernel: [&lt;ffffffff81644fd8&gt;] ret_from_fork+0x58/0x90 </pre> </div></div> <p>It is the result of inability to create new objects on OST000a(from lstrn09).</p> <p>Hung task's messages were supposed to disappear after lstrn09 failover. But continued to present in messages until lstrn02 crash due to a bug in osp around opd_new_connection. </p> <p>Sergey Cheremencev (c17829@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35078" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35078</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12397" title="osp: race around opd_new_connection" class="issue-link" data-issue-key="LU-12397"><del>LU-12397</del></a> osp: always set opd_new_connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 83381358299125d4b125c017d74172fe3a130d85</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35078/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35078/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12397" title="osp: race around opd_new_connection" class="issue-link" data-issue-key="LU-12397"><del>LU-12397</del></a> osp: always set opd_new_connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1b5abf625462a2b66820b2d07e25619afba504c6</p> <p>Landed for 2.14</p> <p>"Yang Sheng &lt;ys@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47805" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47805</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12397" title="osp: race around opd_new_connection" class="issue-link" data-issue-key="LU-12397"><del>LU-12397</del></a> osp: remove opd_new_connection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2d06a167ac46715b45c8677a1a53b64d95e3e5ff</p> Duplicate LU-16578 Related Development Rank 1|i00hpr: Rank (Obsolete) 9223372036854775807 Severity