https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12136 Lustre <p>We had an issue yesterday on Oak storage with Lustre 2.10.6. MDT0 didn't crash but filesystem got stuck. Several stack traces showed up on <tt>oak-md1-s2</tt> (serving MDT0000). Note: Oak uses DNE1 and another MDT0001 is mounted on <tt>oak-md1-s1</tt>, but I didn't find any stack trace on this one. A restart of MDT0000 fixed the issue (after a workaround to mitigate <a href="https://jira.whamcloud.com/browse/LU-8992" title="osd_object_release() LBUG" class="issue-link" data-issue-key="LU-8992"><del>LU-8992</del></a>).</p> <p>My short-term plan is to upgrade Oak to 2.10.7 in a rolling fashion but I thought it would be of interest to have a ticket to track this issue. I'm also attaching kernel logs from this server as oak-md1-s2-kernel.log where all stack traces can be seen.</p> <p>First call trace was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 29 09:38:38 oak-md1-s2 kernel: INFO: task mdt00_003:3491 blocked for more than 120 seconds. Mar 29 09:38:38 oak-md1-s2 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 29 09:38:38 oak-md1-s2 kernel: mdt00_003 D ffffffff00000000 0 3491 2 0x00000080 Mar 29 09:38:38 oak-md1-s2 kernel: ffff88201e3f74b8 0000000000000046 ffff88201e3c3f40 ffff88201e3f7fd8 Mar 29 09:38:38 oak-md1-s2 kernel: ffff88201e3f7fd8 ffff88201e3f7fd8 ffff88201e3c3f40 ffff88201e3c3f40 Mar 29 09:38:38 oak-md1-s2 kernel: ffff88101fc13248 ffff88101fc13240 fffffffe00000001 ffffffff00000000 Mar 29 09:38:38 oak-md1-s2 kernel: Call Trace: Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff816a94e9&gt;] schedule+0x29/0x70 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff816aadd5&gt;] rwsem_down_write_failed+0x225/0x3a0 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff81332047&gt;] call_rwsem_down_write_failed+0x17/0x30 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff816a87cd&gt;] down_write+0x2d/0x3d Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc121b34f&gt;] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0fa49a1&gt;] ? qsd_op_begin0+0x181/0x940 [lquota] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0ed322f&gt;] ? ldiskfs_xattr_ibody_get+0xef/0x1a0 [ldiskfs] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc12204d1&gt;] lod_qos_prep_create+0x1291/0x17f0 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1220bf9&gt;] ? lod_prepare_inuse+0x1c9/0x2e0 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1220f6d&gt;] lod_prepare_create+0x25d/0x360 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc121578e&gt;] lod_declare_striped_create+0x1ee/0x970 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1217c04&gt;] lod_declare_create+0x1e4/0x540 [lod] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc12828cf&gt;] mdd_declare_create_object_internal+0xdf/0x2f0 [mdd] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1274023&gt;] mdd_declare_create+0x53/0xe20 [mdd] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1277ec9&gt;] mdd_create+0x879/0x1400 [mdd] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc114ab93&gt;] mdt_reint_open+0x2173/0x3190 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0931dde&gt;] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc112fad3&gt;] ? ucred_set_jobid+0x53/0x70 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc113fa40&gt;] mdt_reint_rec+0x80/0x210 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc112131b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc1121842&gt;] mdt_intent_reint+0x162/0x430 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc112c5ae&gt;] mdt_intent_policy+0x43e/0xc70 [mdt] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0afc12f&gt;] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0af5277&gt;] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b1e9e3&gt;] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b46bc0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0ba3e92&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0ba7d95&gt;] tgt_request_handle+0x925/0x1370 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b50bf6&gt;] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b4d228&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff810c4822&gt;] ? default_wake_function+0x12/0x20 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b54332&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffffc0b538a0&gt;] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff816b4f58&gt;] ret_from_fork+0x58/0x90 Mar 29 09:38:38 oak-md1-s2 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> 3.10.0-693.2.2.el7_lustre.pl1.x86_64 LU-12136

mdt threads blocked in mdd_create

Bug Critical Open Unresolved Alex Zhuravlev Stephane Thiell Sat, 30 Mar 2019 17:02:58 +0000 Mon, 10 Oct 2022 23:29:44 +0000 Lustre 2.10.6 1 13 <p>please, attach logs from OST.</p> <p>Hi Stephane,</p> <p>Any luck in getting the OST logs as requested?</p> <p>Thanks.</p> <p>Joe</p> <p>Hi Alex and Joe – I just attached the OSS kernel logs, thanks!<br/> Note: We upgraded Oak servers to 2.10.7 last night.</p> <p>Alex, did you ever get a chance to look into this code? It seems like we should not be holding this global mutex over RPC calls. If the OST is unavailable, or doesn't have any objects, then the call should trigger the precreated thread and return immediately, rather than blocking the MDS thread. </p> <p>hmm, the cause for reboot was LBUG, AFAICS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Mar 29 10:28:48 oak-md1-s2 kernel: LustreError: 118086:0:(osd_handler.c:1982:osd_object_release()) LBUG Mar 29 10:28:48 oak-md1-s2 kernel: Pid: 118086, comm: tgt_recover_0 3.10.0-693.2.2.el7_lustre.pl1.x86_64 #1 SMP Sat Sep 30 00:44:25 PDT 2017 Mar 29 10:28:48 oak-md1-s2 kernel: Call Trace: Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffff8103a212&gt;] save_stack_trace_tsk+0x22/0x40 Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc08247cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc082487c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc0fe3660&gt;] osd_get_ldiskfs_dirent_param+0x0/0x130 [osd_ldiskfs] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc096ac4d&gt;] lu_object_put+0x17d/0x3d0 [obdclass] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc0c0e738&gt;] distribute_txn_replay_handle+0x418/0xc00 [ptlrpc] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc0b54987&gt;] replay_request_or_update.isra.21+0x247/0x8c0 [ptlrpc] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffffc0b55616&gt;] target_recovery_thread+0x616/0x1360 [ptlrpc] Mar 29 10:28:48 oak-md1-s2 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 </pre> </div></div> <p>Andreas, normally we don't wait on RPC in that path - when specific OST is out of precreated objects, but once we hit it then the calling thread wats for obd_timeout..</p> <p>Any update on getting a fix for this issue?  We have hit it at NOAA at least twice, with no sign of any LBUG.  This is with lustre-2.10.7_ddn8.  Here is the call trace in case it helps:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Oct 21 23:08:11 lfs-mds-1-1 kernel: INFO: task mdt00_002:17259 blocked <span class="code-keyword">for</span> more than 120 seconds. Oct 21 23:08:11 lfs-mds-1-1 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Oct 21 23:08:11 lfs-mds-1-1 kernel: mdt00_002 D ffff96927956d140 0 17259 2 0x00000000 Oct 21 23:08:11 lfs-mds-1-1 kernel: Call Trace: Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa4b68e19&gt;] schedule+0x29/0x70 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa4b6a705&gt;] rwsem_down_write_failed+0x225/0x3a0 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa47874a7&gt;] call_rwsem_down_write_failed+0x17/0x30 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa4b6814d&gt;] down_write+0x2d/0x3d Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1624cef&gt;] lod_alloc_qos.constprop.17+0x1af/0x1590 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc12d7a47&gt;] ? qsd_op_begin0+0xc7/0x940 [lquota] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa467966d&gt;] ? __getblk+0x2d/0x300 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1628d51&gt;] lod_qos_prep_create+0x12b1/0x1810 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1629479&gt;] ? lod_prepare_inuse+0x1c9/0x2e0 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc16297ed&gt;] lod_prepare_create+0x25d/0x360 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc161e11e&gt;] lod_declare_striped_create+0x1ee/0x970 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1620594&gt;] lod_declare_create+0x1e4/0x550 [lod] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc168dc72&gt;] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc167d1a3&gt;] mdd_declare_create+0x53/0xe40 [mdd] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1681739&gt;] mdd_create+0x9c9/0x1420 [mdd] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1552685&gt;] mdt_reint_open+0x2175/0x3190 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0ff2037&gt;] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1547403&gt;] mdt_reint_rec+0x83/0x210 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc152818b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc15286b2&gt;] mdt_intent_reint+0x162/0x480 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc152b50b&gt;] mdt_intent_opc+0x1eb/0xaf0 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0ff4060&gt;] ? lustre_swab_ldlm_policy_data+0x30/0x30 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1533e70&gt;] mdt_intent_policy+0x130/0x320 [mdt] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0fa2226&gt;] ldlm_lock_enqueue+0x396/0x960 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0fcbcd7&gt;] ldlm_handle_enqueue0+0xa97/0x1640 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0ff40e0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1054882&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc10587ba&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0ffe12b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc0ffa758&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa44d6af2&gt;] ? default_wake_function+0x12/0x20 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa44cbb8b&gt;] ? __wake_up_common+0x5b/0x90 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1001872&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffc1000de0&gt;] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa44c1d21&gt;] kthread+0xd1/0xe0 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa44c1c50&gt;] ? insert_kthread_work+0x40/0x40 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa4b75c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 Oct 21 23:08:11 lfs-mds-1-1 kernel: [&lt;ffffffffa44c1c50&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>Hi Nathan,</p> <p>Just FYI, we've not seen this problem on our 2.10 servers since we use 2.10.8 + 3 patches (<a href="https://github.com/stanford-rc/lustre/commits/b2_10_oak" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/stanford-rc/lustre/commits/b2_10_oak</a>). I'm wondering if you have "<a href="https://jira.whamcloud.com/browse/LU-12018" title="deadlock on OSS: quota reintegration vs memory release" class="issue-link" data-issue-key="LU-12018"><del>LU-12018</del></a> quota: do not start a thread under memory pressure". It's a critical patch that is missing from 2.10.8 IMHO.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38428" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38428</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12136" title="mdt threads blocked in mdd_create" class="issue-link" data-issue-key="LU-12136">LU-12136</a> osp: blocked-in-precreation stats<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3c054c4b39c49a750666698c52385d4f7133a98</p> <p>the number of improvements have been made in this area (<a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a>, <a href="https://jira.whamcloud.com/browse/LU-15082" title="osp can stop refreshing statfs data" class="issue-link" data-issue-key="LU-15082"><del>LU-15082</del></a>, <a href="https://jira.whamcloud.com/browse/LU-15724" title="MDT failover hang" class="issue-link" data-issue-key="LU-15724"><del>LU-15724</del></a>, <a href="https://jira.whamcloud.com/browse/LU-13073" title="Multiple MDS deadlocks (in lod_qos_prep_create) after OSS crash" class="issue-link" data-issue-key="LU-13073"><del>LU-13073</del></a>, <a href="https://jira.whamcloud.com/browse/LU-13513" title="osp_precreate_reserve() should not take d-&gt;opd_pre_status == -EIO as fatal" class="issue-link" data-issue-key="LU-13513"><del>LU-13513</del></a>, <a href="https://jira.whamcloud.com/browse/LU-12397" title="osp: race around opd_new_connection" class="issue-link" data-issue-key="LU-12397"><del>LU-12397</del></a>, etc). probably makes sense to check is this still a problem with the newer versions?</p> <p>Thanks Alex! Agreed that this is an old ticket and is probably not relevant anymore.</p> Related LU-10697 Development Rank 1|i00e8v: Rank (Obsolete) 9223372036854775807 Severity