https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11613 Lustre <p>Hi </p> <p>We are experiencing the Lustre MDS and OSS server deadlock with high CPU usage and once the server is rebooted the file system goes back to normal. By looking at the stack traces this looks similar to <a href="https://jira.whamcloud.com/browse/LU-11465" title="OSS/MDS deadlock in 2.10.5" class="issue-link" data-issue-key="LU-11465"><del>LU-11465</del></a></p> <p>Here are the stack trace from MDS server.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Nov  3 19:02:14 mds rsyslogd: -- MARK -- Nov  3 19:07:07 mds kernel: [468204.667688] INFO: task jbd2/dm-0:5862 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov  3 19:07:07 mds kernel: [468204.687998] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov  3 19:07:07 mds kernel: [468204.711686] jbd2/dm-0       D ffff883fcf614f10     0  5862      2 0x00000080 Nov  3 19:07:07 mds kernel: [468204.711693] Call Trace: Nov  3 19:07:07 mds kernel: [468204.719277]  [&lt;ffffffff81694cd9&gt;] schedule+0x29/0x70 Nov  3 19:07:07 mds kernel: [468204.719303]  [&lt;ffffffffa01822b6&gt;] jbd2_journal_commit_transaction+0x246/0x19a0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719309]  [&lt;ffffffff810cac9e&gt;] ? account_entity_dequeue+0xae/0xd0 Nov  3 19:07:07 mds kernel: [468204.719312]  [&lt;ffffffff810ce77c&gt;] ? dequeue_entity+0x11c/0x5e0 Nov  3 19:07:07 mds kernel: [468204.719315]  [&lt;ffffffff810c8185&gt;] ? sched_clock_cpu+0x85/0xc0 Nov  3 19:07:07 mds kernel: [468204.719318]  [&lt;ffffffff8102954d&gt;] ? __switch_to+0xcd/0x4b0 Nov  3 19:07:07 mds kernel: [468204.719322]  [&lt;ffffffff810b20a0&gt;] ? wake_up_atomic_t+0x30/0x30 Nov  3 19:07:07 mds kernel: [468204.719324]  [&lt;ffffffff816946f7&gt;] ? __schedule+0x477/0xa30 Nov  3 19:07:07 mds kernel: [468204.719328]  [&lt;ffffffff810998ee&gt;] ? try_to_del_timer_sync+0x5e/0x90 Nov  3 19:07:07 mds kernel: [468204.719334]  [&lt;ffffffffa0188ab9&gt;] kjournald2+0xc9/0x260 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719336]  [&lt;ffffffff810b20a0&gt;] ? wake_up_atomic_t+0x30/0x30 Nov  3 19:07:07 mds kernel: [468204.719341]  [&lt;ffffffffa01889f0&gt;] ? commit_timeout+0x10/0x10 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719343]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Nov  3 19:07:07 mds kernel: [468204.719345]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:07 mds kernel: [468204.719347]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 Nov  3 19:07:07 mds kernel: [468204.719349]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:07 mds kernel: [468204.719351] INFO: task mdt00_002:5872 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov  3 19:07:07 mds kernel: [468204.719352] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov  3 19:07:07 mds kernel: [468204.719354] mdt00_002       D ffff881fc0dd4f10     0  5872      2 0x00000080 Nov  3 19:07:07 mds kernel: [468204.719354] Call Trace: Nov  3 19:07:07 mds kernel: [468204.719356]  [&lt;ffffffff81694cd9&gt;] schedule+0x29/0x70 Nov  3 19:07:07 mds kernel: [468204.719361]  [&lt;ffffffffa017f085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719362]  [&lt;ffffffff810b20a0&gt;] ? wake_up_atomic_t+0x30/0x30 Nov  3 19:07:07 mds kernel: [468204.719367]  [&lt;ffffffffa017f368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719371]  [&lt;ffffffffa017f5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719385]  [&lt;ffffffffa1087fba&gt;] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.719390]  [&lt;ffffffff811df622&gt;] ? kmem_cache_alloc+0x1c2/0x1f0 Nov  3 19:07:07 mds kernel: [468204.719394]  [&lt;ffffffffa017fa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.719401]  [&lt;ffffffffa1069c8e&gt;] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.719410]  [&lt;ffffffffa0fcb909&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Nov  3 19:07:07 mds kernel: [468204.719415]  [&lt;ffffffffa1069c8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.719430]  [&lt;ffffffffa1232da7&gt;] mdt_empty_transno+0xf7/0x840 [mdt] Nov  3 19:07:07 mds kernel: [468204.719439]  [&lt;ffffffffa1235ede&gt;] mdt_mfd_open+0x8de/0xe70 [mdt] Nov  3 19:07:07 mds kernel: [468204.719446]  [&lt;ffffffffa120e012&gt;] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Nov  3 19:07:07 mds kernel: [468204.719454]  [&lt;ffffffffa12369eb&gt;] mdt_finish_open+0x57b/0x690 [mdt] Nov  3 19:07:07 mds kernel: [468204.719462]  [&lt;ffffffffa12382c8&gt;] mdt_reint_open+0x17c8/0x3190 [mdt] Nov  3 19:07:07 mds kernel: [468204.719495]  [&lt;ffffffffa09b2311&gt;] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Nov  3 19:07:07 mds kernel: [468204.719513]  [&lt;ffffffffa09b726e&gt;] ? lu_ucred+0x1e/0x30 [obdclass] Nov  3 19:07:07 mds kernel: [468204.719523]  [&lt;ffffffffa121d9e5&gt;] ? mdt_ucred+0x15/0x20 [mdt] Nov  3 19:07:07 mds kernel: [468204.719532]  [&lt;ffffffffa121e2b1&gt;] ? mdt_root_squash+0x21/0x430 [mdt] Nov  3 19:07:07 mds kernel: [468204.719542]  [&lt;ffffffffa122daf3&gt;] mdt_reint_rec+0x83/0x210 [mdt] Nov  3 19:07:07 mds kernel: [468204.719551]  [&lt;ffffffffa120f33b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Nov  3 19:07:07 mds kernel: [468204.719560]  [&lt;ffffffffa120f862&gt;] mdt_intent_reint+0x162/0x430 [mdt] Nov  3 19:07:07 mds kernel: [468204.719569]  [&lt;ffffffffa121a631&gt;] mdt_intent_policy+0x441/0xc70 [mdt] Nov  3 19:07:07 mds kernel: [468204.719614]  [&lt;ffffffffa0b801db&gt;] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719640]  [&lt;ffffffffa0b792ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719678]  [&lt;ffffffffa0ba2b53&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719715]  [&lt;ffffffffa0bcada0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719761]  [&lt;ffffffffa0c28262&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719800]  [&lt;ffffffffa0c2beca&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719836]  [&lt;ffffffffa0bd44bb&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719872]  [&lt;ffffffffa0bd29b8&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719877]  [&lt;ffffffff810c4d40&gt;] ? wake_up_state+0x20/0x20 Nov  3 19:07:07 mds kernel: [468204.719911]  [&lt;ffffffffa0bd84a2&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719946]  [&lt;ffffffffa0bd7a10&gt;] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.719952]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Nov  3 19:07:07 mds kernel: [468204.719955]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:07 mds kernel: [468204.719958]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 Nov  3 19:07:07 mds kernel: [468204.719961]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:07 mds kernel: [468204.719963] INFO: task mdt01_001:5874 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov  3 19:07:07 mds kernel: [468204.719965] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov  3 19:07:07 mds kernel: [468204.719971] mdt01_001       D ffff881fc0dd0fd0     0  5874      2 0x00000080 Nov  3 19:07:07 mds kernel: [468204.719974] Call Trace: Nov  3 19:07:07 mds kernel: [468204.720005]  [&lt;ffffffffa0bae0a0&gt;] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720010]  [&lt;ffffffff81694cd9&gt;] schedule+0x29/0x70 Nov  3 19:07:07 mds kernel: [468204.720018]  [&lt;ffffffffa017f085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.720022]  [&lt;ffffffff810b20a0&gt;] ? wake_up_atomic_t+0x30/0x30 Nov  3 19:07:07 mds kernel: [468204.720028]  [&lt;ffffffffa017f368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.720035]  [&lt;ffffffffa017f5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] Nov  3 19:07:07 mds kernel: [468204.720048]  [&lt;ffffffffa1087fba&gt;] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.720053]  [&lt;ffffffff811df622&gt;] ? kmem_cache_alloc+0x1c2/0x1f0 Nov  3 19:07:07 mds kernel: [468204.720059]  [&lt;ffffffffa017fa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] Nov  3 19:07:07 mds kernel: [468204.720068]  [&lt;ffffffffa1069c8e&gt;] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.720078]  [&lt;ffffffffa0fcb909&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Nov  3 19:07:07 mds kernel: [468204.720088]  [&lt;ffffffffa1069c8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:07:07 mds kernel: [468204.720100]  [&lt;ffffffffa1232da7&gt;] mdt_empty_transno+0xf7/0x840 [mdt] Nov  3 19:07:07 mds kernel: [468204.720112]  [&lt;ffffffffa1235ede&gt;] mdt_mfd_open+0x8de/0xe70 [mdt] Nov  3 19:07:07 mds kernel: [468204.720124]  [&lt;ffffffffa120e012&gt;] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Nov  3 19:07:07 mds kernel: [468204.720136]  [&lt;ffffffffa12369eb&gt;] mdt_finish_open+0x57b/0x690 [mdt] Nov  3 19:07:07 mds kernel: [468204.720149]  [&lt;ffffffffa12382c8&gt;] mdt_reint_open+0x17c8/0x3190 [mdt] Nov  3 19:07:07 mds kernel: [468204.720170]  [&lt;ffffffffa09b2311&gt;] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Nov  3 19:07:07 mds kernel: [468204.720190]  [&lt;ffffffffa09b726e&gt;] ? lu_ucred+0x1e/0x30 [obdclass] Nov  3 19:07:07 mds kernel: [468204.720202]  [&lt;ffffffffa121d9e5&gt;] ? mdt_ucred+0x15/0x20 [mdt] Nov  3 19:07:07 mds kernel: [468204.720214]  [&lt;ffffffffa121e2b1&gt;] ? mdt_root_squash+0x21/0x430 [mdt] Nov  3 19:07:07 mds kernel: [468204.720227]  [&lt;ffffffffa122daf3&gt;] mdt_reint_rec+0x83/0x210 [mdt] Nov  3 19:07:07 mds kernel: [468204.720239]  [&lt;ffffffffa120f33b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Nov  3 19:07:07 mds kernel: [468204.720249]  [&lt;ffffffffa120f862&gt;] mdt_intent_reint+0x162/0x430 [mdt] Nov  3 19:07:07 mds kernel: [468204.720262]  [&lt;ffffffffa121a631&gt;] mdt_intent_policy+0x441/0xc70 [mdt] Nov  3 19:07:07 mds kernel: [468204.720291]  [&lt;ffffffffa0b801db&gt;] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720319]  [&lt;ffffffffa0b792ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720350]  [&lt;ffffffffa0ba2b53&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720382]  [&lt;ffffffffa0bcada0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720418]  [&lt;ffffffffa0c28262&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720456]  [&lt;ffffffffa0c2beca&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Nov  3 19:07:07 mds kernel: [468204.720491]  [&lt;ffffffffa0bd44bb&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Nov  3 19:07:14 mds kernel: [468204.720525]  [&lt;ffffffffa0bd29b8&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Nov  3 19:07:14 mds kernel: [468204.720529]  [&lt;ffffffff810c4d52&gt;] ? default_wake_function+0x12/0x20 Nov  3 19:07:14 mds kernel: [468204.720531]  [&lt;ffffffff810bad2b&gt;] ? __wake_up_common+0x5b/0x90 Nov  3 19:07:14 mds kernel: [468204.720564]  [&lt;ffffffffa0bd84a2&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Nov  3 19:07:14 mds kernel: [468204.720597]  [&lt;ffffffffa0bd7a10&gt;] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Nov  3 19:07:14 mds kernel: [468204.720600]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Nov  3 19:07:14 mds kernel: [468204.720602]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:14 mds kernel: [468204.720603]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 Nov  3 19:07:14 mds kernel: [468204.720605]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 Nov  3 19:07:14 mds kernel: [468204.720609] INFO: task osp-pre-141-0:5909 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov  3 19:07:14 mds kernel: [468204.720610] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov  3 19:07:14 mds kernel: [468204.720612] osp-pre-141-0   D ffff881fd0621fa0     0  5909      2 0x00000080 </pre> </div></div> <p> </p> <p> </p> <p> </p> <p>Here are the stack trace from the OSS server.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Nov  3 19:08:40 oss kernel: [469273.871420] Pid: 85809, comm: ll_ost_io01_185 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 SMP Mon Aug 27 23:04:41 UTC 2018 Nov  3 19:08:40 oss kernel: [469273.871421] Call Trace: Nov  3 19:08:40 oss kernel: [469273.871450]  [&lt;ffffffffa0179085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] Nov  3 19:08:40 oss kernel: [469273.892337]  [&lt;ffffffffa0179368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] Nov  3 19:08:40 oss kernel: [469273.892340]  [&lt;ffffffffa01795e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] Nov  3 19:08:40 oss kernel: [469273.892344]  [&lt;ffffffffa0179a93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] Nov  3 19:08:40 oss kernel: [469273.892358]  [&lt;ffffffffa14a7909&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Nov  3 19:08:40 oss kernel: [469273.892370]  [&lt;ffffffffa108dc8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:08:40 oss kernel: [469273.892378]  [&lt;ffffffffa111127e&gt;] ofd_trans_start+0x6e/0xf0 [ofd] Nov  3 19:08:40 oss kernel: [469273.892383]  [&lt;ffffffffa111772b&gt;] ofd_commitrw_write+0x94b/0x1c90 [ofd] Nov  3 19:08:40 oss kernel: [469273.892394]  [&lt;ffffffffa111b779&gt;] ofd_commitrw+0x4c9/0xae0 [ofd] Nov  3 19:08:40 oss kernel: [469273.892456]  [&lt;ffffffffa0c52510&gt;] obd_commitrw+0x2f3/0x336 [ptlrpc] Nov  3 19:08:40 oss kernel: [469273.892504]  [&lt;ffffffffa0c25145&gt;] tgt_brw_write+0xfb5/0x1780 [ptlrpc] Nov  3 19:08:40 oss kernel: [469273.892539]  [&lt;ffffffffa0c20eca&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Nov  3 19:08:40 oss kernel: [469273.892570]  [&lt;ffffffffa0bc94bb&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Nov  3 19:08:40 oss kernel: [469273.892600]  [&lt;ffffffffa0bcd4a2&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Nov  3 19:08:40 oss kernel: [469273.892609]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Nov  3 19:08:40 oss kernel: [469273.892616]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 Nov  3 19:08:40 oss kernel: [469273.892624]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff Nov  3 19:08:40 oss kernel: [469273.892627] LustreError: dumping log to /tmp/lustre-log.1541297320.85809 Nov  3 19:08:53 oss kernel: [469287.156081] LNet: Service thread pid 85694 was inactive <span class="code-keyword">for</span> 301.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Nov  3 19:08:53 oss kernel: [469287.207070] Pid: 85694, comm: ll_ost_io00_185 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 SMP Mon Aug 27 23:04:41 UTC 2018 Nov  3 19:08:53 oss kernel: [469287.207071] Call Trace: Nov  3 19:08:53 oss kernel: [469287.207100]  [&lt;ffffffffa0179085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] Nov  3 19:08:53 oss kernel: [469287.227995]  [&lt;ffffffffa0179368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] Nov  3 19:08:53 oss kernel: [469287.227999]  [&lt;ffffffffa01795e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] Nov  3 19:08:53 oss kernel: [469287.228002]  [&lt;ffffffffa0179a93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] Nov  3 19:08:53 oss kernel: [469287.228010]  [&lt;ffffffffa14a7909&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Nov  3 19:08:53 oss kernel: [469287.228024]  [&lt;ffffffffa108dc8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov  3 19:08:53 oss kernel: [469287.228036]  [&lt;ffffffffa1117ce3&gt;] ofd_commitrw_write+0xf03/0x1c90 [ofd] Nov  3 19:08:53 oss kernel: [469287.228042]  [&lt;ffffffffa111b779&gt;] ofd_commitrw+0x4c9/0xae0 [ofd] Nov  3 19:08:53 oss kernel: [469287.228084]  [&lt;ffffffffa0c52510&gt;] obd_commitrw+0x2f3/0x336 [ptlrpc] Nov  3 19:08:53 oss kernel: [469287.228125]  [&lt;ffffffffa0c25145&gt;] tgt_brw_write+0xfb5/0x1780 [ptlrpc] Nov  3 19:08:53 oss kernel: [469287.228160]  [&lt;ffffffffa0c20eca&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Nov  3 19:08:53 oss kernel: [469287.228192]  [&lt;ffffffffa0bc94bb&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Nov  3 19:08:53 oss kernel: [469287.228224]  [&lt;ffffffffa0bcd4a2&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Nov  3 19:08:53 oss kernel: [469287.228227]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Nov  3 19:08:53 oss kernel: [469287.228230]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 Nov  3 19:08:53 oss kernel: [469287.228235]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p> </p> <p> </p> <p>If you need complete crash dump then let me know and I will upload it.</p> <p>Thank You,</p> <p>                  Manish</p> <p> </p> CentOS Linux release 7.5.1804 (Core) <br/> Kernel Version 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 <br/> e2fsprogs-libs-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-static-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-devel-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-debuginfo-1.44.3.wc1-0.el7.x86_64 LU-11613

MDS and OSS locked up wait_transaction_locked+0x85/0xd0 [jbd2]

Bug Critical Resolved Duplicate Alex Zhuravlev Manish Sun, 4 Nov 2018 06:48:43 +0000 Tue, 27 Aug 2019 17:55:10 +0000 Tue, 27 Aug 2019 17:55:10 +0000 Lustre 2.10.5 0 13 <p>Manish</p> <p>It sounds like this is a disruptive bug to production rather than the whole system is completely out of service so I have adjusted the severity accordingly.</p> <p>Alex</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>can you please attach backtraces for all the processes?</p> <p> </p> <p>Hi Alex,</p> <p>I have uploaded the stack trace to the FTP site and file name is "foreach_bt.txt" under ticket dir.</p> <p>Let me know if you are able to view that file.</p> <p>Thank You,</p> <p>                  Manish</p> <p>Hi,</p> <p>We hit this issue again with one of our other file system and the stack traces looks similar. I have already uploaded the new stack trace to FTP site and file names are </p> <p>foreach_bt_mds_11-05-2018.txt</p> <p>foreach_bt_oss_11-05-2018.txt</p> <p> </p> <p>Can we please raise the level of priority as this is now ongoing issues with most of our lustre file system with lustre upgrade with 2.10.5 version.</p> <p>Thank You,</p> <p>                  Manish</p> <p>thanks, it will take some time to analyse the traces.</p> <p> </p> <p>Manish, please tell how many MDSs are used and whether LFSCK was running at that time?</p> <p> </p> <p>There is only one mds and fsck was not running. </p> <p>FYI, our lockups associated with <a href="https://jira.whamcloud.com/browse/LU-11465" title="OSS/MDS deadlock in 2.10.5" class="issue-link" data-issue-key="LU-11465"><del>LU-11465</del></a> show a high load average on the affected server (MDS for our case), but the CPU utilization is <em>not</em> high.  In fact the MDS is quite responsive, CPUs are pretty much idle, but the high load average is due to all of the stuck "D-state" processes which are runnable, but deadlocked.</p> <p>Another difference from our <a href="https://jira.whamcloud.com/browse/LU-11465" title="OSS/MDS deadlock in 2.10.5" class="issue-link" data-issue-key="LU-11465"><del>LU-11465</del></a> situation is that rebooting the stuck server does not resolve the problem for us.  We must force-unmount or reboot all the clients as well.</p> <p> </p> <p>it would be very helpful if you can provide Lustre logs and/or dmesg messages. I think I understand the cause (and have patches in testing), but I still don't understand how we got into that situation..</p> <p> </p> <p>do you use (or used) project quota?</p> <p> </p> <p>AFAICS, some client sends chown/chgrp to the quota-enabled MDS which in turn needs to consult with OST on possibility to move file into another group. this file is being actively written so OST hits a deadlock (<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a>). until that client is stopped it keeps resending RPC after reboot.</p> <p>though I can't really explain that OST hits it again and again.. that's supposed to be a race-like condition.</p> <p>Hi Alex,</p> <p>We have never used project quota. I will share dmesg logs soon with Oleg and you can get more details form him.</p> <p>Thank You,</p> <p>                   Manish</p> <p>Hi Alex,</p> <p>Sorry for the confusion about what logs I can share as I am trying to get familiar with security limitations on sharing data. Well so I have uploaded the dmesg logs on FTP site un ticket directory and files are "mds-vmcore-dmesg.txt" and "oss-vmcore-dmesg.txt".</p> <p>Thank You,</p> <p>                  Manish</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ pwd /scratch/ftp/uploads/LU-11613 [alexeyzh@ssh-2 LU-11613]$ ls -l total 4448 -rw-r--r-- 1 nobody ftp  115858 Nov  4 16:42 foreach_backtrace.zip -rw-r--r-- 1 nobody ftp 1724395 Nov  5 19:23 foreach_bt_mds_11-05-2018.txt -rw-r--r-- 1 nobody ftp 1315475 Nov  5 19:23 foreach_bt_oss_11-05-2018.txt -rw-r--r-- 1 nobody ftp 1389778 Nov  4 19:16 foreach_bt.txt</pre> </div></div> <p>probably into a different dir?</p> <p>Hi Alex,</p> <p>No it's in same dir.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ftp.whamcloud.com:/uploads/LU-11613&gt; ls -rw-r--r--    1 99       50         115858 Nov 04 16:42 foreach_backtrace.zip -rw-r--r--    1 99       50        1389778 Nov 04 19:16 foreach_bt.txt -rw-r--r--    1 99       50        1724395 Nov 05 19:23 foreach_bt_mds_11-05-2018.txt -rw-r--r--    1 99       50        1315475 Nov 05 19:23 foreach_bt_oss_11-05-2018.txt -rw-r--r--    1 99       50        1042487 Nov 07 16:25 mds-vmcore-dmesg.txt -rw-r--r--    1 99       50        1042487 Nov 07 16:26 oss-vmcore-dmesg.txt </pre> </div></div> <p> </p> <p>Please re-fresh dir at your end.</p> <p>Thank You,</p> <p>                   Manish</p> <p>did you upgrade Lustre recently? if yes, what the previous version was?<br/> any idea about the application(s) the cluster is running? do you think chown/chgrp is a frequent operation?</p> <p>I'm able to reproduce the issue and the fix seem to work, but we'd like to understand why did you start to experience the issue.</p> <p>Yes these filesystem were updated from 2.7 to 2.10.5 and that is when we started to see this issue.</p> <p>It is unlikely that there are frequent chown/chgrop operation.</p> <p>These are our larger filesystems and there are many applications running at the same time. It is difficult to isolate the issue to a simple type of app.</p> <p>Hit this problem at NOAA too, within 1 day of upgrading servers from 2.7 to 2.10.5 and clients from 2.10.4 to 2.10.5.  See DDN Case #112165 for details.</p> <p>At NOAA, chgrp is commonly used (to keep ownership aligned with top-level "project" directory, which we soon hope to replace with project quotas).</p> <p> </p> <p>We use chgrp for the same reasons as the NOAA poster above.</p> <p>While we wait for the fix to be released, can we avoid this lockup by disabling quota enforcement, or removing all our group quotas?</p> <p>Or must we disable the chown/chgrp commands cluster-wide?</p> <p> </p> <p>Adding MDS call traces from DDN 112165</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Nov 8 17:18:50 lfs-mds-1-1 kernel: Lustre: lfs1-MDT0000: Connection restored to 4de02ddb-11dd-0c3b-e5a4-6b3f42554a8e (at 10.179.38.35@o2ib) Nov 8 18:58:23 lfs-mds-1-1 kernel: Lustre: lfs1-MDT0000: Connection restored to a81b824f-2064-b0a5-e927-1bb9bd32bb32 (at 10.179.126.13@o2ib) Nov 8 19:11:19 lfs-mds-1-1 kernel: INFO: task jbd2/dm-21-8:41769 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov 8 19:11:19 lfs-mds-1-1 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov 8 19:11:19 lfs-mds-1-1 kernel: jbd2/dm-21-8 D ffff922a486a9fa0 0 41769 2 0x00000000 Nov 8 19:11:19 lfs-mds-1-1 kernel: Call Trace: Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffaeb14029&gt;] schedule+0x29/0x70 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04a82ac&gt;] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4d52ce&gt;] ? account_entity_dequeue+0xae/0xd0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4d8e4c&gt;] ? dequeue_entity+0x11c/0x5e0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4d2765&gt;] ? sched_clock_cpu+0x85/0xc0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae42959e&gt;] ? __switch_to+0xce/0x580 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bc610&gt;] ? wake_up_atomic_t+0x30/0x30 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffaeb139fc&gt;] ? __schedule+0x41c/0xa20 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4a3eae&gt;] ? try_to_del_timer_sync+0x5e/0x90 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04aeab9&gt;] kjournald2+0xc9/0x260 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bc610&gt;] ? wake_up_atomic_t+0x30/0x30 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04ae9f0&gt;] ? commit_timeout+0x10/0x10 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb621&gt;] kthread+0xd1/0xe0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb550&gt;] ? insert_kthread_work+0x40/0x40 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffaeb205f7&gt;] ret_from_fork_nospec_begin+0x21/0x21 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb550&gt;] ? insert_kthread_work+0x40/0x40 Nov 8 19:11:19 lfs-mds-1-1 kernel: INFO: task mdt00_002:41819 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov 8 19:11:19 lfs-mds-1-1 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov 8 19:11:19 lfs-mds-1-1 kernel: mdt00_002 D ffff922a48a9bf40 0 41819 2 0x00000000 Nov 8 19:11:19 lfs-mds-1-1 kernel: Call Trace: Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f05240&gt;] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffaeb14029&gt;] schedule+0x29/0x70 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04a5085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bc610&gt;] ? wake_up_atomic_t+0x30/0x30 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04a5368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04a55e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc136a06a&gt;] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae5f7bc2&gt;] ? kmem_cache_alloc+0x1c2/0x1f0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc04a5a93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc134bcde&gt;] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1225179&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc134bcde&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc159b5d7&gt;] mdt_empty_transno+0xf7/0x840 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc159e70e&gt;] mdt_mfd_open+0x8de/0xe70 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1576042&gt;] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc159f21b&gt;] mdt_finish_open+0x57b/0x690 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc15a0af8&gt;] mdt_reint_open+0x17c8/0x3190 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f1fef7&gt;] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0d5ca5e&gt;] ? lu_ucred+0x1e/0x30 [obdclass] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1585ad5&gt;] ? mdt_ucred+0x15/0x20 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1586401&gt;] ? mdt_root_squash+0x21/0x430 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1596213&gt;] mdt_reint_rec+0x83/0x210 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc157736b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1577892&gt;] mdt_intent_reint+0x162/0x480 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc1582721&gt;] mdt_intent_policy+0x441/0xc70 [mdt] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0ed71db&gt;] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0ed02ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0ef9cf3&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f21fa0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f824e2&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f8641a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f2bfeb&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f28618&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4cf682&gt;] ? default_wake_function+0x12/0x20 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4c52ab&gt;] ? __wake_up_common+0x5b/0x90 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f2f732&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffc0f2eca0&gt;] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb621&gt;] kthread+0xd1/0xe0 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb550&gt;] ? insert_kthread_work+0x40/0x40 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffaeb205f7&gt;] ret_from_fork_nospec_begin+0x21/0x21 Nov 8 19:11:19 lfs-mds-1-1 kernel: [&lt;ffffffffae4bb550&gt;] ? insert_kthread_work+0x40/0x40 Nov 8 19:11:19 lfs-mds-1-1 kernel: INFO: task mdt02_000:41823 blocked <span class="code-keyword">for</span> more than 120 seconds. </pre> </div></div> <p>The revert of <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> in 2.10.6 should remove the possibility to hit this issue</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=cwhite_ddn" class="user-hover" rel="cwhite_ddn">cwhite_ddn</a> please add all remaining stack traces if possible</p> <p>Let's not conflate similar (or even identical) issues from different sites with this support request from NASA. <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=manishpatel" class="user-hover" rel="manishpatel">manishpatel</a> has this issue been seen with 2.10.6?</p> <p>Hi Peter,</p> <p> </p> <p>We are still on 2.10.5 version and working few other issues to address with this one. So we may not get 2.10.6 version and may rather end up with newer released version based on downtime slots we get. Also we have got the new build ready without <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch, but all depends when we get a chance to implement it, so if that happens then we will update it and let you know if we still experience this issues.</p> <p>Thank You,</p> <p>                   Manish</p> <p>The patch <a href="https://review.whamcloud.com/31293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31293</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction</tt>" landed.</p> Duplicate LU-10048 Related LU-11465 Development Rank 1|i005nz: Rank (Obsolete) 9223372036854775807 Severity