https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11465 Lustre <p>After an upgrade of Lustre servers from Centos 6 and IEEL/DDN version of Lustre 2.5 to Centos 7 and Lustre 2.10.5 we are experiencing stability issues related to and MDS/OSS deadlock, most likely caused by hung OST thread. The issue is visible on one of two filesystems, the smaller one, both are working on identical HW and SW stack, so the issue is most likely related to a specific workload on the filesystem. This is mostly an midterm storage filesystem, so more metadata ops happen on it than on the scratch one. We initially thought that the issue might be caused by our try to enable project quotas (tune2fs -O project on all targets), so we turned it off (tune2fs -O ^project), which didn't change a thing. Also all combinations of MDT/OST quota enforcement (initally it was 'g', than 'gp', now 'none') doesn't change a thing, so probably it is not related to the problem. In terms of relevant information, we run on sync_journal=1 as a workaround to a problem with memory accounting on the clients suggested by DDN some time ago. The crash happens at least once per few hours, sometimes sequentially one after another, on different OSS nodes. I also tagged 2.10.6 as affected, as we tried everything from 2.10.4 to current b2_10.</p> <p>On OSS we get:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [Tue Oct 2 11:56:44 2018] LNet: Service thread pid 11401 was inactive <span class="code-keyword">for</span> 200.27s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [Tue Oct 2 11:56:44 2018] Pid: 11401, comm: ll_ost_out00_00 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:44 2018] Call Trace: [Tue Oct 2 11:56:44 2018] [&lt;ffffffff8395aa77&gt;] call_rwsem_down_write_failed+0x17/0x30 [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0eeaa8c&gt;] osd_write_lock+0x5c/0xe0 [osd_ldiskfs] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b5b119&gt;] out_tx_attr_set_exec+0x69/0x3f0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b55591&gt;] out_tx_end+0xe1/0x5c0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b596d3&gt;] out_handle+0x1453/0x1bc0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b4f38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0af7e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0afb592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffff836bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:44 2018] [&lt;ffffffff83d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:44 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:44 2018] LustreError: dumping log to /tmp/lustre-log.1538474204.11401 [Tue Oct 2 11:56:44 2018] Pid: 35664, comm: ll_ost_io00_098 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:44 2018] Call Trace: [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0e7e495&gt;] jbd2_log_wait_commit+0xc5/0x140 [jbd2] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0e76533&gt;] jbd2_journal_stop+0x343/0x3d0 [jbd2] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc112ab4c&gt;] __ldiskfs_journal_stop+0x3c/0xb0 [ldiskfs] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0efb783&gt;] osd_trans_stop+0x183/0x850 [osd_ldiskfs] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc107d352&gt;] ofd_trans_stop+0x22/0x60 [ofd] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc10835f4&gt;] ofd_commitrw_write+0x7e4/0x1c90 [ofd] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc10877a9&gt;] ofd_commitrw+0x4c9/0xae0 [ofd] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b80864&gt;] obd_commitrw+0x2f3/0x336 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b5338d&gt;] tgt_brw_write+0xffd/0x17d0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0b4f38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0af7e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0afb592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffff836bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:45 2018] [&lt;ffffffff83d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:45 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:45 2018] Pid: 35678, comm: ll_ost_io00_112 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:45 2018] Call Trace: [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0e7e495&gt;] jbd2_log_wait_commit+0xc5/0x140 [jbd2] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0e76533&gt;] jbd2_journal_stop+0x343/0x3d0 [jbd2] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc112ab4c&gt;] __ldiskfs_journal_stop+0x3c/0xb0 [ldiskfs] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0efb783&gt;] osd_trans_stop+0x183/0x850 [osd_ldiskfs] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc107d352&gt;] ofd_trans_stop+0x22/0x60 [ofd] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc10835f4&gt;] ofd_commitrw_write+0x7e4/0x1c90 [ofd] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc10877a9&gt;] ofd_commitrw+0x4c9/0xae0 [ofd] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0b80864&gt;] obd_commitrw+0x2f3/0x336 [ptlrpc] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0b5338d&gt;] tgt_brw_write+0xffd/0x17d0 [ptlrpc] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0b4f38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0af7e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:45 2018] [&lt;ffffffffc0afb592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:45 2018] [&lt;ffffffff836bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:45 2018] [&lt;ffffffff83d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:45 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p> </p> <p> </p> <p>On MDS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [Tue Oct 2 11:56:44 2018] LNet: Service thread pid 69588 was inactive <span class="code-keyword">for</span> 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [Tue Oct 2 11:56:44 2018] Pid: 69588, comm: mdt00_095 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:44 2018] Call Trace: [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0f7e140&gt;] ptlrpc_set_wait+0x4c0/0x920 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0f7e61d&gt;] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc14de463&gt;] osp_remote_sync+0xd3/0x200 [osp] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc14c6dd0&gt;] osp_attr_set+0x4c0/0x5d0 [osp] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1421c6b&gt;] lod_sub_attr_set+0x1cb/0x460 [lod] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1403be6&gt;] lod_obj_stripe_attr_set_cb+0x16/0x30 [lod] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc140fa96&gt;] lod_obj_for_each_stripe+0xb6/0x230 [lod] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1411043&gt;] lod_attr_set+0x2f3/0x9a0 [lod] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc14810a0&gt;] mdd_attr_set_internal+0x120/0x2a0 [mdd] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1481e8d&gt;] mdd_attr_set+0x8ad/0xce0 [mdd] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc13645f5&gt;] mdt_reint_setattr+0xba5/0x1060 [mdt] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1364b33&gt;] mdt_reint_rec+0x83/0x210 [mdt] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc134636b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc1351f07&gt;] mdt_reint+0x67/0x140 [mdt] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0fee38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0f96e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffffc0f9a592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:44 2018] [&lt;ffffffff946bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:44 2018] [&lt;ffffffff94d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:44 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:44 2018] LustreError: dumping log to /tmp/lustre-log.1538474204.69588 [Tue Oct 2 11:56:49 2018] LNet: Service thread pid 7634 was inactive <span class="code-keyword">for</span> 200.51s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [Tue Oct 2 11:56:49 2018] Pid: 7634, comm: mdt_rdpg00_000 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:49 2018] Call Trace: [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06efa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc118aa99&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc12cac8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1002512&gt;] top_trans_start+0x702/0x940 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc13ec3f1&gt;] lod_trans_start+0x31/0x40 [lod] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc148c1ba&gt;] mdd_trans_start+0x1a/0x20 [mdd] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1481b69&gt;] mdd_attr_set+0x589/0xce0 [mdd] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136c1d6&gt;] mdt_mfd_close+0x1a6/0x610 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1371951&gt;] mdt_close_internal+0x121/0x220 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1371c70&gt;] mdt_close+0x220/0x780 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fee38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f96e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f9a592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffff946bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:49 2018] [&lt;ffffffff94d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:49 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:49 2018] LustreError: dumping log to /tmp/lustre-log.1538474210.7634 [Tue Oct 2 11:56:49 2018] Pid: 69508, comm: mdt00_067 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:49 2018] Call Trace: [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06efa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc118aa99&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc12cac8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1369de7&gt;] mdt_empty_transno+0xf7/0x840 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136cf1e&gt;] mdt_mfd_open+0x8de/0xe70 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136da2b&gt;] mdt_finish_open+0x57b/0x690 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136f308&gt;] mdt_reint_open+0x17c8/0x3190 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1364b33&gt;] mdt_reint_rec+0x83/0x210 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc134636b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1346892&gt;] mdt_intent_reint+0x162/0x430 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1351671&gt;] mdt_intent_policy+0x441/0xc70 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f3b2ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f64b53&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fea452&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fee38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f96e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f9a592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffff946bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:49 2018] [&lt;ffffffff94d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:49 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:49 2018] Pid: 69607, comm: mdt00_105 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:49 2018] Call Trace: [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06efa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc118aa99&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc12cac8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1369de7&gt;] mdt_empty_transno+0xf7/0x840 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136cf1e&gt;] mdt_mfd_open+0x8de/0xe70 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136da2b&gt;] mdt_finish_open+0x57b/0x690 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136f308&gt;] mdt_reint_open+0x17c8/0x3190 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1364b33&gt;] mdt_reint_rec+0x83/0x210 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc134636b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1346892&gt;] mdt_intent_reint+0x162/0x430 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1351671&gt;] mdt_intent_policy+0x441/0xc70 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f3b2ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f64b53&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fea452&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fee38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f96e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f9a592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffff946bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:49 2018] [&lt;ffffffff94d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:49 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Tue Oct 2 11:56:49 2018] Pid: 69594, comm: mdt00_098 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [Tue Oct 2 11:56:49 2018] Call Trace: [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06ef5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc06efa93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc118aa99&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc12cac8e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1002512&gt;] top_trans_start+0x702/0x940 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc13ec3f1&gt;] lod_trans_start+0x31/0x40 [lod] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc148c1ba&gt;] mdd_trans_start+0x1a/0x20 [mdd] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc14763c0&gt;] mdd_create+0xbe0/0x1400 [mdd] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc136fcb5&gt;] mdt_reint_open+0x2175/0x3190 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1364b33&gt;] mdt_reint_rec+0x83/0x210 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc134636b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1346892&gt;] mdt_intent_reint+0x162/0x430 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc1351671&gt;] mdt_intent_policy+0x441/0xc70 [mdt] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f3b2ba&gt;] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f64b53&gt;] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fea452&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0fee38a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f96e4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffffc0f9a592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [Tue Oct 2 11:56:49 2018] [&lt;ffffffff946bae31&gt;] kthread+0xd1/0xe0 [Tue Oct 2 11:56:49 2018] [&lt;ffffffff94d1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [Tue Oct 2 11:56:49 2018] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>Please have a look at above, full dmesg and lustre log files are attached as an archive.</p> <p> </p> CentOS 7, 3.10.0-862.2.3.el7_lustre.x86_64, 1 MDS (+1 HA pair), 4 OSS LU-11465

OSS/MDS deadlock in 2.10.5

Bug Critical Resolved Incomplete Alex Zhuravlev Marek Magrys Wed, 3 Oct 2018 17:07:13 +0000 Fri, 12 Aug 2022 21:52:51 +0000 Fri, 12 Aug 2022 21:52:51 +0000 Lustre 2.10.4 Lustre 2.10.5 Lustre 2.10.6 3 19 <p>looks like a <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> duplicate</p> <p>Might be, but mentioned LU was a minor priority with fix just to master (2.12). In our case we just had the same problem on our main filesystem,  so I think it would be good to backport the resolution to 2.10,  as 2.10.6 release is around the corner.</p> <p>Is there a patch for 2.10.5  available somewhere?</p> <p>We seem to be having the same deadlock here.  Sometimes rebooting the MDS will clear it, sometimes not.  All Lustre servers are at 2.10.5, CentOS-7.5, were upgraded from IEEL-2.3/CentOS-6.  Clients are 2.10.2, CentOS-7.4.</p> <p> Oh, our OSTs are all ZFS, with MDT being ldiskfs.  Symptoms we see are hundreds of threads on the MDS in "D" state, and all clients hang.  Similar log entries as the othe poster.</p> <p>Suggestions on how to recover from or avoid this issue would be very welcome.</p> <p> </p> <p>Marion do you have any idea which aplication causes this problem in your environment?<br/> We @CYFRONET are still trying to isolate the problematic type of workload in order to make simple reproducer out of it.</p> <p>If you can share what applications are you running, maybe we'll find some common factor</p> <p>Thanks</p> <p>No, we've not identified a triggering workload.  We have a mixture of genomics analysis (many tiny jobs), MRI analysis, and Relion (CryoEM) MPI.  But so far no correlation with the issue occurring.</p> <p>We upgraded on 09-October, ran for about a week without anything unusual.  We did enable jobstats on Friday the 12th, and the first incident happened Sunday, with another on Tuesday (both of those cleared by rebooting just the MDS).</p> <p>Yesterday and today, we've had multiple instances, and rebooting the MDS/OSSs has not cleared things.  Immediately after rebooting, the MDS gets on the order of one stuck thread for each client (250+), and I/O hangs on clients.  We've been unable to clear the hangs except by rebooting all clients and servers, so our cluster has been unusable in that time.</p> <p>We disabled jobstats this morning, just prior to rebooting everything again, and are waiting to see if that has helped.</p> <p>Again, we need help, both with a patch, and with suggestions on an easier way to recover from this.</p> <p>We are still seeing the problem occur after disabling jobstats.</p> <p> </p> <p>For Whamcloud/DDN folks, we also had a case (#111147) with DDN engineering to assist with diagnosing this issue at OHSU.  You should be able to see the details here:</p> <p><a href="https://community.ddn.com/50038000012Pme2" class="external-link" target="_blank" rel="nofollow noopener">https://community.ddn.com/50038000012Pme2</a></p> <p> </p> <p>I'm able to reproduce the issue and couple patches under <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> seem to solve this. they are in testing now..</p> <p>Alex, we can test lustre with and without the patch on our testing instance tommorow if you can share the reproducer code.</p> <p>I think that the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch may be the root cause of for this problem. It sends a sync during chgrp to the OSTs, but I suspect that this induces a deadlock when the OSTs are trying to get new quota from the master.</p> <p>I'm thinking that we should just revert the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch for now, since it is also causing performance issues. In the meantime, it means that quota would no longer properly handle file group ownership changes if group quota is enabled.</p> <p>Hongchao, in the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch it may be OK to remove the <tt>dt_sync()</tt> call, and instead ensure in the OSP code to properly order the chgrp with previous setattr requests. That could avoid the deadlock here, as well as the performance problems reported in <a href="https://jira.whamcloud.com/browse/LU-11303" title="slow chgrp as user when quotas are enabled" class="issue-link" data-issue-key="LU-11303"><del>LU-11303</del></a>. Also, there is already a mechanism to force sync permission changes via <tt>permission_needs_sync()</tt> so we definitely should <b>not</b> be doing two syncs per setattr.</p> <p>I've pushed <a href="https://review.whamcloud.com/33676" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33676</a> to revert the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch since I think it is causing as many problems as it solves.</p> <p>Andreas, I think the deadlock wasn't caused by dt_sync() but rather by the sync settr RPC to OST (which is to verify the group has enough quota).</p> <p>In either case, reverting the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch should avoid this problem. I'll note that this issue is marked as affecting 2.10.4, 2.10.5, and 2.10.6, and the <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> patch was landed in 2.10.4 so it is at least a strong correlation for being the root cause.</p> <p>For 2.10.5 the <a href="https://review.whamcloud.com/33682" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33682</a> revert of <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> has passed testing and is the simplest solution. For 2.12 or 2.13 we are looking into a proper solution that preserves chgrp behavior without impacting the performance or inducing deadlocks.</p> <p>Ok, we could take the patches for a spin on one of our production systems. Should we pull both <a href="https://review.whamcloud.com/33682" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33682</a> (revert of <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a>) and <a href="https://review.whamcloud.com/33586" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33586</a> ("<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate") or would you recommend to stick just to 33682 now?</p> <p>Frequency of the issue got much lower during last weeks, so it will take some time before we confirm that the problem is solved.</p> <p>revert of <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> should be enough.</p> <p>Short update:</p> <p>After moving to 2.10.6 RC2 with <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> reverted we still see hung threads on OSSes</p> <p> </p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [507798.983947] LNet: Service thread pid 17130 was inactive <span class="code-keyword">for</span> 200.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [507799.001471] Pid: 17130, comm: ll_ost_io00_094 3.10.0-862.2.3.el7_lustre.x86_64 #1 SMP Tue May 22 17:36:23 UTC 2018 [507799.012231] Call Trace: [507799.014947] [&lt;ffffffffc0eac085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [507799.022084] [&lt;ffffffffc0eac368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [507799.029399] [&lt;ffffffffc0eac5e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [507799.036186] [&lt;ffffffffc0eaca93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [507799.043074] [&lt;ffffffffc0f22009&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [507799.050909] [&lt;ffffffffc0fecb1e&gt;] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [507799.058151] [&lt;ffffffffc11242ae&gt;] ofd_trans_start+0x6e/0xf0 [ofd] [507799.064513] [&lt;ffffffffc112a75b&gt;] ofd_commitrw_write+0x94b/0x1c90 [ofd] [507799.071400] [&lt;ffffffffc112e7a9&gt;] ofd_commitrw+0x4c9/0xae0 [ofd] [507799.077676] [&lt;ffffffffc0bb7824&gt;] obd_commitrw+0x2f3/0x336 [ptlrpc] [507799.084277] [&lt;ffffffffc0b8a38d&gt;] tgt_brw_write+0xffd/0x17d0 [ptlrpc] [507799.091046] [&lt;ffffffffc0b8638a&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [507799.098231] [&lt;ffffffffc0b2ee4b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [507799.106355] [&lt;ffffffffc0b32592&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [507799.112928] [&lt;ffffffffbc4bae31&gt;] kthread+0xd1/0xe0 [507799.118077] [&lt;ffffffffbcb1f5f7&gt;] ret_from_fork_nospec_end+0x0/0x39 [507799.124610] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff   </pre> </div></div> <p> </p> <p>have you updated MDS? can you provide all stack traces please?</p> <p>We have updated all servers: MDS + OSS  together.</p> <p>Please find the attached stack traces from servers, and corresponding stacks from clients affected by the hangups</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31602/31602_cyf_dec10_client_stacks.log" title="cyf_dec10_client_stacks.log attached to LU-11465">cyf_dec10_client_stacks.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31603/31603_cyf_dec10_server_stacks.log" title="cyf_dec10_server_stacks.log attached to LU-11465">cyf_dec10_server_stacks.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span><br/> There were no stack traces on MDS today. There were some in past days so i can attach them also if needed</p> <p> </p> <p> </p> <p>hm, I see only writing threads on OSS (ofd_commitrw_write()) and around), but nobody else. either the log is missing important trace(s) or it's something different. is it possible to dump all threads (echo t &gt;/proc/sysrq-trigger) and attach them to the ticket?</p> <p>Wi will bump servers kernel to latest RHEL 7.6 with patches from 2.10.6, our kernels from 2.10.4 lacked some ext4 patches which might be the root cause of the mentioned stack traces.</p> <p>Has this bug been really fixed in master?</p> <p><a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> hasn't been reverted in master, instead the MDS transaction became async.</p> <p>However, this bug is not a deadlock between MDS and OSS as the commit message claims. It's an OSS deadlock, MDS just waits for a reply forever with an open transaction freezing other MDS journal users. OSS deadlocks because it locks oo_sem and gets a transaction handle in different order for common operations and out operations, e.g.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> crash&gt; bt ffff88081329cf10 PID: 50457 TASK: ffff88081329cf10 CPU: 14 COMMAND: <span class="code-quote">"ll_ost_out01_00"</span> #0 [ffff88026aca3988] __schedule at ffffffff816b3de4 #1 [ffff88026aca3a10] schedule at ffffffff816b4409 #2 [ffff88026aca3a20] rwsem_down_write_failed at ffffffff816b5cf5 #3 [ffff88026aca3ab8] call_rwsem_down_write_failed at ffffffff81338247 #4 [ffff88026aca3b00] down_write at ffffffff816b356d #5 [ffff88026aca3b18] osd_write_lock at ffffffffc151eb0c [osd_ldiskfs] #6 [ffff88026aca3b40] out_tx_attr_set_exec at ffffffffc0ea3399 [ptlrpc] #7 [ffff88026aca3b78] out_tx_end at ffffffffc0e9d771 [ptlrpc] #8 [ffff88026aca3bb8] out_handle at ffffffffc0ea1952 [ptlrpc] #9 [ffff88026aca3cf8] tgt_request_handle at ffffffffc0e988ba [ptlrpc] #10 [ffff88026aca3d40] ptlrpc_server_handle_request at ffffffffc0e3df13 [ptlrpc] #11 [ffff88026aca3de0] ptlrpc_main at ffffffffc0e41862 [ptlrpc] #12 [ffff88026aca3ec8] kthread at ffffffff810b4031 #13 [ffff88026aca3f50] ret_from_fork at ffffffff816c155d crash&gt; bt 0xffff88038d7eaf70 PID: 114059 TASK: ffff88038d7eaf70 CPU: 11 COMMAND: <span class="code-quote">"ll_ost00_040"</span> #0 [ffff8804b073b968] __schedule at ffffffff816b3de4 #1 [ffff8804b073b9f8] schedule at ffffffff816b4409 #2 [ffff8804b073ba08] wait_transaction_locked at ffffffffc0739085 [jbd2] #3 [ffff8804b073ba60] add_transaction_credits at ffffffffc0739368 [jbd2] #4 [ffff8804b073bac0] start_this_handle at ffffffffc07395e1 [jbd2] #5 [ffff8804b073bb58] jbd2__journal_start at ffffffffc0739a93 [jbd2] #6 [ffff8804b073bba0] __ldiskfs_journal_start_sb at ffffffffc1469e59 [ldiskfs] #7 [ffff8804b073bbe0] osd_trans_start at ffffffffc152a2ce [osd_ldiskfs] #8 [ffff8804b073bc18] ofd_trans_start at ffffffffc15b01ae [ofd] #9 [ffff8804b073bc48] ofd_attr_set at ffffffffc15b3173 [ofd] #10 [ffff8804b073bca0] ofd_setattr_hdl at ffffffffc159e8ed [ofd] #11 [ffff8804b073bcf8] tgt_request_handle at ffffffffc0e988ba [ptlrpc] #12 [ffff8804b073bd40] ptlrpc_server_handle_request at ffffffffc0e3df13 [ptlrpc] #13 [ffff8804b073bde0] ptlrpc_main at ffffffffc0e41862 [ptlrpc] #14 [ffff8804b073bec8] kthread at ffffffff810b4031 #15 [ffff8804b073bf50] ret_from_fork at ffffffff816c155d </pre> </div></div> <blockquote><p>looks like a <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> duplicate</p></blockquote> <p>Ah, sorry, I missed this link while first reading this ticket. It really seems to be fixing this issue:</p> <p><a href="https://review.whamcloud.com/#/c/31293/60/lustre/ofd/ofd_objects.c" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31293/60/lustre/ofd/ofd_objects.c</a></p> <p>@Andrew Perepechko<br/> The patch you've mentioned is not applying cleanly on b2_10. Do you consider backporting it to the 2.10 line?<br/> In the past month we had 10+ lockups on OST related to this issue.</p> <p>The patch you mentioned is the subset of <a href="https://jira.whamcloud.com/browse/LU-11613" title="MDS and OSS locked up wait_transaction_locked+0x85/0xd0 [jbd2]" class="issue-link" data-issue-key="LU-11613"><del>LU-11613</del></a> , is there a plan to backport the whole set?<br/> Can the change from  <a href="https://review.whamcloud.com/#/c/31293/60/lustre/ofd/ofd_objects.c" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31293/60/lustre/ofd/ofd_objects.c</a> be safely used without other changes from <a href="https://jira.whamcloud.com/browse/LU-11613" title="MDS and OSS locked up wait_transaction_locked+0x85/0xd0 [jbd2]" class="issue-link" data-issue-key="LU-11613"><del>LU-11613</del></a>?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=luchuan_sugon" class="user-hover" rel="luchuan_sugon">luchuan_sugon</a>do you still need this ticket open?</p> Duplicate Related LU-10048 LU-5152 LU-11613 Development Rank 1|i003hj: Rank (Obsolete) 9223372036854775807 Severity