Tue Jul 6 08:55:03 2021] PANIC at dnode.c:1635:dnode_setdirty() [Tue Jul 6 08:55:03 2021] Showing stack for process 16376 [Tue Jul 6 08:55:03 2021] CPU: 19 PID: 16376 Comm: mdt01_019 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.2.1.el7_lustre.x86_64 #1 [Tue Jul 6 08:55:03 2021] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.3.6 06/03/2015 [Tue Jul 6 08:55:03 2021] Call Trace: [Tue Jul 6 08:55:03 2021] [] dump_stack+0x19/0x1b [Tue Jul 6 08:55:03 2021] [] spl_dumpstack+0x44/0x50 [spl] [Tue Jul 6 08:55:03 2021] [] spl_panic+0xc9/0x110 [spl] [Tue Jul 6 08:55:03 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:55:03 2021] [] ? __mutex_unlock_slowpath+0x70/0x90 [Tue Jul 6 08:55:03 2021] [] ? dbuf_rele_and_unlock+0x283/0x4c0 [zfs] [Tue Jul 6 08:55:03 2021] [] ? __cv_init+0x41/0x60 [spl] [Tue Jul 6 08:55:03 2021] [] ? dnode_cons+0x24a/0x260 [zfs] [Tue Jul 6 08:55:03 2021] [] ? spl_kmem_cache_alloc+0xbd/0x150 [spl] [Tue Jul 6 08:55:03 2021] [] ? mutex_lock+0x12/0x2f [Tue Jul 6 08:55:03 2021] [] ? dmu_objset_userquota_get_ids+0x23c/0x440 [zfs] [Tue Jul 6 08:55:03 2021] [] dnode_setdirty+0xe9/0xf0 [zfs] [Tue Jul 6 08:55:03 2021] [] dnode_allocate+0x18c/0x230 [zfs] [Tue Jul 6 08:55:03 2021] [] dmu_object_alloc_dnsize+0x34b/0x3e0 [zfs] [Tue Jul 6 08:55:03 2021] [] __osd_object_create+0x82/0x170 [osd_zfs] [Tue Jul 6 08:55:03 2021] [] ? spl_kmem_zalloc+0xd8/0x180 [spl] [Tue Jul 6 08:55:03 2021] [] osd_mkreg+0x7d/0x210 [osd_zfs] [Tue Jul 6 08:55:03 2021] [] ? __kmalloc_node+0x1d1/0x2b0 [Tue Jul 6 08:55:03 2021] [] osd_create+0x316/0xaf0 [osd_zfs] [Tue Jul 6 08:55:03 2021] [] lod_sub_create+0x1f5/0x480 [lod] [Tue Jul 6 08:55:03 2021] [] lod_create+0x69/0x340 [lod] [Tue Jul 6 08:55:03 2021] [] ? osd_trans_create+0x410/0x410 [osd_zfs] [Tue Jul 6 08:55:03 2021] [] mdd_create_object_internal+0xc3/0x300 [mdd] [Tue Jul 6 08:55:03 2021] [] mdd_create_object+0x7b/0x820 [mdd] [Tue Jul 6 08:55:03 2021] [] mdd_create+0xdd8/0x14a0 [mdd] [Tue Jul 6 08:55:03 2021] [] mdt_reint_open+0x224f/0x3240 [mdt] [Tue Jul 6 08:55:03 2021] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Tue Jul 6 08:55:03 2021] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Jul 6 08:55:03 2021] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Jul 6 08:55:03 2021] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Tue Jul 6 08:55:03 2021] [] mdt_intent_open+0x82/0x3a0 [mdt] [Tue Jul 6 08:55:03 2021] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Tue Jul 6 08:55:03 2021] [] mdt_intent_policy+0x435/0xd80 [mdt] [Tue Jul 6 08:55:03 2021] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Tue Jul 6 08:55:03 2021] [] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [Tue Jul 6 08:55:03 2021] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Tue Jul 6 08:55:03 2021] [] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:55:03 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:55:03 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:55:03 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:55:03 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:55:03 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:55:03 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:55:03 2021] [] ? insert_kthread_work+0x40/0x40 Message from syslogd@emds1 at Jul 6 08:55:03 ... kernel:VERIFY(dnode_add_ref(dn, (void *)(uintptr_t)tx->tx_txg)) failed Message from syslogd@emds1 at Jul 6 08:55:03 ... kernel:PANIC at dnode.c:1635:dnode_setdirty() [Tue Jul 6 08:58:10 2021] INFO: task txg_quiesce:5801 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] txg_quiesce D ffff9abb7fcdacc0 0 5801 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] ? task_rq_unlock+0x20/0x20 [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] txg_quiesce_thread+0x2fb/0x410 [zfs] [Tue Jul 6 08:58:10 2021] [] ? txg_init+0x2b0/0x2b0 [zfs] [Tue Jul 6 08:58:10 2021] [] thread_generic_wrapper+0x73/0x80 [spl] [Tue Jul 6 08:58:10 2021] [] ? __thread_exit+0x20/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt_rdpg01_000:8836 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt_rdpg01_000 D ffff9aeb7f15acc0 0 8836 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] dmu_tx_wait+0x213/0x3c0 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_tx_assign+0x92/0x490 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_trans_start+0x199/0x440 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] top_trans_start+0x702/0x940 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? mdd_xattr_changelog_type.isra.35+0xc8/0x180 [mdd] [Tue Jul 6 08:58:10 2021] [] lod_trans_start+0x34/0x40 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_trans_start+0x1a/0x20 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_xattr_set+0x445/0x1880 [mdd] [Tue Jul 6 08:58:10 2021] [] ? lod_read_unlock+0x3b/0xd0 [lod] [Tue Jul 6 08:58:10 2021] [] ? mdd_read_unlock+0x3b/0xd0 [mdd] [Tue Jul 6 08:58:10 2021] [] mdt_set_som+0xf4/0x2f0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_big_xattr_get+0x11a/0x810 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_lsom_update+0x120/0x440 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_mfd_close+0x47f/0x870 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_close_internal+0x121/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_close+0x220/0x780 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt_rdpg01_001:8837 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt_rdpg01_001 D ffff9aeb7f15acc0 0 8837 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] dmu_tx_wait+0x213/0x3c0 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_tx_assign+0x92/0x490 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_trans_start+0x199/0x440 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] top_trans_start+0x702/0x940 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? mdd_xattr_changelog_type.isra.35+0xc8/0x180 [mdd] [Tue Jul 6 08:58:10 2021] [] lod_trans_start+0x34/0x40 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_trans_start+0x1a/0x20 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_xattr_set+0x445/0x1880 [mdd] [Tue Jul 6 08:58:10 2021] [] ? lod_read_unlock+0x3b/0xd0 [lod] [Tue Jul 6 08:58:10 2021] [] ? mdd_read_unlock+0x3b/0xd0 [mdd] [Tue Jul 6 08:58:10 2021] [] mdt_set_som+0xf4/0x2f0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_big_xattr_get+0x11a/0x810 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_lsom_update+0x120/0x440 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_mfd_close+0x47f/0x870 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_close_internal+0x121/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_close+0x220/0x780 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt01_003:12254 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt01_003 D ffff9aeb7f29acc0 0 12254 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] dmu_tx_wait+0x213/0x3c0 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_tx_assign+0x92/0x490 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_trans_start+0x199/0x440 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] top_trans_start+0x702/0x940 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? lod_declare_attr_set+0xba/0x6b0 [lod] [Tue Jul 6 08:58:10 2021] [] lod_trans_start+0x34/0x40 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_trans_start+0x1a/0x20 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_attr_set+0x649/0xda0 [mdd] [Tue Jul 6 08:58:10 2021] [] ? mdt_reint_object_lock+0x2c/0x60 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_attr_set+0x1ea/0x770 [mdt] [Tue Jul 6 08:58:10 2021] [] ? tgt_fmd_put+0x23/0x130 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] mdt_reint_setattr+0x599/0xc50 [mdt] [Tue Jul 6 08:58:10 2021] [] ? ucred_set_audit_enabled.isra.14+0x22/0x60 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint+0x67/0x140 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt00_003:13028 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt00_003 D ffff9abb7f7dacc0 0 13028 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] ? task_rq_unlock+0x20/0x20 [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:10 2021] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:10 2021] [] ? tsi2mdt_info+0x40/0x120 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt00_013:13047 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt00_013 D ffff9abb7fb5acc0 0 13047 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] dmu_tx_wait+0x213/0x3c0 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_tx_assign+0x92/0x490 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_trans_start+0x199/0x440 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] mdt_empty_transno+0xf7/0x850 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_mfd_open+0x8de/0xe70 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_pack_acl2body+0x1c2/0x9f0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_finish_open+0x64b/0x760 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_open+0x178c/0x3240 [mdt] [Tue Jul 6 08:58:10 2021] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Tue Jul 6 08:58:10 2021] [] ? lu_ucred+0x1e/0x30 [obdclass] [Tue Jul 6 08:58:10 2021] [] ? mdt_ucred+0x15/0x20 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_root_squash+0x21/0x430 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_intent_open+0x82/0x3a0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Tue Jul 6 08:58:10 2021] [] mdt_intent_policy+0x435/0xd80 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [Tue Jul 6 08:58:10 2021] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Tue Jul 6 08:58:10 2021] [] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt00_024:13137 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt00_024 D ffff9abb7fc9acc0 0 13137 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:10 2021] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:10 2021] [] ? tsi2mdt_info+0x40/0x120 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt00_025:13139 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt00_025 D ffff9abb7fc5acc0 0 13139 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:10 2021] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:10 2021] [] ? tsi2mdt_info+0x40/0x120 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt01_018:16375 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt01_018 D ffff9aeb7f45acc0 0 16375 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:10 2021] [] dmu_tx_wait+0x213/0x3c0 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_tx_assign+0x92/0x490 [zfs] [Tue Jul 6 08:58:10 2021] [] osd_trans_start+0x199/0x440 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] top_trans_start+0x702/0x940 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] lod_trans_start+0x34/0x40 [lod] [Tue Jul 6 08:58:10 2021] [] mdd_trans_start+0x1a/0x20 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_rename+0x66c/0x14a0 [mdd] [Tue Jul 6 08:58:10 2021] [] mdt_reint_rename+0x1a67/0x2900 [mdt] [Tue Jul 6 08:58:10 2021] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint+0x67/0x140 [mdt] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? queued_spin_lock_slowpath+0xb/0xf [Tue Jul 6 08:58:10 2021] [] ? _raw_spin_lock+0x20/0x30 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] INFO: task mdt01_019:16376 blocked for more than 120 seconds. [Tue Jul 6 08:58:10 2021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Jul 6 08:58:10 2021] mdt01_019 D ffff9aeb7f05acc0 0 16376 2 0x00000000 [Tue Jul 6 08:58:10 2021] Call Trace: [Tue Jul 6 08:58:10 2021] [] schedule+0x29/0x70 [Tue Jul 6 08:58:10 2021] [] spl_panic+0xf5/0x110 [spl] [Tue Jul 6 08:58:10 2021] [] ? wake_up_atomic_t+0x30/0x30 [Tue Jul 6 08:58:10 2021] [] ? __mutex_unlock_slowpath+0x70/0x90 [Tue Jul 6 08:58:10 2021] [] ? dbuf_rele_and_unlock+0x283/0x4c0 [zfs] [Tue Jul 6 08:58:10 2021] [] ? __cv_init+0x41/0x60 [spl] [Tue Jul 6 08:58:10 2021] [] ? dnode_cons+0x24a/0x260 [zfs] [Tue Jul 6 08:58:10 2021] [] ? spl_kmem_cache_alloc+0xbd/0x150 [spl] [Tue Jul 6 08:58:10 2021] [] ? mutex_lock+0x12/0x2f [Tue Jul 6 08:58:10 2021] [] ? dmu_objset_userquota_get_ids+0x23c/0x440 [zfs] [Tue Jul 6 08:58:10 2021] [] dnode_setdirty+0xe9/0xf0 [zfs] [Tue Jul 6 08:58:10 2021] [] dnode_allocate+0x18c/0x230 [zfs] [Tue Jul 6 08:58:10 2021] [] dmu_object_alloc_dnsize+0x34b/0x3e0 [zfs] [Tue Jul 6 08:58:10 2021] [] __osd_object_create+0x82/0x170 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] ? spl_kmem_zalloc+0xd8/0x180 [spl] [Tue Jul 6 08:58:10 2021] [] osd_mkreg+0x7d/0x210 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] ? __kmalloc_node+0x1d1/0x2b0 [Tue Jul 6 08:58:10 2021] [] osd_create+0x316/0xaf0 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] lod_sub_create+0x1f5/0x480 [lod] [Tue Jul 6 08:58:10 2021] [] lod_create+0x69/0x340 [lod] [Tue Jul 6 08:58:10 2021] [] ? osd_trans_create+0x410/0x410 [osd_zfs] [Tue Jul 6 08:58:10 2021] [] mdd_create_object_internal+0xc3/0x300 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_create_object+0x7b/0x820 [mdd] [Tue Jul 6 08:58:10 2021] [] mdd_create+0xdd8/0x14a0 [mdd] [Tue Jul 6 08:58:10 2021] [] mdt_reint_open+0x224f/0x3240 [mdt] [Tue Jul 6 08:58:10 2021] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Tue Jul 6 08:58:10 2021] [] mdt_reint_rec+0x83/0x210 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] mdt_intent_open+0x82/0x3a0 [mdt] [Tue Jul 6 08:58:10 2021] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Tue Jul 6 08:58:10 2021] [] mdt_intent_policy+0x435/0xd80 [mdt] [Tue Jul 6 08:58:10 2021] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [Tue Jul 6 08:58:10 2021] [] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [Tue Jul 6 08:58:10 2021] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Tue Jul 6 08:58:10 2021] [] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Tue Jul 6 08:58:10 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __wake_up+0x13/0x20 [Tue Jul 6 08:58:10 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] ? __schedule+0x3af/0x860 [Tue Jul 6 08:58:10 2021] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Tue Jul 6 08:58:10 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:10 2021] [] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jul 6 08:58:10 2021] [] ? insert_kthread_work+0x40/0x40 [Tue Jul 6 08:58:24 2021] LNet: Service thread pid 13139 was inactive for 200.45s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Tue Jul 6 08:58:24 2021] Pid: 13139, comm: mdt00_025 3.10.0-1160.2.1.el7_lustre.x86_64 #1 SMP Wed Dec 9 20:53:35 UTC 2020 [Tue Jul 6 08:58:24 2021] Call Trace: [Tue Jul 6 08:58:24 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:24 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:24 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:24 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:24 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:24 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:24 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:24 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:24 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:24 2021] [] ret_from_fork_nospec_end+0x0/0x39 [Tue Jul 6 08:58:24 2021] [] 0xffffffffffffffff [Tue Jul 6 08:58:24 2021] LustreError: dumping log to /tmp/lustre-log.1625587104.13139 [Tue Jul 6 08:58:24 2021] Pid: 13028, comm: mdt00_003 3.10.0-1160.2.1.el7_lustre.x86_64 #1 SMP Wed Dec 9 20:53:35 UTC 2020 [Tue Jul 6 08:58:24 2021] Call Trace: [Tue Jul 6 08:58:24 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:24 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:24 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:24 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:24 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:24 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:24 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:24 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:24 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] kthread+0xd1/0xe0 [Tue Jul 6 08:58:24 2021] [] ret_from_fork_nospec_end+0x0/0x39 [Tue Jul 6 08:58:24 2021] [] 0xffffffffffffffff [Tue Jul 6 08:58:24 2021] Pid: 13137, comm: mdt00_024 3.10.0-1160.2.1.el7_lustre.x86_64 #1 SMP Wed Dec 9 20:53:35 UTC 2020 [Tue Jul 6 08:58:24 2021] Call Trace: [Tue Jul 6 08:58:24 2021] [] cv_wait_common+0x125/0x150 [spl] [Tue Jul 6 08:58:24 2021] [] __cv_wait+0x15/0x20 [spl] [Tue Jul 6 08:58:24 2021] [] txg_wait_synced+0xef/0x140 [zfs] [Tue Jul 6 08:58:24 2021] [] osd_object_sync+0x16e/0x180 [osd_zfs] [Tue Jul 6 08:58:24 2021] [] lod_object_sync+0x3e/0xe0 [lod] [Tue Jul 6 08:58:24 2021] [] mdd_object_sync+0x64/0x180 [mdd] [Tue Jul 6 08:58:24 2021] [] mdt_object_sync.isra.54+0x11d/0x270 [mdt] [Tue Jul 6 08:58:24 2021] [] mdt_sync+0x11f/0x290 [mdt] [Tue Jul 6 08:58:24 2021] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Tue Jul 6 08:58:24 2021] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [Tue Jul 6 08:58:24 2021] [