Nov 13 14:12:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 306.649994
Nov 13 14:12:52 ascratch-oss03  [2405496.042715] INFO: task kswapd0:449 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03  [2405496.057717]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.042715] INFO: task kswapd0:449 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.057717]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.074725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03  [2405496.074725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03  [2405496.091245] task:kswapd0         state:D stack:    0 pid:  449 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.091245] task:kswapd0         state:D stack:    0 pid:  449 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.108258] Call Trace:
Nov 13 14:12:52 ascratch-oss03  [2405496.108258] Call Trace:
Nov 13 14:12:52 ascratch-oss03  [2405496.119594]  __schedule+0x2bd/0x760
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.119594]  __schedule+0x2bd/0x760
Nov 13 14:12:52 ascratch-oss03  [2405496.131642]  schedule+0x37/0xa0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.131642]  schedule+0x37/0xa0
Nov 13 14:12:52 ascratch-oss03  [2405496.143333]  rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.143333]  rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.156690]  lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:52 ascratch-oss03  [2405496.156690]  lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:52 ascratch-oss03  [2405496.170621]  do_shrink_slab+0x54/0x2a0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.170621]  do_shrink_slab+0x54/0x2a0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.183027]  shrink_slab+0xbe/0x2e0
Nov 13 14:12:52 ascratch-oss03  [2405496.183027]  shrink_slab+0xbe/0x2e0
Nov 13 14:12:52 ascratch-oss03  [2405496.195173]  ? __switch_to_asm+0x35/0x70
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.195173]  ? __switch_to_asm+0x35/0x70
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.207734]  shrink_node+0x19c/0x6b0
Nov 13 14:12:52 ascratch-oss03  [2405496.207734]  shrink_node+0x19c/0x6b0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.220175]  balance_pgdat+0x2d7/0x550
Nov 13 14:12:52 ascratch-oss03  [2405496.220175]  balance_pgdat+0x2d7/0x550
Nov 13 14:12:52 ascratch-oss03  [2405496.232474]  kswapd+0x201/0x3c0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.232474]  kswapd+0x201/0x3c0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.244095]  ? finish_wait+0x80/0x80
Nov 13 14:12:52 ascratch-oss03  [2405496.244095]  ? finish_wait+0x80/0x80
Nov 13 14:12:52 ascratch-oss03  [2405496.256377]  ? balance_pgdat+0x550/0x550
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.256377]  ? balance_pgdat+0x550/0x550
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.268804]  kthread+0x116/0x130
Nov 13 14:12:52 ascratch-oss03  [2405496.268804]  kthread+0x116/0x130
Nov 13 14:12:52 ascratch-oss03  [2405496.280341]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.280341]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.292920]  ret_from_fork+0x1f/0x40
Nov 13 14:12:52 ascratch-oss03  [2405496.292920]  ret_from_fork+0x1f/0x40
Nov 13 14:12:52 ascratch-oss03  [2405496.304426] INFO: task dbuf_evict:16024 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.304426] INFO: task dbuf_evict:16024 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.319552]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03  [2405496.319552]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03  [2405496.336000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.336000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.352137] task:dbuf_evict      state:D stack:    0 pid:16024 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03  [2405496.352137] task:dbuf_evict      state:D stack:    0 pid:16024 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03  [2405496.368853] Call Trace:
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.368853] Call Trace:
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.379626]  __schedule+0x2bd/0x760
Nov 13 14:12:52 ascratch-oss03  [2405496.379626]  __schedule+0x2bd/0x760
Nov 13 14:12:52 ascratch-oss03  [2405496.391433]  schedule+0x37/0xa0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.391433]  schedule+0x37/0xa0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.402851]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:52 ascratch-oss03  [2405496.402851]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:52 ascratch-oss03  [2405496.415846]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.415846]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:52 ascratch-oss03  [2405496.428557]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.428557]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:52 ascratch-oss03  [2405496.441091]  ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.441091]  ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:52 ascratch-oss03  [2405496.453784]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.453784]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.466348]  dbuf_evict_thread+0x122/0x1d0 [zfs]
Nov 13 14:12:52 ascratch-oss03  [2405496.466348]  dbuf_evict_thread+0x122/0x1d0 [zfs]
Nov 13 14:12:52 ascratch-oss03  [2405496.479222]  ? dbuf_evict_one+0x120/0x120 [zfs]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.479222]  ? dbuf_evict_one+0x120/0x120 [zfs]
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.491995]  thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:52 ascratch-oss03  [2405496.491995]  thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:52 ascratch-oss03  [2405496.505172]  kthread+0x116/0x130
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.505172]  kthread+0x116/0x130
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.516675]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:52 ascratch-oss03  [2405496.516675]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:52 ascratch-oss03  [2405496.529271]  ret_from_fork+0x1f/0x40
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.529271]  ret_from_fork+0x1f/0x40
Nov 13 14:12:52 ascratch-oss03  [2405496.540939] INFO: task z_wr_int:276737 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.540939] INFO: task z_wr_int:276737 blocked for more than 120 seconds.
Nov 13 14:12:52 ascratch-oss03  [2405496.555841]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.555841]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.571949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03  [2405496.571949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:52 ascratch-oss03  [2405496.587551] task:z_wr_int        state:D stack:    0 pid:276737 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.587551] task:z_wr_int        state:D stack:    0 pid:276737 ppid:     2 flags:0x80004000
Nov 13 14:12:52 ascratch-oss03 kernel: [2405496.603642] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405496.603642] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405496.613459]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.613459]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.624098]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405496.624098]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405496.634110]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.634110]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.645325]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405496.645325]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405496.656254]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.656254]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.666998]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.666998]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.677711]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.677711]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.688933]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.688933]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.700232]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.700232]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.711652]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.711652]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.722209]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.722209]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.732729]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.732729]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.742867]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.742867]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.753074]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.753074]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.763332]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03  [2405496.763332]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03  [2405496.772851]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.772851]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.785459]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405496.785459]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405496.796400]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.796400]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.805859]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03  [2405496.805859]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03  [2405496.816798]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.816798]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.826674] INFO: task z_wr_int:276738 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03  [2405496.826674] INFO: task z_wr_int:276738 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03  [2405496.839927]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.839927]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.854860] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03  [2405496.854860] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03  [2405496.869440] task:z_wr_int        state:D stack:    0 pid:276738 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.869440] task:z_wr_int        state:D stack:    0 pid:276738 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.884962] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405496.884962] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405496.894133]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.894133]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.904237]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405496.904237]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405496.913837]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.913837]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.924717]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405496.924717]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405496.935270]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.935270]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.945750]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.945750]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.956330]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.956330]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.967547]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.967547]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.978962]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.978962]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405496.990558]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405496.990558]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.001289]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.001289]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.011976]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.011976]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.022309]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.022309]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.032561]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.032561]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.043052]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03  [2405497.043052]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03  [2405497.052935]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.052935]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.065685]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.065685]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.076794]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03  [2405497.076794]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03  [2405497.086372]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.086372]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.097039]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03  [2405497.097039]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03  [2405497.106702] INFO: task z_wr_int:276741 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.106702] INFO: task z_wr_int:276741 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.119800]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03  [2405497.119800]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03  [2405497.134248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.134248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.148397] task:z_wr_int        state:D stack:    0 pid:276741 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03  [2405497.148397] task:z_wr_int        state:D stack:    0 pid:276741 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03  [2405497.163329] Call Trace:
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.163329] Call Trace:
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.172210]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03  [2405497.172210]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03  [2405497.182125]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.182125]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.191664]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03  [2405497.191664]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03  [2405497.202625]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.202625]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405497.213352]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.213352]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.223961]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.223961]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.234627]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.234627]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.245913]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.245913]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.257387]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.257387]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.269068]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.269068]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.279926]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.279926]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.290709]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.290709]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.301132]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.301132]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.311422]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.311422]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.321941]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.321941]  ? wake_up_q+0x80/0x80
Nov 13 14:12:53 ascratch-oss03  [2405497.331661]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.331661]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.344412]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.344412]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.355518]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.355518]  kthread+0x116/0x130
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.365055]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03  [2405497.365055]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:53 ascratch-oss03  [2405497.375737]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.375737]  ret_from_fork+0x1f/0x40
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.385426] INFO: task z_wr_int:276743 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03  [2405497.385426] INFO: task z_wr_int:276743 blocked for more than 120 seconds.
Nov 13 14:12:53 ascratch-oss03  [2405497.398321]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.398321]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.412776] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03  [2405497.412776] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:53 ascratch-oss03  [2405497.426929] task:z_wr_int        state:D stack:    0 pid:276743 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.426929] task:z_wr_int        state:D stack:    0 pid:276743 ppid:     2 flags:0x80004000
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.441873] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405497.441873] Call Trace:
Nov 13 14:12:53 ascratch-oss03  [2405497.450968]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.450968]  __schedule+0x2bd/0x760
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.460899]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405497.460899]  schedule+0x37/0xa0
Nov 13 14:12:53 ascratch-oss03  [2405497.470451]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.470451]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.481432]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405497.481432]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:53 ascratch-oss03  [2405497.492170]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.492170]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.502788]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.502788]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.513481]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.513481]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.524778]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.524778]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.536336]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.536336]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.548087]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.548087]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.558947]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.558947]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.569727]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.569727]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:53 ascratch-oss03  [2405497.580153]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.580153]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:53 ascratch-oss03 kernel: [2405497.590433]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:53 ascratch-oss03  [2405497.590433]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405497.600919]  ? wake_up_q+0x80/0x80
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.600919]  ? wake_up_q+0x80/0x80
Nov 13 14:12:54 ascratch-oss03  [2405497.610634]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.610634]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.623379]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405497.623379]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405497.634486]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.634486]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.644024]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03  [2405497.644024]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03  [2405497.654881]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.654881]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.664540] INFO: task z_wr_int:276744 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03  [2405497.664540] INFO: task z_wr_int:276744 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03  [2405497.677429]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.677429]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.691880] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03  [2405497.691880] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03  [2405497.706035] task:z_wr_int        state:D stack:    0 pid:276744 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.706035] task:z_wr_int        state:D stack:    0 pid:276744 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.721150] Call Trace:
Nov 13 14:12:54 ascratch-oss03  [2405497.721150] Call Trace:
Nov 13 14:12:54 ascratch-oss03  [2405497.730038]  __schedule+0x2bd/0x760
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.730038]  __schedule+0x2bd/0x760
ov 13 14:12:54 ascratch-oss03 kernel: [2405497.739954]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03  [2405497.739954]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03  [2405497.749646]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.749646]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.760609]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:54 ascratch-oss03  [2405497.760609]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:54 ascratch-oss03  [2405497.771339]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.771339]  dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.781940]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.781940]  dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.792612]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.792612]  dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.803889]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.803889]  ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.815361]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.815361]  ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.827030]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.827030]  dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.837871]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.837871]  arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.848643]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.848643]  zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.859055]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.859055]  zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405497.869338]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.869338]  taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.879820]  ? wake_up_q+0x80/0x80
Nov 13 14:12:54 ascratch-oss03  [2405497.879820]  ? wake_up_q+0x80/0x80
Nov 13 14:12:54 ascratch-oss03  [2405497.889527]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.889527]  ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.902267]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405497.902267]  ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405497.913376]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.913376]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.922917]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03  [2405497.922917]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03  [2405497.933598]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.933598]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03  [2405497.943282] INFO: task txg_sync:285715 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.943282] INFO: task txg_sync:285715 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.956178]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03  [2405497.956178]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.970631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03  [2405497.970631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03  [2405497.984816] task:txg_sync        state:D stack:    0 pid:285715 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.984816] task:txg_sync        state:D stack:    0 pid:285715 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03 kernel: [2405497.999753] Call Trace:
Nov 13 14:12:54 ascratch-oss03  [2405497.999753] Call Trace:
Nov 13 14:12:54 ascratch-oss03  [2405498.008639]  __schedule+0x2bd/0x760
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.008639]  __schedule+0x2bd/0x760
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.018739]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03  [2405498.018739]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03  [2405498.028287]  schedule_timeout+0x197/0x300
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.028287]  schedule_timeout+0x197/0x300
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.038704]  ? __next_timer_interrupt+0xf0/0xf0
Nov 13 14:12:54 ascratch-oss03  [2405498.038704]  ? __next_timer_interrupt+0xf0/0xf0
Nov 13 14:12:54 ascratch-oss03  [2405498.049715]  ? zio_issue_async+0xe/0x20 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.049715]  ? zio_issue_async+0xe/0x20 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.060486]  ? zio_nowait+0xb6/0x140 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.060486]  ? zio_nowait+0xb6/0x140 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.070928]  io_schedule_timeout+0x19/0x40
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.070928]  io_schedule_timeout+0x19/0x40
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.081342]  __cv_timedwait_common+0x12e/0x160 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.081342]  __cv_timedwait_common+0x12e/0x160 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.092672]  ? finish_wait+0x80/0x80
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.092672]  ? finish_wait+0x80/0x80
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.102598]  __cv_timedwait_io+0x15/0x20 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.102598]  __cv_timedwait_io+0x15/0x20 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.113500]  zio_wait+0x133/0x270 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.113500]  zio_wait+0x133/0x270 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.123775]  dsl_pool_sync+0x106/0x510 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.123775]  dsl_pool_sync+0x106/0x510 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.134507]  spa_sync+0x56d/0xfc0 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.134507]  spa_sync+0x56d/0xfc0 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.144695]  ? _cond_resched+0x15/0x30
Nov 13 14:12:54 ascratch-oss03  [2405498.144695]  ? _cond_resched+0x15/0x30
Nov 13 14:12:54 ascratch-oss03  [2405498.154787]  ? spa_txg_history_init_io+0xfe/0x110 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.154787]  ? spa_txg_history_init_io+0xfe/0x110 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.166374]  txg_sync_thread+0x29f/0x480 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.166374]  txg_sync_thread+0x29f/0x480 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.177174]  ? txg_thread_exit.isra.11+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.177174]  ? txg_thread_exit.isra.11+0x60/0x60 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.188669]  ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.188669]  ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.199233]  thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.199233]  thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.210320]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03  [2405498.210320]  kthread+0x116/0x130
Nov 13 14:12:54 ascratch-oss03  [2405498.219800]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.219800]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.230197]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03  [2405498.230197]  ret_from_fork+0x1f/0x40
Nov 13 14:12:54 ascratch-oss03  [2405498.239633] INFO: task ll_ost_io00_002:286084 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.239633] INFO: task ll_ost_io00_002:286084 blocked for more than 120 seconds.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.253271]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03  [2405498.253271]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:54 ascratch-oss03  [2405498.267773] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.267773] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.281967] task:ll_ost_io00_002 state:D stack:    0 pid:286084 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03  [2405498.281967] task:ll_ost_io00_002 state:D stack:    0 pid:286084 ppid:     2 flags:0x80004000
Nov 13 14:12:54 ascratch-oss03  [2405498.296917] Call Trace:
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.296917] Call Trace:
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.305809]  __schedule+0x2bd/0x760
Nov 13 14:12:54 ascratch-oss03  [2405498.305809]  __schedule+0x2bd/0x760
Nov 13 14:12:54 ascratch-oss03  [2405498.315934]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.315934]  schedule+0x37/0xa0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.325505]  rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:54 ascratch-oss03  [2405498.325505]  rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.336699]  lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:54 ascratch-oss03  [2405498.336699]  lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:54 ascratch-oss03  [2405498.348498]  do_shrink_slab+0x54/0x2a0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.348498]  do_shrink_slab+0x54/0x2a0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.358708]  shrink_slab+0xbe/0x2e0
Nov 13 14:12:54 ascratch-oss03  [2405498.358708]  shrink_slab+0xbe/0x2e0
Nov 13 14:12:54 ascratch-oss03  [2405498.368615]  shrink_node+0x19c/0x6b0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.368615]  shrink_node+0x19c/0x6b0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.378585]  do_try_to_free_pages+0xc9/0x3e0
Nov 13 14:12:54 ascratch-oss03  [2405498.378585]  do_try_to_free_pages+0xc9/0x3e0
Nov 13 14:12:54 ascratch-oss03  [2405498.389295]  try_to_free_pages+0xef/0x1c0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.389295]  try_to_free_pages+0xef/0x1c0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.399736]  __alloc_pages_slowpath+0x39c/0xcd0
Nov 13 14:12:54 ascratch-oss03  [2405498.399736]  __alloc_pages_slowpath+0x39c/0xcd0
Nov 13 14:12:54 ascratch-oss03  [2405498.410715]  ? __kmalloc+0x102/0x240
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.410715]  ? __kmalloc+0x102/0x240
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.420850]  __alloc_pages_nodemask+0x2db/0x310
Nov 13 14:12:54 ascratch-oss03  [2405498.420850]  __alloc_pages_nodemask+0x2db/0x310
Nov 13 14:12:54 ascratch-oss03  [2405498.431735]  __get_free_pages+0xa/0x30
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.431735]  __get_free_pages+0xa/0x30
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.441805]  spl_cache_refill+0x360/0x6b0 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.441805]  spl_cache_refill+0x360/0x6b0 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.452686]  ? _cond_resched+0x15/0x30
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.452686]  ? _cond_resched+0x15/0x30
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.462746]  ? mutex_lock+0xe/0x30
Nov 13 14:12:54 ascratch-oss03  [2405498.462746]  ? mutex_lock+0xe/0x30
Nov 13 14:12:54 ascratch-oss03  [2405498.472378]  ? aggsum_add+0x17a/0x190 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.472378]  ? aggsum_add+0x17a/0x190 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.482670]  spl_kmem_cache_alloc+0x13c/0x160 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.482670]  spl_kmem_cache_alloc+0x13c/0x160 [spl]
Nov 13 14:12:54 ascratch-oss03  [2405498.493555]  arc_buf_alloc_impl.isra.43+0x1a0/0x320 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.493555]  arc_buf_alloc_impl.isra.43+0x1a0/0x320 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.504857]  arc_alloc_buf+0x6b/0xd0 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.504857]  arc_alloc_buf+0x6b/0xd0 [zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.514774]  arc_loan_buf+0x1f/0x50 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.514774]  arc_loan_buf+0x1f/0x50 [zfs]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.524586]  osd_bufs_get+0x74a/0x1080 [osd_zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.524586]  osd_bufs_get+0x74a/0x1080 [osd_zfs]
Nov 13 14:12:54 ascratch-oss03  [2405498.535132]  ofd_preprw_write.isra.28+0x56a/0x1240 [ofd]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.535132]  ofd_preprw_write.isra.28+0x56a/0x1240 [ofd]
Nov 13 14:12:54 ascratch-oss03  [2405498.546434]  ofd_preprw+0x7aa/0x8f0 [ofd]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.546434]  ofd_preprw+0x7aa/0x8f0 [ofd]
Nov 13 14:12:54 ascratch-oss03  [2405498.556418]  obd_preprw+0x19d/0x360 [ptlrpc]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.556418]  obd_preprw+0x19d/0x360 [ptlrpc]
Nov 13 14:12:54 ascratch-oss03  [2405498.566570]  tgt_brw_write+0x10e2/0x1b10 [ptlrpc]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.566570]  tgt_brw_write+0x10e2/0x1b10 [ptlrpc]
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.577157]  ? enqueue_task_fair+0x95/0x580
Nov 13 14:12:54 ascratch-oss03  [2405498.577157]  ? enqueue_task_fair+0x95/0x580
Nov 13 14:12:54 ascratch-oss03  [2405498.587258]  ? flush_work+0x42/0x1c0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.587258]  ? flush_work+0x42/0x1c0
Nov 13 14:12:54 ascratch-oss03 kernel: [2405498.596679]  ? internal_add_timer+0x42/0x60
Nov 13 14:12:54 ascratch-oss03  [2405498.596679]  ? internal_add_timer+0x42/0x60
Nov 13 14:12:55 ascratch-oss03  [2405498.606700]  ? _cond_resched+0x15/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.606700]  ? _cond_resched+0x15/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.616465]  ? mutex_lock+0xe/0x30
Nov 13 14:12:55 ascratch-oss03  [2405498.616465]  ? mutex_lock+0xe/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.625672]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.625672]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.636590]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.636590]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.648274]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.648274]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.660105]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.660105]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.670519]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.670519]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.681518]  kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03  [2405498.681518]  kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03  [2405498.690651]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.690651]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.701016]  ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03  [2405498.701016]  ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03  [2405498.710484] INFO: task ll_ost_io01_001:286086 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.710484] INFO: task ll_ost_io01_001:286086 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.723935]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03  [2405498.723935]       Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.738398] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03  [2405498.738398] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03  [2405498.752556] task:ll_ost_io01_001 state:D stack:    0 pid:286086 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.752556] task:ll_ost_io01_001 state:D stack:    0 pid:286086 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.767357] Call Trace:
Nov 13 14:12:55 ascratch-oss03  [2405498.767357] Call Trace:
Nov 13 14:12:55 ascratch-oss03  [2405498.775989]  __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.775989]  __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.785580]  schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03  [2405498.785580]  schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03  [2405498.794626]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.794626]  schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.804987]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03  [2405498.804987]  __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03  [2405498.815138]  dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.815138]  dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.824855]  dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.824855]  dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.834991]  dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.834991]  dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.844523]  dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.844523]  dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.855749]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.855749]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.866858]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.866858]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:12:55 ascratch-oss03  [2405498.877320]  ? kiblnd_launch_tx+0xd1/0xc00 [ko2iblnd]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.877320]  ? kiblnd_launch_tx+0xd1/0xc00 [ko2iblnd]
Nov 13 14:12:55 ascratch-oss03  [2405498.888250]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.888250]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:12:55 ascratch-oss03  [2405498.899381]  ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.899381]  ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:12:55 ascratch-oss03  [2405498.909316]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.909316]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.919987]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.919987]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.930490]  ? __kmalloc+0x102/0x240
Nov 13 14:12:55 ascratch-oss03  [2405498.930490]  ? __kmalloc+0x102/0x240
Nov 13 14:12:55 ascratch-oss03  [2405498.939959]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.939959]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Nov 13 14:12:55 ascratch-oss03  [2405498.951269]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.951269]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.961855]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.961855]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.973114]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.973114]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.984687]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.984687]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405498.996046]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405498.996046]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405499.007554]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.007554]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.018667]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405499.018667]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405499.030524]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.030524]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405499.042509]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.042509]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03  [2405499.053107]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.053107]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.064285]  kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03  [2405499.064285]  kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03  [2405499.073675]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.073675]  ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: [2405499.084346]  ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03  [2405499.084346]  ret_from_fork+0x1f/0x40
Nov 13 14:12:52 ascratch-oss03 kernel: INFO: task kswapd0:449 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:kswapd0         state:D stack:    0 pid:  449 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:55 ascratch-oss03 kernel: lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:55 ascratch-oss03 kernel: do_shrink_slab+0x54/0x2a0
Nov 13 14:12:55 ascratch-oss03 kernel: shrink_slab+0xbe/0x2e0
Nov 13 14:12:55 ascratch-oss03 kernel: ? __switch_to_asm+0x35/0x70
Nov 13 14:12:55 ascratch-oss03 kernel: shrink_node+0x19c/0x6b0
Nov 13 14:12:55 ascratch-oss03 kernel: balance_pgdat+0x2d7/0x550
Nov 13 14:12:55 ascratch-oss03 kernel: kswapd+0x201/0x3c0
Nov 13 14:12:55 ascratch-oss03 kernel: ? finish_wait+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? balance_pgdat+0x550/0x550
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task dbuf_evict:16024 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:dbuf_evict      state:D stack:    0 pid:16024 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_thread+0x122/0x1d0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dbuf_evict_one+0x120/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task z_wr_int:276737 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:z_wr_int        state:D stack:    0 pid:276737 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? wake_up_q+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task z_wr_int:276738 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:z_wr_int        state:D stack:    0 pid:276738 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? wake_up_q+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task z_wr_int:276741 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:z_wr_int        state:D stack:    0 pid:276741 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? wake_up_q+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task z_wr_int:276743 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:z_wr_int        state:D stack:    0 pid:276743 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? wake_up_q+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task z_wr_int:276744 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:z_wr_int        state:D stack:    0 pid:276744 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dir_diduse_space+0x146/0x180 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? dsl_dataset_block_born+0x240/0x380 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_write_done+0xe9/0x220 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_write_done+0x8f/0x410 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_done+0x400/0x1010 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_execute+0x90/0xf0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: taskq_thread+0x2ed/0x530 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? wake_up_q+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_taskq_member.isra.12.constprop.18+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task txg_sync:285715 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:txg_sync        state:D stack:    0 pid:285715 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_timeout+0x197/0x300
Nov 13 14:12:55 ascratch-oss03 kernel: ? __next_timer_interrupt+0xf0/0xf0
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_issue_async+0xe/0x20 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? zio_nowait+0xb6/0x140 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: io_schedule_timeout+0x19/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: __cv_timedwait_common+0x12e/0x160 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? finish_wait+0x80/0x80
Nov 13 14:12:55 ascratch-oss03 kernel: __cv_timedwait_io+0x15/0x20 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: zio_wait+0x133/0x270 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dsl_pool_sync+0x106/0x510 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: spa_sync+0x56d/0xfc0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? _cond_resched+0x15/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: ? spa_txg_history_init_io+0xfe/0x110 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: txg_sync_thread+0x29f/0x480 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? txg_thread_exit.isra.11+0x60/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? __thread_exit+0x20/0x20 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task ll_ost_io00_002:286084 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:ll_ost_io00_002 state:D stack:    0 pid:286084 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Nov 13 14:12:55 ascratch-oss03 kernel: lu_cache_shrink_count+0x1e/0x130 [obdclass]
Nov 13 14:12:55 ascratch-oss03 kernel: do_shrink_slab+0x54/0x2a0
Nov 13 14:12:55 ascratch-oss03 kernel: shrink_slab+0xbe/0x2e0
Nov 13 14:12:55 ascratch-oss03 kernel: shrink_node+0x19c/0x6b0
Nov 13 14:12:55 ascratch-oss03 kernel: do_try_to_free_pages+0xc9/0x3e0
Nov 13 14:12:55 ascratch-oss03 kernel: try_to_free_pages+0xef/0x1c0
Nov 13 14:12:55 ascratch-oss03 kernel: __alloc_pages_slowpath+0x39c/0xcd0
Nov 13 14:12:55 ascratch-oss03 kernel: ? __kmalloc+0x102/0x240
Nov 13 14:12:55 ascratch-oss03 kernel: __alloc_pages_nodemask+0x2db/0x310
Nov 13 14:12:55 ascratch-oss03 kernel: __get_free_pages+0xa/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: spl_cache_refill+0x360/0x6b0 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: ? _cond_resched+0x15/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: ? mutex_lock+0xe/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: ? aggsum_add+0x17a/0x190 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: spl_kmem_cache_alloc+0x13c/0x160 [spl]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_buf_alloc_impl.isra.43+0x1a0/0x320 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_alloc_buf+0x6b/0xd0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: arc_loan_buf+0x1f/0x50 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: osd_bufs_get+0x74a/0x1080 [osd_zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ofd_preprw_write.isra.28+0x56a/0x1240 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: ofd_preprw+0x7aa/0x8f0 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: obd_preprw+0x19d/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: tgt_brw_write+0x10e2/0x1b10 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? enqueue_task_fair+0x95/0x580
Nov 13 14:12:55 ascratch-oss03 kernel: ? flush_work+0x42/0x1c0
Nov 13 14:12:55 ascratch-oss03 kernel: ? internal_add_timer+0x42/0x60
Nov 13 14:12:55 ascratch-oss03 kernel: ? _cond_resched+0x15/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: ? mutex_lock+0xe/0x30
Nov 13 14:12:55 ascratch-oss03 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:12:55 ascratch-oss03 kernel: INFO: task ll_ost_io01_001:286086 blocked for more than 120 seconds.
Nov 13 14:12:55 ascratch-oss03 kernel:      Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Nov 13 14:12:55 ascratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 13 14:12:55 ascratch-oss03 kernel: task:ll_ost_io01_001 state:D stack:    0 pid:286086 ppid:     2 flags:0x80004000
Nov 13 14:12:55 ascratch-oss03 kernel: Call Trace:
Nov 13 14:12:55 ascratch-oss03 kernel: __schedule+0x2bd/0x760
Nov 13 14:12:55 ascratch-oss03 kernel: schedule+0x37/0xa0
Nov 13 14:12:55 ascratch-oss03 kernel: schedule_preempt_disabled+0xa/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: __mutex_lock.isra.6+0x2b5/0x4a0
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:12:55 ascratch-oss03 kernel: ? kiblnd_launch_tx+0xd1/0xc00 [ko2iblnd]
Nov 13 14:12:55 ascratch-oss03 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:12:55 ascratch-oss03 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? __kmalloc+0x102/0x240
Nov 13 14:12:55 ascratch-oss03 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Nov 13 14:12:55 ascratch-oss03 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Nov 13 14:12:55 ascratch-oss03 kernel: kthread+0x116/0x130
Nov 13 14:12:55 ascratch-oss03 kernel: ? kthread_flush_work_fn+0x10/0x10
Nov 13 14:12:55 ascratch-oss03 kernel: ret_from_fork+0x1f/0x40
Nov 13 14:13:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 318.380005
Nov 13 14:13:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 325.119995
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770545] ptlrpc_watchdog_fire: 17 callbacks suppressed
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770549] Lustre: ll_ost_io01_187: service thread pid 520306 was inactive for 200.147 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770551] Lustre: ll_ost_io01_180: service thread pid 520293 was inactive for 200.338 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770553] Lustre: ll_ost_io00_183: service thread pid 520588 was inactive for 201.697 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770555] Pid: 520212, comm: ll_ost_io01_143 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770556] Lustre: Skipped 15 previous similar messages
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770558] Lustre: Skipped 2 previous similar messages
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770558] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770651] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770676] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770705] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770731] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770757] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770770] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770785] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770790] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770861] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770899] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770934] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770968] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770979] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770989] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770990] Pid: 519689, comm: ll_ost_io01_118 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.770991] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771031] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771056] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771081] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771107] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771132] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771140] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771146] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771151] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771189] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771226] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771260] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771295] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771296] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771298] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771300] Pid: 520588, comm: ll_ost_io00_183 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771300] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771373] [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771398] [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771423] [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771435] [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771447] [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771527] [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771564] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771602] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771638] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771643] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.771646] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03  [2405569.770545] ptlrpc_watchdog_fire: 17 callbacks suppressed
Nov 13 14:14:06 ascratch-oss03 kernel: [2405569.782218] Lustre: Skipped 52 previous similar messages
Nov 13 14:14:06 ascratch-oss03  [2405569.770549] Lustre: ll_ost_io01_187: service thread pid 520306 was inactive for 200.147 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03  [2405569.770551] Lustre: ll_ost_io01_180: service thread pid 520293 was inactive for 200.338 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03  [2405569.770553] Lustre: ll_ost_io00_183: service thread pid 520588 was inactive for 201.697 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:14:06 ascratch-oss03  [2405569.770555] Pid: 520212, comm: ll_ost_io01_143 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03  [2405569.770556] Lustre: Skipped 15 previous similar messages
Nov 13 14:14:06 ascratch-oss03  [2405569.770558] Lustre: Skipped 2 previous similar messages
Nov 13 14:14:06 ascratch-oss03  [2405569.770558] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03  [2405569.770651] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: ptlrpc_watchdog_fire: 17 callbacks suppressed
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: ll_ost_io01_187: service thread pid 520306 was inactive for 200.147 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: ll_ost_io01_180: service thread pid 520293 was inactive for 200.338 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: ll_ost_io00_183: service thread pid 520588 was inactive for 201.697 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:14:06 ascratch-oss03 kernel: Pid: 520212, comm: ll_ost_io01_143 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: Skipped 15 previous similar messages
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: Skipped 2 previous similar messages
Nov 13 14:14:06 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03 kernel: Pid: 519689, comm: ll_ost_io01_118 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03 kernel: Pid: 520588, comm: ll_ost_io00_183 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: Skipped 52 previous similar messages
Nov 13 14:14:06 ascratch-oss03  [2405569.770676] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.770705] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.770731] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.770757] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.770770] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.770785] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03  [2405569.770790] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03  [2405569.770861] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.770899] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.770934] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.770968] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.770979] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03  [2405569.770989] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03  [2405569.770990] Pid: 519689, comm: ll_ost_io01_118 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03  [2405569.770991] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03  [2405569.771031] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771056] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771081] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771107] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771132] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771140] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771146] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:14:06 ascratch-oss03  [2405569.771151] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:14:06 ascratch-oss03  [2405569.771189] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771226] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771260] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771295] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771296] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03  [2405569.771298] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03  [2405569.771300] Pid: 520588, comm: ll_ost_io00_183 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:14:06 ascratch-oss03  [2405569.771300] Call Trace TBD:
Nov 13 14:14:06 ascratch-oss03  [2405569.771373] [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771398] [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771423] [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771435] [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:14:06 ascratch-oss03  [2405569.771447] [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:14:06 ascratch-oss03  [2405569.771527] [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771564] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771602] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771638] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:14:06 ascratch-oss03  [2405569.771643] [<0>] kthread+0x116/0x130
Nov 13 14:14:06 ascratch-oss03  [2405569.771646] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:14:06 ascratch-oss03  [2405569.782218] Lustre: Skipped 52 previous similar messages
Nov 13 14:14:06 ascratch-oss03  [2405570.470403] Lustre: ll_ost_io01_124: service thread pid 519953 was inactive for 201.302 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: [2405570.470403] Lustre: ll_ost_io01_124: service thread pid 519953 was inactive for 201.302 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:06 ascratch-oss03 kernel: Lustre: ll_ost_io01_124: service thread pid 519953 was inactive for 201.302 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03  [2405573.866541] Lustre: ll_ost_io01_153: service thread pid 520235 was inactive for 201.198 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03 kernel: [2405573.866541] Lustre: ll_ost_io01_153: service thread pid 520235 was inactive for 201.198 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03 kernel: [2405573.866542] Lustre: ll_ost_io01_083: service thread pid 519431 was inactive for 202.477 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03 kernel: [2405573.866545] Lustre: Skipped 13 previous similar messages
Nov 13 14:14:10 ascratch-oss03 kernel: [2405573.894806] Lustre: Skipped 53 previous similar messages
Nov 13 14:14:10 ascratch-oss03  [2405573.866542] Lustre: ll_ost_io01_083: service thread pid 519431 was inactive for 202.477 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03  [2405573.866545] Lustre: Skipped 13 previous similar messages
Nov 13 14:14:10 ascratch-oss03  [2405573.894806] Lustre: Skipped 53 previous similar messages
Nov 13 14:14:10 ascratch-oss03 kernel: Lustre: ll_ost_io01_153: service thread pid 520235 was inactive for 201.198 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03 kernel: Lustre: ll_ost_io01_083: service thread pid 519431 was inactive for 202.477 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:10 ascratch-oss03 kernel: Lustre: Skipped 13 previous similar messages
Nov 13 14:14:10 ascratch-oss03 kernel: Lustre: Skipped 53 previous similar messages
Nov 13 14:14:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 329.190002
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.962535] Lustre: ll_ost01_024: service thread pid 458724 was inactive for 202.971 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.962537] Lustre: ll_ost_io01_056: service thread pid 518575 was inactive for 200.073 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.962538] Lustre: ll_ost_io01_190: service thread pid 520313 was inactive for 203.072 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.962541] Lustre: Skipped 3 previous similar messages
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.962541] Lustre: Skipped 6 previous similar messages
Nov 13 14:14:14 ascratch-oss03  [2405577.962535] Lustre: ll_ost01_024: service thread pid 458724 was inactive for 202.971 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: [2405577.992317] Lustre: Skipped 30 previous similar messages
Nov 13 14:14:14 ascratch-oss03  [2405577.962537] Lustre: ll_ost_io01_056: service thread pid 518575 was inactive for 200.073 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03  [2405577.962538] Lustre: ll_ost_io01_190: service thread pid 520313 was inactive for 203.072 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03  [2405577.962541] Lustre: Skipped 3 previous similar messages
Nov 13 14:14:14 ascratch-oss03  [2405577.962541] Lustre: Skipped 6 previous similar messages
Nov 13 14:14:14 ascratch-oss03  [2405577.992317] Lustre: Skipped 30 previous similar messages
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: ll_ost01_024: service thread pid 458724 was inactive for 202.971 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: ll_ost_io01_056: service thread pid 518575 was inactive for 200.073 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: ll_ost_io01_190: service thread pid 520313 was inactive for 203.072 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: Skipped 3 previous similar messages
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: Skipped 6 previous similar messages
Nov 13 14:14:14 ascratch-oss03 kernel: Lustre: Skipped 30 previous similar messages
Nov 13 14:14:30 ascratch-oss03  [2405594.346503] Lustre: ll_ost_io01_074: service thread pid 519387 was inactive for 202.910 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:30 ascratch-oss03  [2405594.378733] Lustre: Skipped 83 previous similar messages
Nov 13 14:14:30 ascratch-oss03 kernel: [2405594.346503] Lustre: ll_ost_io01_074: service thread pid 519387 was inactive for 202.910 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:30 ascratch-oss03 kernel: [2405594.378733] Lustre: Skipped 83 previous similar messages
Nov 13 14:14:30 ascratch-oss03 kernel: Lustre: ll_ost_io01_074: service thread pid 519387 was inactive for 202.910 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:14:30 ascratch-oss03 kernel: Lustre: Skipped 83 previous similar messages
Nov 13 14:14:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 331.940002
Nov 13 14:15:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 334.100006
Nov 13 14:15:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 336.049988
Nov 13 14:16:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 337.829987
Nov 13 14:16:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 338.910004
Nov 13 14:17:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 340.250000
Nov 13 14:17:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 341.559998
Nov 13 14:18:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 343.519989
Nov 13 14:18:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 345.380005
Nov 13 14:18:56 ascratch-oss03  [2405860.586030] Lustre: ll_ost_io00_117: service thread pid 519669 was inactive for 352.529 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:18:57 ascratch-oss03  [2405860.618908] Lustre: Skipped 18 previous similar messages
Nov 13 14:18:57 ascratch-oss03 kernel: [2405860.586030] Lustre: ll_ost_io00_117: service thread pid 519669 was inactive for 352.529 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:18:57 ascratch-oss03 kernel: [2405860.618908] Lustre: Skipped 18 previous similar messages
Nov 13 14:18:57 ascratch-oss03 kernel: Lustre: ll_ost_io00_117: service thread pid 519669 was inactive for 352.529 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:18:57 ascratch-oss03 kernel: Lustre: Skipped 18 previous similar messages
Nov 13 14:19:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 347.500000
Nov 13 14:19:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 348.720001
Nov 13 14:20:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 349.500000
Nov 13 14:20:20 ascratch-oss03 systemd[1]: Starting system activity accounting tool...
Nov 13 14:20:20 ascratch-oss03 systemd[1]: sysstat-collect.service: Succeeded.
Nov 13 14:20:20 ascratch-oss03 systemd[1]: Started system activity accounting tool.
Nov 13 14:20:37 ascratch-oss03  [2405961.193856] Lustre: 518995:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d8825c07 x1749022444503040/t0(0) o3->28207b59-0676-4a3c-95a8-1029943e593b@172.22.16.115@o2ib20:247/0 lens 488/440 e 24 to 0 dl 1668345642 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u98:6'
Nov 13 14:20:37 ascratch-oss03 kernel: [2405961.193856] Lustre: 518995:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d8825c07 x1749022444503040/t0(0) o3->28207b59-0676-4a3c-95a8-1029943e593b@172.22.16.115@o2ib20:247/0 lens 488/440 e 24 to 0 dl 1668345642 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u98:6'
Nov 13 14:20:37 ascratch-oss03 kernel: Lustre: 518995:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d8825c07 x1749022444503040/t0(0) o3->28207b59-0676-4a3c-95a8-1029943e593b@172.22.16.115@o2ib20:247/0 lens 488/440 e 24 to 0 dl 1668345642 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u98:6'
Nov 13 14:20:38 ascratch-oss03  [2405962.345860] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000002477b53e x1749022410686912/t0(0) o4->fd60270e-5704-43f8-8152-589a84c52c2f@172.22.16.116@o2ib20:248/0 lens 488/448 e 24 to 0 dl 1668345643 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u100:6.'
Nov 13 14:20:38 ascratch-oss03  [2405962.405933] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message
Nov 13 14:20:38 ascratch-oss03 kernel: [2405962.345860] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000002477b53e x1749022410686912/t0(0) o4->fd60270e-5704-43f8-8152-589a84c52c2f@172.22.16.116@o2ib20:248/0 lens 488/448 e 24 to 0 dl 1668345643 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u100:6.'
Nov 13 14:20:38 ascratch-oss03 kernel: [2405962.405933] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message
Nov 13 14:20:38 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000002477b53e x1749022410686912/t0(0) o4->fd60270e-5704-43f8-8152-589a84c52c2f@172.22.16.116@o2ib20:248/0 lens 488/448 e 24 to 0 dl 1668345643 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u100:6.'
Nov 13 14:20:38 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message
Nov 13 14:20:39 ascratch-oss03  [2405963.369856] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000157b9c06 x1749021768286912/t0(0) o3->2178e911-1911-4fa0-b727-8a1167b63f3f@172.22.18.228@o2ib20:249/0 lens 488/440 e 24 to 0 dl 1668345644 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:7.0'
Nov 13 14:20:39 ascratch-oss03  [2405963.431996] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages
Nov 13 14:20:39 ascratch-oss03 kernel: [2405963.369856] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000157b9c06 x1749021768286912/t0(0) o3->2178e911-1911-4fa0-b727-8a1167b63f3f@172.22.18.228@o2ib20:249/0 lens 488/440 e 24 to 0 dl 1668345644 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:7.0'
Nov 13 14:20:39 ascratch-oss03 kernel: [2405963.431996] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages
Nov 13 14:20:39 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000157b9c06 x1749021768286912/t0(0) o3->2178e911-1911-4fa0-b727-8a1167b63f3f@172.22.18.228@o2ib20:249/0 lens 488/440 e 24 to 0 dl 1668345644 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:7.0'
Nov 13 14:20:39 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages
Nov 13 14:20:41 ascratch-oss03  [2405965.417843] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000f29a33df x1749014296460864/t0(0) o3->a85a350f-f8a8-48bc-8bb2-5e92f3e44408@172.22.18.182@o2ib20:251/0 lens 504/440 e 24 to 0 dl 1668345646 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:20:41 ascratch-oss03  [2405965.480054] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 53 previous similar messages
Nov 13 14:20:41 ascratch-oss03 kernel: [2405965.417843] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000f29a33df x1749014296460864/t0(0) o3->a85a350f-f8a8-48bc-8bb2-5e92f3e44408@172.22.18.182@o2ib20:251/0 lens 504/440 e 24 to 0 dl 1668345646 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:20:41 ascratch-oss03 kernel: [2405965.480054] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 53 previous similar messages
Nov 13 14:20:41 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000f29a33df x1749014296460864/t0(0) o3->a85a350f-f8a8-48bc-8bb2-5e92f3e44408@172.22.18.182@o2ib20:251/0 lens 504/440 e 24 to 0 dl 1668345646 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:20:41 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 53 previous similar messages
Nov 13 14:20:43 ascratch-oss03  [2405966.898422] Lustre: ascratch-OST0006: Client 28207b59-0676-4a3c-95a8-1029943e593b (at 172.22.16.115@o2ib20) reconnecting
Nov 13 14:20:43 ascratch-oss03 kernel: [2405966.898422] Lustre: ascratch-OST0006: Client 28207b59-0676-4a3c-95a8-1029943e593b (at 172.22.16.115@o2ib20) reconnecting
Nov 13 14:20:43 ascratch-oss03 kernel: Lustre: ascratch-OST0006: Client 28207b59-0676-4a3c-95a8-1029943e593b (at 172.22.16.115@o2ib20) reconnecting
Nov 13 14:20:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 351.230011
Nov 13 14:20:44 ascratch-oss03  [2405967.599726] Lustre: ascratch-OST0007: Client 783c69e7-658d-42b5-935a-a79012c6ce1d (at 172.22.18.59@o2ib20) reconnecting
Nov 13 14:20:44 ascratch-oss03 kernel: [2405967.599726] Lustre: ascratch-OST0007: Client 783c69e7-658d-42b5-935a-a79012c6ce1d (at 172.22.18.59@o2ib20) reconnecting
Nov 13 14:20:44 ascratch-oss03 kernel: Lustre: ascratch-OST0007: Client 783c69e7-658d-42b5-935a-a79012c6ce1d (at 172.22.18.59@o2ib20) reconnecting
Nov 13 14:20:45 ascratch-oss03  [2405968.647491] Lustre: ascratch-OST0008: Client af56d4a1-a0dc-4a7f-9b99-7be3f96f9011 (at 172.22.17.139@o2ib20) reconnecting
Nov 13 14:20:45 ascratch-oss03  [2405968.670651] Lustre: Skipped 3 previous similar messages
Nov 13 14:20:45 ascratch-oss03 kernel: [2405968.647491] Lustre: ascratch-OST0008: Client af56d4a1-a0dc-4a7f-9b99-7be3f96f9011 (at 172.22.17.139@o2ib20) reconnecting
Nov 13 14:20:45 ascratch-oss03 kernel: [2405968.670651] Lustre: Skipped 3 previous similar messages
Nov 13 14:20:45 ascratch-oss03 kernel: Lustre: ascratch-OST0008: Client af56d4a1-a0dc-4a7f-9b99-7be3f96f9011 (at 172.22.17.139@o2ib20) reconnecting
Nov 13 14:20:45 ascratch-oss03 kernel: Lustre: Skipped 3 previous similar messages
Nov 13 14:20:45 ascratch-oss03  [2405969.513834] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d1c8f0c3 x1749011518716288/t0(0) o4->1d2b9910-6d97-49c8-afb3-8e96904ee3b2@172.22.17.7@o2ib20:255/0 lens 488/448 e 24 to 0 dl 1668345650 ref 2 fl Interpret:/0/0 rc 0/0 job:'loop0.0'
Nov 13 14:20:45 ascratch-oss03  [2405969.580602] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 81 previous similar messages
Nov 13 14:20:46 ascratch-oss03 kernel: [2405969.513834] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d1c8f0c3 x1749011518716288/t0(0) o4->1d2b9910-6d97-49c8-afb3-8e96904ee3b2@172.22.17.7@o2ib20:255/0 lens 488/448 e 24 to 0 dl 1668345650 ref 2 fl Interpret:/0/0 rc 0/0 job:'loop0.0'
Nov 13 14:20:46 ascratch-oss03 kernel: [2405969.580602] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 81 previous similar messages
Nov 13 14:20:46 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000d1c8f0c3 x1749011518716288/t0(0) o4->1d2b9910-6d97-49c8-afb3-8e96904ee3b2@172.22.17.7@o2ib20:255/0 lens 488/448 e 24 to 0 dl 1668345650 ref 2 fl Interpret:/0/0 rc 0/0 job:'loop0.0'
Nov 13 14:20:46 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 81 previous similar messages
Nov 13 14:20:47 ascratch-oss03  [2405971.177856] Lustre: ascratch-OST0007: Client 6d6e3dec-91ba-45d5-b36d-4792dbb7180b (at 172.22.18.181@o2ib20) reconnecting
Nov 13 14:20:47 ascratch-oss03  [2405971.202203] Lustre: Skipped 25 previous similar messages
Nov 13 14:20:47 ascratch-oss03 kernel: [2405971.177856] Lustre: ascratch-OST0007: Client 6d6e3dec-91ba-45d5-b36d-4792dbb7180b (at 172.22.18.181@o2ib20) reconnecting
Nov 13 14:20:47 ascratch-oss03 kernel: [2405971.202203] Lustre: Skipped 25 previous similar messages
Nov 13 14:20:47 ascratch-oss03 kernel: Lustre: ascratch-OST0007: Client 6d6e3dec-91ba-45d5-b36d-4792dbb7180b (at 172.22.18.181@o2ib20) reconnecting
Nov 13 14:20:47 ascratch-oss03 kernel: Lustre: Skipped 25 previous similar messages
Nov 13 14:20:51 ascratch-oss03  [2405975.508306] Lustre: ascratch-OST0008: Client 0cfb7fac-f9ce-4fd7-90f2-79c8381888cb (at 172.22.18.185@o2ib20) reconnecting
Nov 13 14:20:51 ascratch-oss03  [2405975.532702] Lustre: Skipped 25 previous similar messages
Nov 13 14:20:51 ascratch-oss03 kernel: [2405975.508306] Lustre: ascratch-OST0008: Client 0cfb7fac-f9ce-4fd7-90f2-79c8381888cb (at 172.22.18.185@o2ib20) reconnecting
Nov 13 14:20:51 ascratch-oss03 kernel: [2405975.532702] Lustre: Skipped 25 previous similar messages
Nov 13 14:20:51 ascratch-oss03 kernel: Lustre: ascratch-OST0008: Client 0cfb7fac-f9ce-4fd7-90f2-79c8381888cb (at 172.22.18.185@o2ib20) reconnecting
Nov 13 14:20:51 ascratch-oss03 kernel: Lustre: Skipped 25 previous similar messages
Nov 13 14:20:53 ascratch-oss03  [2405976.687257] Lustre: ascratch-OST0006: deleting orphan objects from 0x0:11220230 to 0x0:11221425
Nov 13 14:20:53 ascratch-oss03 kernel: [2405976.687257] Lustre: ascratch-OST0006: deleting orphan objects from 0x0:11220230 to 0x0:11221425
Nov 13 14:20:53 ascratch-oss03 kernel: Lustre: ascratch-OST0006: deleting orphan objects from 0x0:11220230 to 0x0:11221425
Nov 13 14:20:54 ascratch-oss03  [2405977.577822] Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000007c47bc6e x1749025834794176/t0(0) o4->cf67d5f4-48a0-4498-938a-4230c51f9325@172.22.16.79@o2ib20:263/0 lens 856/536 e 22 to 0 dl 1668345658 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:10.'
Nov 13 14:20:54 ascratch-oss03  [2405977.646997] Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 57 previous similar messages
Nov 13 14:20:54 ascratch-oss03 kernel: [2405977.577822] Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000007c47bc6e x1749025834794176/t0(0) o4->cf67d5f4-48a0-4498-938a-4230c51f9325@172.22.16.79@o2ib20:263/0 lens 856/536 e 22 to 0 dl 1668345658 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:10.'
Nov 13 14:20:54 ascratch-oss03 kernel: [2405977.646997] Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 57 previous similar messages
Nov 13 14:20:54 ascratch-oss03 kernel: Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000007c47bc6e x1749025834794176/t0(0) o4->cf67d5f4-48a0-4498-938a-4230c51f9325@172.22.16.79@o2ib20:263/0 lens 856/536 e 22 to 0 dl 1668345658 ref 2 fl Interpret:/0/0 rc 0/0 job:'kworker/u99:10.'
Nov 13 14:20:54 ascratch-oss03 kernel: Lustre: 519682:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 57 previous similar messages
Nov 13 14:21:00 ascratch-oss03  [2405984.003386] Lustre: ascratch-OST0006: Client 725bd21a-cf04-492a-ae4c-baf25512514a (at 172.22.16.51@o2ib20) reconnecting
Nov 13 14:21:00 ascratch-oss03  [2405984.027944] Lustre: Skipped 31 previous similar messages
Nov 13 14:21:00 ascratch-oss03 kernel: [2405984.003386] Lustre: ascratch-OST0006: Client 725bd21a-cf04-492a-ae4c-baf25512514a (at 172.22.16.51@o2ib20) reconnecting
Nov 13 14:21:00 ascratch-oss03 kernel: [2405984.027944] Lustre: Skipped 31 previous similar messages
Nov 13 14:21:00 ascratch-oss03 kernel: Lustre: ascratch-OST0006: Client 725bd21a-cf04-492a-ae4c-baf25512514a (at 172.22.16.51@o2ib20) reconnecting
Nov 13 14:21:00 ascratch-oss03 kernel: Lustre: Skipped 31 previous similar messages
Nov 13 14:21:10 ascratch-oss03  [2405994.089790] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000256c71d1 x1742718587152896/t0(0) o4->3e49cfe7-425b-4a05-8403-7daae7a5ca43@172.22.17.121@o2ib20:280/0 lens 488/0 e 12 to 0 dl 1668345675 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1408012'
Nov 13 14:21:10 ascratch-oss03  [2405994.159404] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 198 previous similar messages
Nov 13 14:21:10 ascratch-oss03 kernel: [2405994.089790] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000256c71d1 x1742718587152896/t0(0) o4->3e49cfe7-425b-4a05-8403-7daae7a5ca43@172.22.17.121@o2ib20:280/0 lens 488/0 e 12 to 0 dl 1668345675 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1408012'
Nov 13 14:21:10 ascratch-oss03 kernel: [2405994.159404] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 198 previous similar messages
Nov 13 14:21:10 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000256c71d1 x1742718587152896/t0(0) o4->3e49cfe7-425b-4a05-8403-7daae7a5ca43@172.22.17.121@o2ib20:280/0 lens 488/0 e 12 to 0 dl 1668345675 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1408012'
Nov 13 14:21:10 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 198 previous similar messages
Nov 13 14:21:13 ascratch-oss03  [2405997.111681] LustreError: 1821467:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 602s
Nov 13 14:21:13 ascratch-oss03 kernel: [2405997.111681] LustreError: 1821467:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 602s
Nov 13 14:21:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 354.549988
Nov 13 14:21:13 ascratch-oss03 kernel: LustreError: 1821467:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 602s
Nov 13 14:21:13 ascratch-oss03 pacemaker-attrd[232300]: notice: Setting lustred[ascratch-oss03]: 1 -> 0
Nov 13 14:21:13 ascratch-oss03 pacemaker-controld[232302]: notice: Requesting local execution of stop operation for lustre-OST0006 on ascratch-oss03
Nov 13 14:21:13 ascratch-oss03 Lustre(lustre-OST0006)[1821471]: INFO: Starting to unmount ost06/ost06
Nov 13 14:21:13 ascratch-oss03 systemd[1]: mnt-lustre-local-ascratch\x2dOST0006.mount: Succeeded.
Nov 13 14:21:16 ascratch-oss03  [2406000.106611] Lustre: ascratch-OST0007: Client 1573cdaa-40e4-4679-9437-0547526a7e31 (at 172.22.18.72@o2ib20) reconnecting
Nov 13 14:21:16 ascratch-oss03  [2406000.131834] Lustre: Skipped 68 previous similar messages
Nov 13 14:21:16 ascratch-oss03 kernel: [2406000.106611] Lustre: ascratch-OST0007: Client 1573cdaa-40e4-4679-9437-0547526a7e31 (at 172.22.18.72@o2ib20) reconnecting
Nov 13 14:21:16 ascratch-oss03 kernel: [2406000.131834] Lustre: Skipped 68 previous similar messages
Nov 13 14:21:16 ascratch-oss03 kernel: Lustre: ascratch-OST0007: Client 1573cdaa-40e4-4679-9437-0547526a7e31 (at 172.22.18.72@o2ib20) reconnecting
Nov 13 14:21:16 ascratch-oss03 kernel: Lustre: Skipped 68 previous similar messages
Nov 13 14:21:23 ascratch-oss03  [2406007.163910] LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 612s
Nov 13 14:21:23 ascratch-oss03  [2406007.204091] LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:23 ascratch-oss03 kernel: [2406007.163910] LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 612s
Nov 13 14:21:23 ascratch-oss03 kernel: [2406007.204091] LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:23 ascratch-oss03 kernel: LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 612s
Nov 13 14:21:23 ascratch-oss03 kernel: LustreError: 1821798:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:33 ascratch-oss03  [2406017.243125] LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 622s
Nov 13 14:21:33 ascratch-oss03  [2406017.283349] LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:33 ascratch-oss03 kernel: [2406017.243125] LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 622s
Nov 13 14:21:33 ascratch-oss03 kernel: [2406017.283349] LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:33 ascratch-oss03 kernel: LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 622s
Nov 13 14:21:33 ascratch-oss03 kernel: LustreError: 1822124:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:43 ascratch-oss03  [2406027.322850] LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 632s
Nov 13 14:21:43 ascratch-oss03  [2406027.362918] LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:43 ascratch-oss03 kernel: [2406027.322850] LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 632s
Nov 13 14:21:43 ascratch-oss03 kernel: [2406027.362918] LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 358.529999
Nov 13 14:21:43 ascratch-oss03 kernel: LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 632s
Nov 13 14:21:43 ascratch-oss03 kernel: LustreError: 1822738:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:45 ascratch-oss03  [2406028.841784] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000003d1178dc x1749011010828928/t0(0) o4->026c6a24-fbfc-44d8-a657-68e4824bdce0@172.22.16.4@o2ib20:315/0 lens 488/0 e 8 to 0 dl 1668345710 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1393165'
Nov 13 14:21:45 ascratch-oss03  [2406028.913481] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 426 previous similar messages
Nov 13 14:21:45 ascratch-oss03 kernel: [2406028.841784] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000003d1178dc x1749011010828928/t0(0) o4->026c6a24-fbfc-44d8-a657-68e4824bdce0@172.22.16.4@o2ib20:315/0 lens 488/0 e 8 to 0 dl 1668345710 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1393165'
Nov 13 14:21:45 ascratch-oss03 kernel: [2406028.913481] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 426 previous similar messages
Nov 13 14:21:45 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000003d1178dc x1749011010828928/t0(0) o4->026c6a24-fbfc-44d8-a657-68e4824bdce0@172.22.16.4@o2ib20:315/0 lens 488/0 e 8 to 0 dl 1668345710 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1393165'
Nov 13 14:21:45 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 426 previous similar messages
Nov 13 14:21:51 ascratch-oss03  [2406035.557313] Lustre: ascratch-OST0006: Client 026c6a24-fbfc-44d8-a657-68e4824bdce0 (at 172.22.16.4@o2ib20) reconnecting
Nov 13 14:21:51 ascratch-oss03  [2406035.582508] Lustre: Skipped 83 previous similar messages
Nov 13 14:21:51 ascratch-oss03 kernel: [2406035.557313] Lustre: ascratch-OST0006: Client 026c6a24-fbfc-44d8-a657-68e4824bdce0 (at 172.22.16.4@o2ib20) reconnecting
Nov 13 14:21:51 ascratch-oss03 kernel: [2406035.582508] Lustre: Skipped 83 previous similar messages
Nov 13 14:21:52 ascratch-oss03  [2406035.605051] LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 641s
Nov 13 14:21:52 ascratch-oss03  [2406035.645215] LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:52 ascratch-oss03 kernel: [2406035.605051] LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 641s
Nov 13 14:21:52 ascratch-oss03 kernel: [2406035.645215] LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:21:51 ascratch-oss03 kernel: Lustre: ascratch-OST0006: Client 026c6a24-fbfc-44d8-a657-68e4824bdce0 (at 172.22.16.4@o2ib20) reconnecting
Nov 13 14:21:51 ascratch-oss03 kernel: Lustre: Skipped 83 previous similar messages
Nov 13 14:21:52 ascratch-oss03 kernel: LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 641s
Nov 13 14:21:52 ascratch-oss03 kernel: LustreError: 231983:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 1 previous similar message
Nov 13 14:22:03 ascratch-oss03  [2406047.414372] LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 652s
Nov 13 14:22:03 ascratch-oss03  [2406047.454632] LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 2 previous similar messages
Nov 13 14:22:03 ascratch-oss03 kernel: [2406047.414372] LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 652s
Nov 13 14:22:03 ascratch-oss03 kernel: [2406047.454632] LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 2 previous similar messages
Nov 13 14:22:03 ascratch-oss03 kernel: LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 652s
Nov 13 14:22:03 ascratch-oss03 kernel: LustreError: 1823545:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 2 previous similar messages
Nov 13 14:22:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 361.579987
Nov 13 14:22:23 ascratch-oss03  [2406067.506118] LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 673s
Nov 13 14:22:23 ascratch-oss03  [2406067.546529] LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 3 previous similar messages
Nov 13 14:22:23 ascratch-oss03 kernel: [2406067.506118] LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 673s
Nov 13 14:22:23 ascratch-oss03 kernel: [2406067.546529] LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 3 previous similar messages
Nov 13 14:22:23 ascratch-oss03 kernel: LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 673s
Nov 13 14:22:23 ascratch-oss03 kernel: LustreError: 1824205:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 3 previous similar messages
Nov 13 14:22:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 370.929993
Nov 13 14:22:59 ascratch-oss03  [2406103.273595] Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004fcb6dcd x1749012679893632/t0(0) o4->b539b7d9-4f10-4444-a839-a0072fa9b539@172.22.17.92@o2ib20:389/0 lens 488/448 e 4 to 0 dl 1668345784 ref 2 fl Interpret:/0/0 rc 0/0 job:'1407256'
Nov 13 14:22:59 ascratch-oss03  [2406103.346521] Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:22:59 ascratch-oss03 kernel: [2406103.273595] Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004fcb6dcd x1749012679893632/t0(0) o4->b539b7d9-4f10-4444-a839-a0072fa9b539@172.22.17.92@o2ib20:389/0 lens 488/448 e 4 to 0 dl 1668345784 ref 2 fl Interpret:/0/0 rc 0/0 job:'1407256'
Nov 13 14:22:59 ascratch-oss03 kernel: [2406103.346521] Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:22:59 ascratch-oss03 kernel: Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004fcb6dcd x1749012679893632/t0(0) o4->b539b7d9-4f10-4444-a839-a0072fa9b539@172.22.17.92@o2ib20:389/0 lens 488/448 e 4 to 0 dl 1668345784 ref 2 fl Interpret:/0/0 rc 0/0 job:'1407256'
Nov 13 14:22:59 ascratch-oss03 kernel: Lustre: 337707:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:23:04 ascratch-oss03  [2406107.622894] LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 713s
Nov 13 14:23:04 ascratch-oss03  [2406107.663643] LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 8 previous similar messages
Nov 13 14:23:04 ascratch-oss03 kernel: [2406107.622894] LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 713s
Nov 13 14:23:04 ascratch-oss03 kernel: [2406107.663643] LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 8 previous similar messages
Nov 13 14:23:04 ascratch-oss03 kernel: LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 713s
Nov 13 14:23:04 ascratch-oss03 kernel: LustreError: 1825537:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 8 previous similar messages
Nov 13 14:23:06 ascratch-oss03  [2406109.641144] Lustre: ascratch-OST0007: Client b539b7d9-4f10-4444-a839-a0072fa9b539 (at 172.22.17.92@o2ib20) reconnecting
Nov 13 14:23:06 ascratch-oss03  [2406109.666743] Lustre: Skipped 18 previous similar messages
Nov 13 14:23:06 ascratch-oss03 kernel: [2406109.641144] Lustre: ascratch-OST0007: Client b539b7d9-4f10-4444-a839-a0072fa9b539 (at 172.22.17.92@o2ib20) reconnecting
Nov 13 14:23:06 ascratch-oss03 kernel: [2406109.666743] Lustre: Skipped 18 previous similar messages
Nov 13 14:23:06 ascratch-oss03 kernel: Lustre: ascratch-OST0007: Client b539b7d9-4f10-4444-a839-a0072fa9b539 (at 172.22.17.92@o2ib20) reconnecting
Nov 13 14:23:06 ascratch-oss03 kernel: Lustre: Skipped 18 previous similar messages
Nov 13 14:23:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 389.149994
Nov 13 14:23:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 401.540009
Nov 13 14:24:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 409.149994
Nov 13 14:24:14 ascratch-oss03  [2406177.775438] LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 783s
Nov 13 14:24:14 ascratch-oss03  [2406177.816408] LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 14 previous similar messages
Nov 13 14:24:14 ascratch-oss03 kernel: [2406177.775438] LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 783s
Nov 13 14:24:14 ascratch-oss03 kernel: [2406177.816408] LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 14 previous similar messages
Nov 13 14:24:14 ascratch-oss03 kernel: LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 783s
Nov 13 14:24:14 ascratch-oss03 kernel: LustreError: 1827861:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 14 previous similar messages
Nov 13 14:24:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 413.820007
Nov 13 14:24:57 ascratch-oss03  [2406221.033375] ptlrpc_watchdog_fire: 273 callbacks suppressed
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.033375] ptlrpc_watchdog_fire: 273 callbacks suppressed
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.033378] Lustre: ll_ost_io00_172: service thread pid 520271 was inactive for 585.135 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:24:57 ascratch-oss03  [2406221.033378] Lustre: ll_ost_io00_172: service thread pid 520271 was inactive for 585.135 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:24:57 ascratch-oss03  [2406221.102475] Pid: 520271, comm: ll_ost_io00_172 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.102475] Pid: 520271, comm: ll_ost_io00_172 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.127375] Call Trace TBD:
Nov 13 14:24:57 ascratch-oss03  [2406221.127375] Call Trace TBD:
Nov 13 14:24:57 ascratch-oss03  [2406221.145195] [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.145195] [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:24:57 ascratch-oss03  [2406221.164269] [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.164269] [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.183209] [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:24:57 ascratch-oss03  [2406221.183209] [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:24:57 ascratch-oss03  [2406221.202842] [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.202842] [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.221631] [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:24:57 ascratch-oss03  [2406221.221631] [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:24:57 ascratch-oss03  [2406221.239923] [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.239923] [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.258411] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03  [2406221.258411] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03  [2406221.277255] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.277255] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.296616] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03  [2406221.296616] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: ptlrpc_watchdog_fire: 273 callbacks suppressed
Nov 13 14:24:57 ascratch-oss03 kernel: Lustre: ll_ost_io00_172: service thread pid 520271 was inactive for 585.135 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:24:57 ascratch-oss03 kernel: Pid: 520271, comm: ll_ost_io00_172 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:24:57 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] dbuf_destroy+0x1cb/0x3f0 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] dbuf_evict_one+0xfa/0x120 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] dbuf_rele_and_unlock+0x626/0x660 [zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] osd_bufs_put+0x114/0x1f0 [osd_zfs]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] ofd_commitrw+0x2ad/0xd80 [ofd]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] tgt_brw_read+0x1243/0x1fa0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:24:57 ascratch-oss03  [2406221.314333] [<0>] kthread+0x116/0x130
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.314333] [<0>] kthread+0x116/0x130
Nov 13 14:24:57 ascratch-oss03 kernel: [2406221.330765] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:24:57 ascratch-oss03  [2406221.330765] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:24:57 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:25:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 416.799988
Nov 13 14:25:15 ascratch-oss03  [2406239.363316] Lustre: ascratch-OST0008: Client cdd299a6-a6ce-4eaf-818e-6ca7369a3f9a (at 172.22.19.16@o2ib20) reconnecting
Nov 13 14:25:15 ascratch-oss03  [2406239.386781] Lustre: Skipped 15 previous similar messages
Nov 13 14:25:15 ascratch-oss03 kernel: [2406239.363316] Lustre: ascratch-OST0008: Client cdd299a6-a6ce-4eaf-818e-6ca7369a3f9a (at 172.22.19.16@o2ib20) reconnecting
Nov 13 14:25:15 ascratch-oss03 kernel: [2406239.386781] Lustre: Skipped 15 previous similar messages
Nov 13 14:25:15 ascratch-oss03 kernel: Lustre: ascratch-OST0008: Client cdd299a6-a6ce-4eaf-818e-6ca7369a3f9a (at 172.22.19.16@o2ib20) reconnecting
Nov 13 14:25:15 ascratch-oss03 kernel: Lustre: Skipped 15 previous similar messages
Nov 13 14:25:27 ascratch-oss03  [2406250.729332] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004ee7b2fb x1749011182548096/t0(0) o4->1e0cb311-492e-4d21-857e-9d284ae111d0@172.22.17.213@o2ib20:537/0 lens 504/0 e 2 to 0 dl 1668345932 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1406518'
Nov 13 14:25:27 ascratch-oss03  [2406250.796145] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:25:27 ascratch-oss03 kernel: [2406250.729332] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004ee7b2fb x1749011182548096/t0(0) o4->1e0cb311-492e-4d21-857e-9d284ae111d0@172.22.17.213@o2ib20:537/0 lens 504/0 e 2 to 0 dl 1668345932 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1406518'
Nov 13 14:25:27 ascratch-oss03 kernel: [2406250.796145] Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:25:27 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@000000004ee7b2fb x1749011182548096/t0(0) o4->1e0cb311-492e-4d21-857e-9d284ae111d0@172.22.17.213@o2ib20:537/0 lens 504/0 e 2 to 0 dl 1668345932 ref 2 fl New:/0/ffffffff rc 0/-1 job:'1406518'
Nov 13 14:25:27 ascratch-oss03 kernel: Lustre: 520252:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages
Nov 13 14:25:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 418.910004
Nov 13 14:26:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 420.549988
Nov 13 14:26:13 ascratch-oss03 pacemaker-execd[232299]: warning: lustre-OST0006_stop_0 process (PID 1821471) timed out
Nov 13 14:26:13 ascratch-oss03 pacemaker-execd[232299]: warning: lustre-OST0006_stop_0[1821471] timed out after 300000ms
Nov 13 14:26:13 ascratch-oss03 pacemaker-controld[232302]: error: Result of stop operation for lustre-OST0006 on ascratch-oss03: Timed Out
Nov 13 14:26:13 ascratch-oss03 pacemaker-attrd[232300]: notice: Setting fail-count-lustre-OST0006#stop_0[ascratch-oss03]: (unset) -> INFINITY
Nov 13 14:26:13 ascratch-oss03 pacemaker-attrd[232300]: notice: Setting last-failure-lustre-OST0006#stop_0[ascratch-oss03]: (unset) -> 1668345973
Nov 13 14:26:13 ascratch-oss03 pacemaker-controld[232302]: notice: Requesting local execution of stop operation for lustre-OST0007 on ascratch-oss03
Nov 13 14:26:14 ascratch-oss03 Lustre(lustre-OST0007)[1831754]: INFO: Starting to unmount ost07/ost07
Nov 13 14:26:14 ascratch-oss03 systemd[1]: mnt-lustre-local-ascratch\x2dOST0007.mount: Succeeded.
Nov 13 14:26:24 ascratch-oss03  [2406308.002441] LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 913s
Nov 13 14:26:24 ascratch-oss03  [2406308.039220] LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 27 previous similar messages
Nov 13 14:26:24 ascratch-oss03 kernel: [2406308.002441] LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 913s
Nov 13 14:26:24 ascratch-oss03 kernel: [2406308.039220] LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 27 previous similar messages
Nov 13 14:26:24 ascratch-oss03 kernel: LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 913s
Nov 13 14:26:24 ascratch-oss03 kernel: LustreError: 1832091:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 27 previous similar messages
Nov 13 14:26:35 ascratch-oss03  [2406319.337205] Lustre: ll_ost_io00_116: service thread pid 519654 was inactive for 652.714 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:26:35 ascratch-oss03  [2406319.382037] Pid: 519654, comm: ll_ost_io00_116 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.337205] Lustre: ll_ost_io00_116: service thread pid 519654 was inactive for 652.714 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.382037] Pid: 519654, comm: ll_ost_io00_116 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.405184] Call Trace TBD:
Nov 13 14:26:35 ascratch-oss03  [2406319.405184] Call Trace TBD:
Nov 13 14:26:35 ascratch-oss03  [2406319.420644] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.420644] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:26:35 ascratch-oss03  [2406319.437656] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.437656] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.454583] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:26:35 ascratch-oss03  [2406319.454583] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:26:35 ascratch-oss03  [2406319.470813] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.470813] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.488634] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:26:35 ascratch-oss03  [2406319.488634] [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:26:35 ascratch-oss03  [2406319.506081] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.506081] [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.522630] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:26:35 ascratch-oss03  [2406319.522630] [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:26:35 ascratch-oss03  [2406319.539610] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.539610] [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.555140] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03  [2406319.555140] [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03  [2406319.570956] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.570956] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03 kernel: [2406319.587115] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03  [2406319.587115] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:26:36 ascratch-oss03  [2406319.603916] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:26:36 ascratch-oss03 kernel: [2406319.603916] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:26:36 ascratch-oss03 kernel: [2406319.619057] [<0>] kthread+0x116/0x130
Nov 13 14:26:36 ascratch-oss03  [2406319.619057] [<0>] kthread+0x116/0x130
Nov 13 14:26:36 ascratch-oss03  [2406319.632784] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:26:36 ascratch-oss03 kernel: [2406319.632784] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:26:35 ascratch-oss03 kernel: Lustre: ll_ost_io00_116: service thread pid 519654 was inactive for 652.714 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:26:35 ascratch-oss03 kernel: Pid: 519654, comm: ll_ost_io00_116 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:26:35 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] ofd_preprw+0x6e7/0x8f0 [ofd]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:26:35 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:26:36 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:26:36 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:26:36 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:26:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 421.459991
Nov 13 14:27:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 422.070007
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.873090] Lustre: ll_ost00_182: service thread pid 2191053 was inactive for 694.167 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.873091] Lustre: ll_ost_io00_180: service thread pid 520571 was inactive for 703.327 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:27:41 ascratch-oss03  [2406384.873090] Lustre: ll_ost00_182: service thread pid 2191053 was inactive for 694.167 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:27:41 ascratch-oss03  [2406384.873091] Lustre: ll_ost_io00_180: service thread pid 520571 was inactive for 703.327 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:27:41 ascratch-oss03  [2406384.947194] Pid: 2191053, comm: ll_ost00_182 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.947194] Pid: 2191053, comm: ll_ost00_182 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.967046] Call Trace TBD:
Nov 13 14:27:41 ascratch-oss03  [2406384.967046] Call Trace TBD:
Nov 13 14:27:41 ascratch-oss03  [2406384.979666] [<0>] cv_wait_common+0xaf/0x130 [spl]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.979666] [<0>] cv_wait_common+0xaf/0x130 [spl]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406384.994372] [<0>] txg_wait_synced_impl+0xc2/0x110 [zfs]
Nov 13 14:27:41 ascratch-oss03  [2406384.994372] [<0>] txg_wait_synced_impl+0xc2/0x110 [zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.009096] [<0>] txg_wait_synced+0xc/0x40 [zfs]
Nov 13 14:27:41 ascratch-oss03  [2406385.009096] [<0>] txg_wait_synced+0xc/0x40 [zfs]
Nov 13 14:27:41 ascratch-oss03  [2406385.023272] [<0>] osd_object_sync+0x2af/0x2c0 [osd_zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.023272] [<0>] osd_object_sync+0x2af/0x2c0 [osd_zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.037837] [<0>] tgt_sync+0x15d/0x260 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03  [2406385.037837] [<0>] tgt_sync+0x15d/0x260 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03  [2406385.051671] [<0>] ofd_sync_hdl+0x159/0x590 [ofd]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.051671] [<0>] ofd_sync_hdl+0x159/0x590 [ofd]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.065613] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03  [2406385.065613] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03  [2406385.080429] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.080429] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03  [2406385.096135] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.096135] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.110379] [<0>] kthread+0x116/0x130
Nov 13 14:27:41 ascratch-oss03  [2406385.110379] [<0>] kthread+0x116/0x130
Nov 13 14:27:41 ascratch-oss03  [2406385.123648] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:27:41 ascratch-oss03 kernel: [2406385.123648] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:27:41 ascratch-oss03 kernel: Lustre: ll_ost00_182: service thread pid 2191053 was inactive for 694.167 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:27:41 ascratch-oss03 kernel: Lustre: ll_ost_io00_180: service thread pid 520571 was inactive for 703.327 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:27:41 ascratch-oss03 kernel: Pid: 2191053, comm: ll_ost00_182 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:27:41 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] cv_wait_common+0xaf/0x130 [spl]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] txg_wait_synced_impl+0xc2/0x110 [zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] txg_wait_synced+0xc/0x40 [zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] osd_object_sync+0x2af/0x2c0 [osd_zfs]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] tgt_sync+0x15d/0x260 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] ofd_sync_hdl+0x159/0x590 [ofd]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:27:41 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:27:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 422.869995
Nov 13 14:28:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.269989
Nov 13 14:28:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.160004
Nov 13 14:29:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.100006
Nov 13 14:29:34 ascratch-oss03  [2406497.805776] Lustre: ascratch-OST0008: Client 79dde588-b0ab-42eb-bea9-8d02d4874207 (at 172.22.18.22@o2ib20) reconnecting
Nov 13 14:29:34 ascratch-oss03  [2406497.826194] Lustre: Skipped 21 previous similar messages
Nov 13 14:29:34 ascratch-oss03 kernel: [2406497.805776] Lustre: ascratch-OST0008: Client 79dde588-b0ab-42eb-bea9-8d02d4874207 (at 172.22.18.22@o2ib20) reconnecting
Nov 13 14:29:34 ascratch-oss03 kernel: [2406497.826194] Lustre: Skipped 21 previous similar messages
Nov 13 14:29:34 ascratch-oss03 kernel: Lustre: ascratch-OST0008: Client 79dde588-b0ab-42eb-bea9-8d02d4874207 (at 172.22.18.22@o2ib20) reconnecting
Nov 13 14:29:34 ascratch-oss03 kernel: Lustre: Skipped 21 previous similar messages
Nov 13 14:29:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.059998
Nov 13 14:29:52 ascratch-oss03  [2406515.944860] Lustre: ll_ost00_076: service thread pid 619753 was inactive for 777.290 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:29:52 ascratch-oss03  [2406515.979625] Lustre: Skipped 2 previous similar messages
Nov 13 14:29:52 ascratch-oss03 kernel: [2406515.944860] Lustre: ll_ost00_076: service thread pid 619753 was inactive for 777.290 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:29:52 ascratch-oss03 kernel: [2406515.979625] Lustre: Skipped 2 previous similar messages
Nov 13 14:29:52 ascratch-oss03 kernel: Lustre: ll_ost00_076: service thread pid 619753 was inactive for 777.290 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Nov 13 14:29:52 ascratch-oss03 kernel: Lustre: Skipped 2 previous similar messages
Nov 13 14:30:09 ascratch-oss03  [2406532.776841] Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000eadd7b67 x1749023493800960/t0(0) o103->422269ad-40ab-4f2c-af24-7727c5b14bac@172.22.16.73@o2ib20:64/0 lens 440/224 e 23 to 0 dl 1668346214 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:30:09 ascratch-oss03  [2406532.834482] Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 59 previous similar messages
Nov 13 14:30:09 ascratch-oss03 kernel: [2406532.776841] Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000eadd7b67 x1749023493800960/t0(0) o103->422269ad-40ab-4f2c-af24-7727c5b14bac@172.22.16.73@o2ib20:64/0 lens 440/224 e 23 to 0 dl 1668346214 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:30:09 ascratch-oss03 kernel: [2406532.834482] Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 59 previous similar messages
Nov 13 14:30:09 ascratch-oss03 kernel: Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply  req@00000000eadd7b67 x1749023493800960/t0(0) o103->422269ad-40ab-4f2c-af24-7727c5b14bac@172.22.16.73@o2ib20:64/0 lens 440/224 e 23 to 0 dl 1668346214 ref 2 fl Interpret:/0/0 rc 0/0 job:''
Nov 13 14:30:09 ascratch-oss03 kernel: Lustre: 557426:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 59 previous similar messages
Nov 13 14:30:09 ascratch-oss03 systemd[1]: Starting system activity accounting tool...
Nov 13 14:30:09 ascratch-oss03 systemd[1]: sysstat-collect.service: Succeeded.
Nov 13 14:30:09 ascratch-oss03 systemd[1]: Started system activity accounting tool.
Nov 13 14:30:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.029999
Nov 13 14:30:25 ascratch-oss03  [2406548.712804] ptlrpc_watchdog_fire: 4 callbacks suppressed
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.712804] ptlrpc_watchdog_fire: 4 callbacks suppressed
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.712808] Lustre: ldlm_cn00_052: service thread pid 557444 was inactive for 610.576 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:30:25 ascratch-oss03  [2406548.712808] Lustre: ldlm_cn00_052: service thread pid 557444 was inactive for 610.576 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:30:25 ascratch-oss03  [2406548.766575] Pid: 557444, comm: ldlm_cn00_052 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.766575] Pid: 557444, comm: ldlm_cn00_052 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.786333] Call Trace TBD:
Nov 13 14:30:25 ascratch-oss03  [2406548.786333] Call Trace TBD:
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.798809] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.798809] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.812810] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.812810] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.826796] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.826796] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.840203] [<0>] dmu_spill_hold_by_dnode+0x2e/0x100 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.840203] [<0>] dmu_spill_hold_by_dnode+0x2e/0x100 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.854990] [<0>] dmu_spill_hold_existing+0x67/0xf0 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.854990] [<0>] dmu_spill_hold_existing+0x67/0xf0 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.869709] [<0>] sa_get_spill+0x30/0x70 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.869709] [<0>] sa_get_spill+0x30/0x70 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.883482] [<0>] sa_attr_op+0x8c/0x3b0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.883482] [<0>] sa_attr_op+0x8c/0x3b0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.897135] [<0>] sa_lookup_locked+0x64/0xb0 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.897135] [<0>] sa_lookup_locked+0x64/0xb0 [zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.911551] [<0>] sa_lookup+0x4f/0x80 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.911551] [<0>] sa_lookup+0x4f/0x80 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.924937] [<0>] osd_object_init0+0x126/0x1040 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.924937] [<0>] osd_object_init0+0x126/0x1040 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03  [2406548.939584] [<0>] osd_object_init+0x18e/0xef0 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.939584] [<0>] osd_object_init+0x18e/0xef0 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.954059] [<0>] lu_object_start.isra.41+0x6f/0x100 [obdclass]
Nov 13 14:30:25 ascratch-oss03  [2406548.954059] [<0>] lu_object_start.isra.41+0x6f/0x100 [obdclass]
Nov 13 14:30:25 ascratch-oss03  [2406548.969260] [<0>] lu_object_find_at+0x3d8/0x930 [obdclass]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.969260] [<0>] lu_object_find_at+0x3d8/0x930 [obdclass]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.984006] [<0>] ofd_object_find+0x30/0xf0 [ofd]
Nov 13 14:30:25 ascratch-oss03 kernel: ptlrpc_watchdog_fire: 4 callbacks suppressed
Nov 13 14:30:25 ascratch-oss03 kernel: Lustre: ldlm_cn00_052: service thread pid 557444 was inactive for 610.576 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:30:25 ascratch-oss03 kernel: Pid: 557444, comm: ldlm_cn00_052 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:30:25 ascratch-oss03 kernel: Call Trace TBD:
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] dmu_spill_hold_by_dnode+0x2e/0x100 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] dmu_spill_hold_existing+0x67/0xf0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] sa_get_spill+0x30/0x70 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] sa_attr_op+0x8c/0x3b0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] sa_lookup_locked+0x64/0xb0 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] sa_lookup+0x4f/0x80 [zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] osd_object_init0+0x126/0x1040 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] osd_object_init+0x18e/0xef0 [osd_zfs]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] lu_object_start.isra.41+0x6f/0x100 [obdclass]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] lu_object_find_at+0x3d8/0x930 [obdclass]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ofd_object_find+0x30/0xf0 [ofd]
Nov 13 14:30:25 ascratch-oss03  [2406548.984006] [<0>] ofd_object_find+0x30/0xf0 [ofd]
Nov 13 14:30:25 ascratch-oss03  [2406548.997908] [<0>] ofd_lvbo_update+0x62d/0xdd0 [ofd]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406548.997908] [<0>] ofd_lvbo_update+0x62d/0xdd0 [ofd]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.012273] [<0>] ldlm_request_cancel+0x4c9/0x780 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.012273] [<0>] ldlm_request_cancel+0x4c9/0x780 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.027058] [<0>] ldlm_handle_cancel+0x243/0x2b0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.027058] [<0>] ldlm_handle_cancel+0x243/0x2b0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.041558] [<0>] ldlm_cancel_handler+0x5f8/0x600 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.041558] [<0>] ldlm_cancel_handler+0x5f8/0x600 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.055916] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.055916] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.070878] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.070878] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03  [2406549.084223] [<0>] kthread+0x116/0x130
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.084223] [<0>] kthread+0x116/0x130
Nov 13 14:30:25 ascratch-oss03 kernel: [2406549.096099] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:30:25 ascratch-oss03  [2406549.096099] [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ofd_lvbo_update+0x62d/0xdd0 [ofd]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ldlm_request_cancel+0x4c9/0x780 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ldlm_handle_cancel+0x243/0x2b0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ldlm_cancel_handler+0x5f8/0x600 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] kthread+0x116/0x130
Nov 13 14:30:25 ascratch-oss03 kernel: [<0>] ret_from_fork+0x1f/0x40
Nov 13 14:30:43 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.019989
Nov 13 14:30:44 ascratch-oss03  [2406568.376393] LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 1173s
Nov 13 14:30:44 ascratch-oss03  [2406568.404750] LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 55 previous similar messages
Nov 13 14:30:44 ascratch-oss03 kernel: [2406568.376393] LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 1173s
Nov 13 14:30:44 ascratch-oss03 kernel: [2406568.404750] LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 55 previous similar messages
Nov 13 14:30:44 ascratch-oss03 kernel: LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 1173s
Nov 13 14:30:44 ascratch-oss03 kernel: LustreError: 1840482:0:(service.c:3644:ptlrpc_svcpt_health_check()) Skipped 55 previous similar messages
Nov 13 14:31:13 ascratch-oss03 pacemaker-controld[232302]: notice: High CPU load detected: 423.010010
Nov 13 14:31:13 ascratch-oss03 pacemaker-execd[232299]: warning: lustre-OST0007_stop_0 process (PID 1831754) timed out
Nov 13 14:31:13 ascratch-oss03 pacemaker-execd[232299]: warning: lustre-OST0007_stop_0[1831754] timed out after 300000ms
Nov 13 14:31:13 ascratch-oss03 pacemaker-controld[232302]: error: Result of stop operation for lustre-OST0007 on ascratch-oss03: Timed Out
Nov 13 14:31:14 ascratch-oss03 pacemaker-attrd[232300]: notice: Setting fail-count-lustre-OST0007#stop_0[ascratch-oss03]: (unset) -> INFINITY
Nov 13 14:31:14 ascratch-oss03 pacemaker-attrd[232300]: notice: Setting last-failure-lustre-OST0007#stop_0[ascratch-oss03]: (unset) -> 1668346273
Nov 13 14:31:14 ascratch-oss03 pacemaker-controld[232302]: notice: Requesting local execution of stop operation for lustre-OST0008 on ascratch-oss03
Nov 13 14:31:14 ascratch-oss03 Lustre(lustre-OST0008)[1841438]: INFO: Starting to unmount ost08/ost08
Nov 13 14:31:14 ascratch-oss03 systemd[1]: mnt-lustre-local-ascratch\x2dOST0008.mount: Succeeded.
Nov 13 14:31:30 ascratch-oss03  [2406614.248693] Lustre: ll_ost_io00_139: service thread pid 519840 was inactive for 856.219 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:31:30 ascratch-oss03  [2406614.286174] Pid: 519840, comm: ll_ost_io00_139 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.248693] Lustre: ll_ost_io00_139: service thread pid 519840 was inactive for 856.219 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.286174] Pid: 519840, comm: ll_ost_io00_139 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.305659] Call Trace TBD:
Nov 13 14:31:30 ascratch-oss03  [2406614.305659] Call Trace TBD:
Nov 13 14:31:30 ascratch-oss03  [2406614.317967] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.317967] [<0>] dbuf_find+0x58/0x1c0 [zfs]
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.331693] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:31:30 ascratch-oss03  [2406614.331693] [<0>] dbuf_hold_impl+0x61/0x610 [zfs]
Nov 13 14:31:30 ascratch-oss03  [2406614.346075] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.346075] [<0>] dbuf_hold+0x2c/0x60 [zfs]
Nov 13 14:31:30 ascratch-oss03 kernel: [2406614.359631] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:31:30 ascratch-oss03  [2406614.359631] [<0>] dmu_buf_hold_array_by_dnode+0xda/0x4a0 [zfs]
Nov 13 14:31:30 ascratch-oss03 kernel: Lustre: ll_ost_io00_139: service thread pid 519840 was inactive for 856.219 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: