Oct 16 01:01:01 ascratch-oss06 anacron[2541001]: Anacron started on 2022-10-16
Oct 16 01:01:01 ascratch-oss06 anacron[2541001]: Normal exit (0 jobs run)
Oct 16 01:01:01 ascratch-oss06 run-parts[2540929]: (/etc/cron.hourly) finished 0anacron
Oct 16 01:10:15 ascratch-oss06 systemd[1]: Starting system activity accounting tool...
Oct 16 01:10:15 ascratch-oss06 systemd[1]: sysstat-collect.service: Succeeded.
Oct 16 01:10:15 ascratch-oss06 systemd[1]: Started system activity accounting tool.
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.558792] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.581250] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) 
obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_i
ntel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) s
es ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_
sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.581284]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.801299] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.842610] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.863700] RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.882661] Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48
 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.929798] RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.951666] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.972682] RDX: 0000000178f0cb80 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:10:18 ascratch-oss06 kernel: [10058095.993457] RBP: 0000000178f0cb81 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.014356] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.034672] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.054798] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.076061] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.094477] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.114587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.134245] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.153654] PKRU: 55555554
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.168388] Call Trace:
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.182829]  ? common_interrupt+0xa/0xf
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.198556]  ? common_interrupt+0xa/0xf
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.214072]  ? dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.229841]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.245193]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.261975]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.278355]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.293762]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.309812]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.324571]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.339931]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.354831]  ? __kmalloc+0x102/0x240
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.368497]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.383705]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.398015]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.413026]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.427628]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.442309]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.456793]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.470728]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.485223]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.499613]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.512622]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.525754]  kthread+0x116/0x130
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.536794]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:10:18 ascratch-oss06 kernel: [10058096.548847]  ret_from_fork+0x1f/0x40
Oct 16 01:10:18 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:10:18 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:10:18 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:10:18 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOE    --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:10:18 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:10:18 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:10:18 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:10:18 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:10:18 ascratch-oss06 kernel: RDX: 0000000178f0cb80 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:10:18 ascratch-oss06 kernel: RBP: 0000000178f0cb81 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:10:18 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:10:18 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:10:18 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:10:18 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:10:18 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:10:18 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:10:18 ascratch-oss06 kernel: Call Trace:
Oct 16 01:10:18 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:10:18 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:10:18 ascratch-oss06 kernel: ? dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:10:18 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:10:18 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:10:18 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:10:18 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
ct 16 01:10:18 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:10:18 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:10:18 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:10:18 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.558975] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.574432] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.574469]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.735230] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.764393] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.779882] RIP: 0010:dbuf_prefetch+0x91/0x5a0 [zfs]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.793365] Code: e2 89 f9 89 7c 24 10 0f af cd 48 d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 <48> 81 c4 d8 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 89 d6 e8
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.829348] RSP: 0018:ffffae45e786f790 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.845716] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.861717] RDX: 0000000308de26fe RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.877835] RBP: 0000000308de26ff R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.893813] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.910118] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.926227] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.943555] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.958513] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.974995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:10:46 ascratch-oss06 kernel: [10058123.991437] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.008148] PKRU: 55555554
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.020096] Call Trace:
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.031716]  ? apic_timer_interrupt+0xa/0x20
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.045556]  ? common_interrupt+0xa/0xf
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.058580]  ? apic_timer_interrupt+0xa/0x20
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.072114]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.085310]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.100024]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.114492]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.128244]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.142647]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.155856]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.169592]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.183345]  ? __kmalloc+0x102/0x240
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.196029]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:10:46 ascratch-oss06 kernel: [10058124.210783]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:10:46 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:10:46 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:10:46 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.126018] rcu: INFO: rcu_sched self-detected stall on CPU
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.138946] rcu: #01123-....: (58173 ticks this GP) idle=a92/1/0x4000000000000002 softirq=463125053/463125053 fqs=12804 
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.157038] #011(t=60032 jiffies g=1827571989 q=259054)
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.169365] NMI backtrace for cpu 23
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.180104] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.207434] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.221657] Call Trace:
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.231248]  <IRQ>
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.240418]  dump_stack+0x5c/0x80
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.250591]  nmi_cpu_backtrace.cold.6+0x13/0x4e
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.262013]  ? lapic_can_unplug_cpu.cold.29+0x37/0x37
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.274174]  nmi_trigger_cpumask_backtrace+0xde/0xe0
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.286048]  rcu_dump_cpu_stacks+0xae/0xdc
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.297002]  rcu_sched_clock_irq.cold.90+0x10a/0x372
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.308977]  ? trigger_load_balance+0x5c/0x220
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.320122]  ? tick_sched_do_timer+0x50/0x50
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.331006]  update_process_times+0x24/0x60
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.342024]  tick_sched_handle+0x22/0x60
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.352601]  tick_sched_timer+0x37/0x70
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.362957]  __hrtimer_run_queues+0x100/0x280
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.373919]  hrtimer_interrupt+0x100/0x220
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.384583]  smp_apic_timer_interrupt+0x6a/0x130
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.395802]  apic_timer_interrupt+0xf/0x20
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.406697]  </IRQ>
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.415418] RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.426994] Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.459509] RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.474384] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.488731] RDX: 000000035de3f22d RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.503101] RBP: 000000035de3f22e R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.517446] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.531829] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.546272]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.557544]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.570357]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.582904]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.594770]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.607529]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.618854]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.630634]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.642647]  ? __kmalloc+0x102/0x240
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.653400]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.665921]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.677723]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.690163]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.702700]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.715177]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.727641]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.739540]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.752147]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.764716]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.775932]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.787436]  kthread+0x116/0x130
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.797023]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:10:53 ascratch-oss06 kernel: [10058130.808142]  ret_from_fork+0x1f/0x40
Oct 16 01:10:53 ascratch-oss06 kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Oct 16 01:10:53 ascratch-oss06 kernel: rcu: #01123-....: (58173 ticks this GP) idle=a92/1/0x4000000000000002 softirq=463125053/463125053 fqs=12804 
Oct 16 01:10:53 ascratch-oss06 kernel: #011(t=60032 jiffies g=1827571989 q=259054)
Oct 16 01:10:53 ascratch-oss06 kernel: NMI backtrace for cpu 23
Oct 16 01:10:53 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:10:53 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:10:53 ascratch-oss06 kernel: Call Trace:
Oct 16 01:10:53 ascratch-oss06 kernel: <IRQ>
Oct 16 01:10:53 ascratch-oss06 kernel: dump_stack+0x5c/0x80
Oct 16 01:10:53 ascratch-oss06 kernel: nmi_cpu_backtrace.cold.6+0x13/0x4e
Oct 16 01:10:53 ascratch-oss06 kernel: ? lapic_can_unplug_cpu.cold.29+0x37/0x37
Oct 16 01:10:53 ascratch-oss06 kernel: nmi_trigger_cpumask_backtrace+0xde/0xe0
Oct 16 01:10:53 ascratch-oss06 kernel: rcu_dump_cpu_stacks+0xae/0xdc
Oct 16 01:10:53 ascratch-oss06 kernel: rcu_sched_clock_irq.cold.90+0x10a/0x372
Oct 16 01:10:53 ascratch-oss06 kernel: ? trigger_load_balance+0x5c/0x220
Oct 16 01:10:53 ascratch-oss06 kernel: ? tick_sched_do_timer+0x50/0x50
Oct 16 01:10:53 ascratch-oss06 kernel: update_process_times+0x24/0x60
Oct 16 01:10:53 ascratch-oss06 kernel: tick_sched_handle+0x22/0x60
Oct 16 01:10:53 ascratch-oss06 kernel: tick_sched_timer+0x37/0x70
Oct 16 01:10:53 ascratch-oss06 kernel: __hrtimer_run_queues+0x100/0x280
Oct 16 01:10:53 ascratch-oss06 kernel: hrtimer_interrupt+0x100/0x220
Oct 16 01:10:53 ascratch-oss06 kernel: smp_apic_timer_interrupt+0x6a/0x130
Oct 16 01:10:53 ascratch-oss06 kernel: apic_timer_interrupt+0xf/0x20
Oct 16 01:10:53 ascratch-oss06 kernel: </IRQ>
Oct 16 01:10:53 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:10:53 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:10:53 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:10:53 ascratch-oss06 kernel: RDX: 000000035de3f22d RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:10:53 ascratch-oss06 kernel: RBP: 000000035de3f22e R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:10:53 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:10:53 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:10:53 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:10:53 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:10:53 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:10:53 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:10:53 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:10:53 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:10:53 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:10:53 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:10:53 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.559186] watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [ll_ost_io00_042:3875873]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.573976] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.574008]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.734500] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.764716] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.780768] RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.794841] Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.832260] RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.849278] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.865787] RDX: 00000004cc4d839d RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.882188] RBP: 00000004cc4d839e R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.898492] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.914810] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.931181] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.948612] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.963671] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.980138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: [10058155.996585] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.013010] PKRU: 55555554
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.024951] Call Trace:
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.036856]  ? common_interrupt+0xa/0xf
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.049874]  ? common_interrupt+0xa/0xf
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.062807]  ? apic_timer_interrupt+0xa/0x20
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.076288]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.089395]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.104094]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.118468]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.132127]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.146404]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.159524]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.173412]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.187069]  ? __kmalloc+0x102/0x240
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.199668]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.214013]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.227648]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.241926]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.256267]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.270750]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.285232]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.299201]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.313714]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.328128]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.341172]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.354325]  kthread+0x116/0x130
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.365385]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:11:18 ascratch-oss06 kernel: [10058156.377468]  ret_from_fork+0x1f/0x40
Oct 16 01:11:18 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [ll_ost_io00_042:3875873]
Oct 16 01:11:18 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:11:18 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:11:18 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:11:18 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:11:18 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:11:18 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:11:18 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:11:18 ascratch-oss06 kernel: RDX: 00000004cc4d839d RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:11:18 ascratch-oss06 kernel: RBP: 00000004cc4d839e R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:11:18 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:11:18 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:11:18 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:11:18 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:11:18 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:11:18 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:11:18 ascratch-oss06 kernel: Call Trace:
Oct 16 01:11:18 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:11:18 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:11:18 ascratch-oss06 kernel: ? apic_timer_interrupt+0xa/0x20
Oct 16 01:11:18 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:11:18 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:11:18 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:11:18 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:11:18 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:11:18 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:11:18 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:11:18 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:11:18 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.559371] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.574853] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.574887]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.736181] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.765344] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.780831] RIP: 0010:dmu_zfetch+0x4a5/0x5d0 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.794137] Code: 02 00 00 00 31 f6 e8 5a ea fd ff 4c 89 f2 4c 39 f3 75 de 4c 39 e5 7d 25 49 8b 7d 50 48 89 ea 41 b8 20 00 00 00 b9 02 00 00 00 <be> 01 00 00 00 48 83 c5 01 e8 2d ea fd ff 49 39 ec 75 db 8b 74 24
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.830074] RSP: 0018:ffffae45e786f8a0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.846410] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.862395] RDX: 000000065eb6b0de RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.878450] RBP: 000000065eb6b0de R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.894395] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.910429] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.926510] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.943786] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.958745] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.975488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: [10058183.991928] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.008384] PKRU: 55555554
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.020314] Call Trace:
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.031971]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.046740]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.061289]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.075145]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.089624]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.102959]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.116732]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.130533]  ? __kmalloc+0x102/0x240
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.143292]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.157850]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.172111]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.186604]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.201209]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.215709]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.230343]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.244553]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.259500]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.274575]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.288230]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.302675]  kthread+0x116/0x130
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.314777]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:11:46 ascratch-oss06 kernel: [10058184.327915]  ret_from_fork+0x1f/0x40
Oct 16 01:11:46 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:11:46 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:11:46 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:11:46 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:11:46 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:11:46 ascratch-oss06 kernel: RIP: 0010:dmu_zfetch+0x4a5/0x5d0 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: Code: 02 00 00 00 31 f6 e8 5a ea fd ff 4c 89 f2 4c 39 f3 75 de 4c 39 e5 7d 25 49 8b 7d 50 48 89 ea 41 b8 20 00 00 00 b9 02 00 00 00 <be> 01 00 00 00 48 83 c5 01 e8 2d ea fd ff 49 39 ec 75 db 8b 74 24
Oct 16 01:11:46 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f8a0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
Oct 16 01:11:46 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:11:46 ascratch-oss06 kernel: RDX: 000000065eb6b0de RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:11:46 ascratch-oss06 kernel: RBP: 000000065eb6b0de R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:11:46 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:11:46 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:11:46 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:11:46 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:11:46 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:11:46 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:11:46 ascratch-oss06 kernel: Call Trace:
Oct 16 01:11:46 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:11:46 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:11:46 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:11:46 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:11:46 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:11:46 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:11:46 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:11:46 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:11:46 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.559556] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.576080] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.576116]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.743398] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.773694] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.789444] RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.803424] Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.840190] RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.856976] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.873682] RDX: 00000007f1d4bb8f RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.889978] RBP: 00000007f1d4bb90 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.906231] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.922404] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.938661] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.956031] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.971257] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:12:14 ascratch-oss06 kernel: [10058211.987769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.004256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.020708] PKRU: 55555554
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.032734] Call Trace:
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.044420]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.057665]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.072749]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.087294]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.101179]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.115649]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.128920]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.142679]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.156464]  ? __kmalloc+0x102/0x240
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.169586]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.184184]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.198049]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.212523]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.227076]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.241572]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.256206]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.270463]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.285426]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.300537]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.314099]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.328079]  kthread+0x116/0x130
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.340003]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:12:14 ascratch-oss06 kernel: [10058212.352911]  ret_from_fork+0x1f/0x40
Oct 16 01:12:14 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:12:14 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:12:14 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:12:14 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:12:14 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:12:14 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:12:14 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:12:14 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:12:14 ascratch-oss06 kernel: RDX: 00000007f1d4bb8f RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:12:14 ascratch-oss06 kernel: RBP: 00000007f1d4bb90 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:12:14 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:12:14 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:12:14 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:12:14 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:12:14 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:12:14 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:12:14 ascratch-oss06 kernel: Call Trace:
Oct 16 01:12:14 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:12:14 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:12:14 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:12:14 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:12:14 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:12:14 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:12:14 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:12:14 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:12:14 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.559740] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.576043] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.576104]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.742104] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.771874] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.787515] RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.801215] Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.837829] RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.854546] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.870837] RDX: 000000098468acf3 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.887024] RBP: 000000098468acf4 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.903255] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.919405] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.935682] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.952927] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.967935] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:12:42 ascratch-oss06 kernel: [10058239.984410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.001124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.017550] PKRU: 55555554
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.029500] Call Trace:
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.041120]  ? common_interrupt+0xa/0xf
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.054128]  ? common_interrupt+0xa/0xf
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.067150]  ? apic_timer_interrupt+0xa/0x20
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.080550]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.093635]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.108245]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.122605]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.136289]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.150546]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.163667]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.177298]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.190945]  ? __kmalloc+0x102/0x240
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.203831]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.218174]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.231843]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.246087]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.260425]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.274735]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.289206]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.303176]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.317674]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.332115]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.344876]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.358008]  kthread+0x116/0x130
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.369094]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:12:42 ascratch-oss06 kernel: [10058240.381162]  ret_from_fork+0x1f/0x40
Oct 16 01:12:42 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:12:42 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:12:42 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:12:42 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:12:42 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:12:42 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x19/0x5a0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: Code: 00 00 e9 b0 fe ff ff e8 85 bf f2 f6 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 81 ec d8 00 00 00 89 0c 24 <44> 89 44 24 0c 65 48 8b 04 25 28 00 00 00 48 89 84 24 d0 00 00 00
Oct 16 01:12:42 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f790 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
Oct 16 01:12:42 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:12:42 ascratch-oss06 kernel: RDX: 000000098468acf3 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:12:42 ascratch-oss06 kernel: RBP: 000000098468acf4 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:12:42 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:12:42 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:12:42 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:12:42 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:12:42 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:12:42 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:12:42 ascratch-oss06 kernel: Call Trace:
Oct 16 01:12:42 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:12:42 ascratch-oss06 kernel: ? common_interrupt+0xa/0xf
Oct 16 01:12:42 ascratch-oss06 kernel: ? apic_timer_interrupt+0xa/0x20
Oct 16 01:12:42 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:12:42 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:12:42 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:12:42 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:12:42 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:12:42 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:12:42 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:12:42 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:12:42 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.559924] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.575405] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.575437]  dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.736209] CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.765683] Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.781171] RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.794655] Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.830636] RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.846989] RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.863182] RDX: 0000000b16b95626 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.879240] RBP: 0000000b16b95627 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.895240] R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.911311] R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.927419] FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.944717] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.959698] CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.976183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: [10058267.992644] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.009079] PKRU: 55555554
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.021037] Call Trace:
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.032756]  dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.046000]  dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.060767]  dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.075306]  osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.089151]  ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.103887]  ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.117149]  tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.131196]  ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.144964]  ? __kmalloc+0x102/0x240
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.157745]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.172328]  ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.186181]  ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.200914]  ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.215444]  ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.229978]  ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.244599]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.258798]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.273744]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.288811]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.302616]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.316584]  kthread+0x116/0x130
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.328478]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:10 ascratch-oss06 kernel: [10058268.341386]  ret_from_fork+0x1f/0x40
Oct 16 01:13:10 ascratch-oss06 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ll_ost_io00_042:3875873]
Oct 16 01:13:10 ascratch-oss06 kernel: Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) netconsole libcfs(OE) rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_umad(OE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass
rapl intel_cstate ipmi_si ipmi_devintf intel_uncore ipmi_msghandler wdat_wdt pcspkr zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) ahci icp(POE) libahci zcommon(POE) mei_me znvpair(POE) ses ioatdma spl(OE) libata enclosure mei joydev i2c_i801 lpc_ich dca acpi_power_meter acpi_cpufreq dm_service_time sd_mod t10_pi sg crct10dif_pclmul mpt3sas(OE) crc32_pclmul raid_class crc32c_intel i40e megaraid_sas ghash_clmulni_intel scsi_transport_sas ib_ipoib(OE) ib_cm(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw tls pci_hyperv_intf dm_multipath sunrpc
Oct 16 01:13:10 ascratch-oss06 kernel: dm_mirror dm_region_hash dm_log dm_mod
Oct 16 01:13:10 ascratch-oss06 kernel: CPU: 23 PID: 3875873 Comm: ll_ost_io00_042 Kdump: loaded Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:10 ascratch-oss06 kernel: Hardware name: Huawei 2488H V5/BC62MBHA0, BIOS 7.79 10/20/2020
Oct 16 01:13:10 ascratch-oss06 kernel: RIP: 0010:dbuf_prefetch+0x9c/0x5a0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: Code: d3 e2 48 39 50 10 73 48 48 8b 84 24 d0 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 fd 04 00 00 48 81 c4 d8 00 00 00 5b 5d 41 5c <41> 5d 41 5e 41 5f c3 48 89 d6 e8 25 4a 02 00 48 85 c0 75 ca 48 8b
Oct 16 01:13:10 ascratch-oss06 kernel: RSP: 0018:ffffae45e786f880 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
Oct 16 01:13:10 ascratch-oss06 kernel: RAX: 0000000000000000 RBX: 00000000ffffe048 RCX: 0000000000000002
Oct 16 01:13:10 ascratch-oss06 kernel: RDX: 0000000b16b95626 RSI: 0000000000000001 RDI: ffff905cfca8b9a8
Oct 16 01:13:10 ascratch-oss06 kernel: RBP: 0000000b16b95627 R08: 0000000000000020 R09: 0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: R10: 000000000000009f R11: ffffffffffffe089 R12: 003ffffffffffff9
Oct 16 01:13:10 ascratch-oss06 kernel: R13: ffff905cfca8bd28 R14: ffff905cfca8bd48 R15: ffff9055452b2a58
Oct 16 01:13:10 ascratch-oss06 kernel: FS:  0000000000000000(0000) GS:ffff907ec11c0000(0000) knlGS:0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 01:13:10 ascratch-oss06 kernel: CR2: 00007fe68d517e5c CR3: 00000039e1e10003 CR4: 00000000007706e0
Oct 16 01:13:10 ascratch-oss06 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 01:13:10 ascratch-oss06 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 16 01:13:10 ascratch-oss06 kernel: PKRU: 55555554
Oct 16 01:13:10 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:10 ascratch-oss06 kernel: dmu_zfetch+0x4b3/0x5d0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: dmu_buf_hold_array_by_dnode+0x393/0x4a0 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: dmu_buf_hold_array_by_bonus+0x56/0x70 [zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: osd_bufs_get+0x1e4/0x1080 [osd_zfs]
Oct 16 01:13:10 ascratch-oss06 kernel: ofd_preprw_read.isra.27+0x3d7/0x13a0 [ofd]
Oct 16 01:13:10 ascratch-oss06 kernel: ofd_preprw+0x6e7/0x8f0 [ofd]
Oct 16 01:13:10 ascratch-oss06 kernel: tgt_brw_read+0x6dc/0x1fa0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? null_alloc_rs+0x96/0x360 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? __kmalloc+0x102/0x240
Oct 16 01:13:10 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:13:10 ascratch-oss06 kernel: ? null_alloc_rs+0xb3/0x360 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? sptlrpc_svc_alloc_rs+0x63/0x350 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? lustre_msg_add_version+0x21/0xa0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? lustre_pack_reply_v2+0x1e7/0x280 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:10 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:10 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:10 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:17 ascratch-oss06 kernel: [10058275.553984] Lustre: ll_ost_io00_016: service thread pid 3787551 was inactive for 200.429 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Oct 16 01:13:17 ascratch-oss06 kernel: [10058275.553986] Lustre: ll_ost_io00_035: service thread pid 3797010 was inactive for 200.429 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Oct 16 01:13:17 ascratch-oss06 kernel: [10058275.553987] Lustre: ll_ost_io00_057: service thread pid 3025791 was inactive for 200.429 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:17 ascratch-oss06 kernel: [10058275.553989] Lustre: ll_ost_io00_002: service thread pid 249965 was inactive for 200.429 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:17 ascratch-oss06 kernel: [10058275.553991] Pid: 3025791, comm: ll_ost_io00_057 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.708174] Call Trace TBD:
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.719591] [<0>] dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.733083] [<0>] dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.746462] [<0>] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.760365] [<0>] osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.774676] [<0>] ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.788157] [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.801320] [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.814334] [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.827490] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:17 ascratch-oss06 kernel: Lustre: ll_ost_io00_016: service thread pid 3787551 was inactive for 200.429 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Oct 16 01:13:17 ascratch-oss06 kernel: Lustre: ll_ost_io00_035: service thread pid 3797010 was inactive for 200.429 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Oct 16 01:13:17 ascratch-oss06 kernel: Lustre: ll_ost_io00_057: service thread pid 3025791 was inactive for 200.429 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:17 ascratch-oss06 kernel: Lustre: ll_ost_io00_002: service thread pid 249965 was inactive for 200.429 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:17 ascratch-oss06 kernel: Pid: 3025791, comm: ll_ost_io00_057 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:18 ascratch-oss06 kernel: Call Trace TBD:
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.840950] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.855188] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.868021] [<0>] kthread+0x116/0x130
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.879615] [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.891448] Pid: 249965, comm: ll_ost_io00_002 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.909640] Call Trace TBD:
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.920709] [<0>] dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.933975] [<0>] dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.947282] [<0>] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.961177] [<0>] osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.975416] [<0>] ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058275.988877] [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.001950] [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.015038] [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.028632] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.042325] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.056840] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.069892] [<0>] kthread+0x116/0x130
Oct 16 01:13:18 ascratch-oss06 kernel: [10058276.081665] [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] kthread+0x116/0x130
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:18 ascratch-oss06 kernel: Pid: 249965, comm: ll_ost_io00_002 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:18 ascratch-oss06 kernel: Call Trace TBD:
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] kthread+0x116/0x130
Oct 16 01:13:18 ascratch-oss06 kernel: [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:21 ascratch-oss06 kernel: [10058279.650010] Lustre: ll_ost_io00_074: service thread pid 210619 was inactive for 200.908 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:21 ascratch-oss06 kernel: [10058279.686255] Pid: 210619, comm: ll_ost_io00_074 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:21 ascratch-oss06 kernel: [10058279.705081] Call Trace TBD:
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.716652] [<0>] cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.730250] [<0>] txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.744309] [<0>] txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.757669] [<0>] osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.771562] [<0>] ofd_write_attr_set+0x49d/0x1070 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.785533] [<0>] ofd_commitrw_write+0x222/0x1ad0 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.799557] [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.812506] [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.825421] [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.838373] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.851480] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.865440] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:21 ascratch-oss06 kernel: Lustre: ll_ost_io00_074: service thread pid 210619 was inactive for 200.908 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Oct 16 01:13:21 ascratch-oss06 kernel: Pid: 210619, comm: ll_ost_io00_074 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Oct 16 01:13:21 ascratch-oss06 kernel: Call Trace TBD:
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ofd_write_attr_set+0x49d/0x1070 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ofd_commitrw_write+0x222/0x1ad0 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] kthread+0x116/0x130
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.877495] [<0>] kthread+0x116/0x130
Oct 16 01:13:22 ascratch-oss06 kernel: [10058279.888067] [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:22 ascratch-oss06 kernel: [<0>] ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.746201] INFO: task dp_sync_taskq:247545 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.760128]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.775209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.790001] task:dp_sync_taskq   state:D stack:    0 pid:247545 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.805526] Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.814983]  __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.825535]  ? dbuf_sync_leaf+0x46b/0x690 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.837107]  schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.847311]  rwsem_down_write_slowpath+0x32a/0x5e0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.859237]  ? dmu_objset_userquota_get_ids+0x2cf/0x480 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.872175]  dnode_sync+0x4eb/0xa30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.883298]  ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.894128]  sync_dnodes_task+0x47/0x120 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.905652]  taskq_thread+0x2ed/0x530 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.916945]  ? wake_up_q+0x80/0x80
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.927273]  ? taskq_thread_spawn+0x50/0x50 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.938781]  kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.948764]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.960026]  ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.970401] INFO: task txg_sync:249338 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.984119]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: [10058283.999656] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: INFO: task dp_sync_taskq:247545 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: task:dp_sync_taskq   state:D stack:    0 pid:247545 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: ? dbuf_sync_leaf+0x46b/0x690 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: rwsem_down_write_slowpath+0x32a/0x5e0
Oct 16 01:13:26 ascratch-oss06 kernel: ? dmu_objset_userquota_get_ids+0x2cf/0x480 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dnode_sync+0x4eb/0xa30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: sync_dnodes_task+0x47/0x120 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: taskq_thread+0x2ed/0x530 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: ? wake_up_q+0x80/0x80
Oct 16 01:13:26 ascratch-oss06 kernel: ? taskq_thread_spawn+0x50/0x50 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: INFO: task txg_sync:249338 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: task:txg_sync        state:D stack:    0 pid:249338 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.014686] task:txg_sync        state:D stack:    0 pid:249338 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.030542] Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.040279]  __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.051053]  schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.061596]  taskq_wait+0x7a/0xc0 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.072619]  ? finish_wait+0x80/0x80
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.083386]  dmu_objset_sync+0x33e/0x550 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.095274]  ? dnode_setdirty+0x8a/0xe0 [zfs]
ct 16 01:13:26 ascratch-oss06 kernel: [10058284.106717]  dsl_dataset_sync+0x6d/0x270 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.117980]  dsl_pool_sync+0xe8/0x510 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.128998]  spa_sync+0x56d/0xfc0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.139223]  ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.149364]  ? spa_txg_history_init_io+0xfe/0x110 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.161166]  txg_sync_thread+0x29f/0x480 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.172082]  ? txg_thread_exit.isra.11+0x60/0x60 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.183685]  ? __thread_exit+0x20/0x20 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.194665]  thread_generic_wrapper+0x6f/0x80 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.206074]  kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.215827]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.226878]  ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.237070] INFO: task ll_ost_io00_002:249965 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.251206]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.266320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.281148] task:ll_ost_io00_002 state:D stack:    0 pid:249965 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.296888] Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.306390]  __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.316848]  schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.326839]  rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.338388]  ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.348876]  dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: taskq_wait+0x7a/0xc0 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: ? finish_wait+0x80/0x80
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_objset_sync+0x33e/0x550 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ? dnode_setdirty+0x8a/0xe0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dsl_dataset_sync+0x6d/0x270 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dsl_pool_sync+0xe8/0x510 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: spa_sync+0x56d/0xfc0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: ? spa_txg_history_init_io+0xfe/0x110 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: txg_sync_thread+0x29f/0x480 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ? txg_thread_exit.isra.11+0x60/0x60 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ? __thread_exit+0x20/0x20 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Oct 16 01:13:26 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: INFO: task ll_ost_io00_002:249965 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: task:ll_ost_io00_002 state:D stack:    0 pid:249965 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:26 ascratch-oss06 kernel: ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.359965]  dmu_tx_count_write+0xe5/0x1a0 [zfs]
ct 16 01:13:26 ascratch-oss06 kernel: [10058284.371135]  dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.382932]  osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.395271]  ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.406628]  ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.417604]  ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.428940]  obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.439978]  tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.451201]  ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.461913]  ? flush_work+0x42/0x1c0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.471958]  ? internal_add_timer+0x42/0x60
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.482606]  ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.492783]  ? mutex_lock+0xe/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.502641]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.514181]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.526409]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.538651]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.549376]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.560580]  kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.569855]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.580403]  ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.590100] INFO: task ll_ost_io00_016:3787551 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.603900]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.618625] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.633186] task:ll_ost_io00_016 state:D stack:    0 pid:3787551 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:26 ascratch-oss06 kernel: ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:26 ascratch-oss06 kernel: ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:26 ascratch-oss06 kernel: ? flush_work+0x42/0x1c0
Oct 16 01:13:26 ascratch-oss06 kernel: ? internal_add_timer+0x42/0x60
Oct 16 01:13:26 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: ? mutex_lock+0xe/0x30
Oct 16 01:13:26 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:26 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:26 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:26 ascratch-oss06 kernel: INFO: task ll_ost_io00_016:3787551 blocked for more than 120 seconds.
Oct 16 01:13:26 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:26 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:26 ascratch-oss06 kernel: task:ll_ost_io00_016 state:D stack:    0 pid:3787551 ppid:     2 flags:0x80004000
Oct 16 01:13:26 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.648436] Call Trace:
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.657568]  __schedule+0x2bd/0x760
ct 16 01:13:26 ascratch-oss06 kernel: [10058284.667743]  schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.677549]  rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.689032]  ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: [10058284.699759]  dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.711008]  dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.722290]  dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.734174]  osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.746405]  ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.757845]  ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.768677]  ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.779952]  obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.790974]  tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.802124]  ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.812625]  ? flush_work+0x42/0x1c0
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.822356]  ? internal_add_timer+0x42/0x60
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.832645]  ? _cond_resched+0x15/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.842453]  ? mutex_lock+0xe/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.851942]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.863176]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.875130]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.887203]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.898074]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.909380]  kthread+0x116/0x130
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.918834]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:26 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:26 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:26 ascratch-oss06 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:26 ascratch-oss06 kernel: ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:26 ascratch-oss06 kernel: dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:27 ascratch-oss06 kernel: ? flush_work+0x42/0x1c0
Oct 16 01:13:27 ascratch-oss06 kernel: ? internal_add_timer+0x42/0x60
Oct 16 01:13:27 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: ? mutex_lock+0xe/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:27 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.929542]  ret_from_fork+0x1f/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.939360] INFO: task ll_ost_io01_023:3796315 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.953265]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.968071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.982548] task:ll_ost_io01_023 state:D stack:    0 pid:3796315 ppid:     2 flags:0x80004000
Oct 16 01:13:27 ascratch-oss06 kernel: [10058284.998016] Call Trace:
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.007192]  __schedule+0x2bd/0x760
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.017443]  schedule+0x37/0xa0
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.027315]  io_schedule+0x12/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.037448]  cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.048438]  ? finish_wait+0x80/0x80
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.058728]  txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.070221]  txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.081001]  osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.092288]  ofd_object_punch+0xdbb/0x1330 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.103505]  ? __req_capsule_get+0x551/0x770 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.115185]  ? lustre_swab_niobuf_remote+0x20/0x20 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.127400]  ofd_punch_hdl+0x4ac/0x710 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.138265]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.149837]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.162100]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.174305]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.185006]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: INFO: task ll_ost_io01_023:3796315 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:27 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:27 ascratch-oss06 kernel: task:ll_ost_io01_023 state:D stack:    0 pid:3796315 ppid:     2 flags:0x80004000
Oct 16 01:13:27 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:27 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:27 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:27 ascratch-oss06 kernel: io_schedule+0x12/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:27 ascratch-oss06 kernel: ? finish_wait+0x80/0x80
Oct 16 01:13:27 ascratch-oss06 kernel: txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_object_punch+0xdbb/0x1330 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: ? __req_capsule_get+0x551/0x770 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? lustre_swab_niobuf_remote+0x20/0x20 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_punch_hdl+0x4ac/0x710 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.196581]  kthread+0x116/0x130
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.206101]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.216870]  ret_from_fork+0x1f/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.226737] INFO: task ll_ost_io00_035:3797010 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.240689]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.255591] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.270172] task:ll_ost_io00_035 state:D stack:    0 pid:3797010 ppid:     2 flags:0x80004000
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.285582] Call Trace:
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.294992]  __schedule+0x2bd/0x760
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.305305]  schedule+0x37/0xa0
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.315224]  rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.326802]  ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.337466]  dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.348823]  dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.360251]  dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.372302]  osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.384745]  ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.396646]  ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.407703]  ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.419162]  obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.430658]  tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.442136]  ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.453059]  ? flush_work+0x42/0x1c0
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.463372]  ? internal_add_timer+0x42/0x60
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.474251]  ? _cond_resched+0x15/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:27 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: INFO: task ll_ost_io00_035:3797010 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:27 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:27 ascratch-oss06 kernel: task:ll_ost_io00_035 state:D stack:    0 pid:3797010 ppid:     2 flags:0x80004000
Oct 16 01:13:27 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:27 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:27 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:27 ascratch-oss06 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:27 ascratch-oss06 kernel: ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:27 ascratch-oss06 kernel: ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:27 ascratch-oss06 kernel: ? flush_work+0x42/0x1c0
Oct 16 01:13:27 ascratch-oss06 kernel: ? internal_add_timer+0x42/0x60
Oct 16 01:13:27 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: ? mutex_lock+0xe/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.484671]  ? mutex_lock+0xe/0x30
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.494834]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.506428]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.518650]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.530885]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.541600]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.552950]  kthread+0x116/0x130
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.562498]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.573307]  ret_from_fork+0x1f/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: [10058285.583241] INFO: task ll_ost00_029:3802518 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel: INFO: task ll_ost00_029:3802518 blocked for more than 120 seconds.
Oct 16 01:13:27 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:27 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:27 ascratch-oss06 kernel: task:ll_ost00_029    state:D stack:    0 pid:3802518 ppid:     2 flags:0x80004000
Oct 16 01:13:27 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:27 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:27 ascratch-oss06 kernel: ? nvt_lookup_name_type.isra.53+0x72/0xb0 [znvpair]
Oct 16 01:13:27 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:27 ascratch-oss06 kernel: io_schedule+0x12/0x40
Oct 16 01:13:27 ascratch-oss06 kernel: cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:27 ascratch-oss06 kernel: ? finish_wait+0x80/0x80
Oct 16 01:13:27 ascratch-oss06 kernel: txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: osd_object_sync+0x2af/0x2c0 [osd_zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_sync+0x15d/0x260 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_sync_hdl+0x159/0x590 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.772949]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.784651]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.797193]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.809535]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.820350]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.831732]  kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.841239]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.852019]  ret_from_fork+0x1f/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.862009] INFO: task ll_ost_io00_057:3025791 blocked for more than 120 seconds.
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.875969]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.890841] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.905418] task:ll_ost_io00_057 state:D stack:    0 pid:3025791 ppid:     2 flags:0x80004000
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.920838] Call Trace:
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.930306]  __schedule+0x2bd/0x760
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.940609]  schedule+0x37/0xa0
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.950527]  rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.962140]  ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.972806]  dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.984169]  dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058285.995782]  dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.007813]  osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.020228]  ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.032073]  ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.043144]  ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.054639]  obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: INFO: task ll_ost_io00_057:3025791 blocked for more than 120 seconds.
Oct 16 01:13:28 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:28 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:28 ascratch-oss06 kernel: task:ll_ost_io00_057 state:D stack:    0 pid:3025791 ppid:     2 flags:0x80004000
Oct 16 01:13:28 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:28 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:28 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:28 ascratch-oss06 kernel: rwsem_down_read_slowpath+0x360/0x3d0
Oct 16 01:13:28 ascratch-oss06 kernel: ? zio_null+0x2b/0x30 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: dmu_tx_check_ioerr+0x1e/0xc0 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: dmu_tx_count_write+0xe5/0x1a0 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: osd_declare_write_commit+0x1da/0x7d0 [osd_zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_commitrw_write+0x65a/0x1ad0 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.065934]  tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.077416]  ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.088361]  ? flush_work+0x42/0x1c0
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.098881]  ? internal_add_timer+0x42/0x60
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.109757]  ? _cond_resched+0x15/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.120177]  ? mutex_lock+0xe/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.130273]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.141957]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.154292]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.166685]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.177485]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.188810]  kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.198508]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.209213]  ret_from_fork+0x1f/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.218983] INFO: task ll_ost_io00_074:210619 blocked for more than 120 seconds.
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.232957]       Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.247769] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.262321] task:ll_ost_io00_074 state:D stack:    0 pid:210619 ppid:     2 flags:0x80004000
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.277557] Call Trace:
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.286776]  __schedule+0x2bd/0x760
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.297261]  schedule+0x37/0xa0
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.307118]  io_schedule+0x12/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.317235]  cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.328437]  ? finish_wait+0x80/0x80
Oct 16 01:13:28 ascratch-oss06 kernel: ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:28 ascratch-oss06 kernel: ? flush_work+0x42/0x1c0
Oct 16 01:13:28 ascratch-oss06 kernel: ? internal_add_timer+0x42/0x60
Oct 16 01:13:28 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: ? mutex_lock+0xe/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: INFO: task ll_ost_io00_074:210619 blocked for more than 120 seconds.
Oct 16 01:13:28 ascratch-oss06 kernel:      Tainted: P          IOEL   --------- -  - 4.18.0-348.7.1.el8_5.x86_64 #1
Oct 16 01:13:28 ascratch-oss06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 16 01:13:28 ascratch-oss06 kernel: task:ll_ost_io00_074 state:D stack:    0 pid:210619 ppid:     2 flags:0x80004000
Oct 16 01:13:28 ascratch-oss06 kernel: Call Trace:
Oct 16 01:13:28 ascratch-oss06 kernel: __schedule+0x2bd/0x760
Oct 16 01:13:28 ascratch-oss06 kernel: schedule+0x37/0xa0
Oct 16 01:13:28 ascratch-oss06 kernel: io_schedule+0x12/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: cv_wait_common+0xaf/0x130 [spl]
Oct 16 01:13:28 ascratch-oss06 kernel: ? finish_wait+0x80/0x80
Oct 16 01:13:28 ascratch-oss06 kernel: txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.338703]  txg_wait_synced_impl+0xc2/0x110 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.350181]  txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.360979]  osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.372268]  ofd_write_attr_set+0x49d/0x1070 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.383614]  ofd_commitrw_write+0x222/0x1ad0 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.395178]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.407063]  ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.417779]  ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.428990]  obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.439745]  tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.450606]  ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.460869]  ? flush_work+0x42/0x1c0
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.470496]  ? internal_add_timer+0x42/0x60
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.480784]  ? _cond_resched+0x15/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.490596]  ? mutex_lock+0xe/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.500110]  tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.511275]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.523220]  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.535348]  ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.546002]  ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.557239]  kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.566677]  ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: [10058286.577329]  ret_from_fork+0x1f/0x40
Oct 16 01:13:28 ascratch-oss06 kernel: txg_wait_synced+0xc/0x40 [zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: osd_trans_stop+0x535/0x570 [osd_zfs]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_write_attr_set+0x49d/0x1070 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_commitrw_write+0x222/0x1ad0 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: ? lprocfs_counter_add+0xd2/0x140 [obdclass]
Oct 16 01:13:28 ascratch-oss06 kernel: ofd_commitrw+0x731/0xd80 [ofd]
Oct 16 01:13:28 ascratch-oss06 kernel: ? obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: obd_commitrw+0x1ac/0x370 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_brw_write+0x1575/0x1b10 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? enqueue_task_fair+0x95/0x580
Oct 16 01:13:28 ascratch-oss06 kernel: ? flush_work+0x42/0x1c0
Oct 16 01:13:28 ascratch-oss06 kernel: ? internal_add_timer+0x42/0x60
Oct 16 01:13:28 ascratch-oss06 kernel: ? _cond_resched+0x15/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: ? mutex_lock+0xe/0x30
Oct 16 01:13:28 ascratch-oss06 kernel: tgt_request_handle+0xc93/0x1a40 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ptlrpc_main+0xc06/0x1560 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: ? ptlrpc_wait_event+0x590/0x590 [ptlrpc]
Oct 16 01:13:28 ascratch-oss06 kernel: kthread+0x116/0x130
Oct 16 01:13:28 ascratch-oss06 kernel: ? kthread_flush_work_fn+0x10/0x10
Oct 16 01:13:28 ascratch-oss06 kernel: ret_from_fork+0x1f/0x40