Feb 27 03:54:36 mds1 kernel: Adding Red Hat flag eBPF/event.
Feb 27 07:44:04 mds1 kernel: connection5:0: detected conn error (1020)
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: re-opening session 5 (reopen_cnt 0)
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: connecting to 169.254.2.4:3260
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: connected local port 41850 to 169.254.2.4:3260
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: login response status 0000
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: login response status 0000
Feb 27 07:44:04 mds1 iscsid[4367]: iscsid: connection5:0 is operational after recovery (1 attempts)
Feb 27 12:58:29 mds1 kernel: mlx5_core 0000:01:00.0 enp1s0: Link up
Feb 27 12:58:29 mds1 kernel: IPv6: ADDRCONF(NETDEV_UP): enp1s0: link is not ready
Feb 27 12:58:30 mds1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0: link becomes ready
Feb 27 12:58:31 mds1 kernel: RPC: Registered named UNIX socket transport module.
Feb 27 12:58:31 mds1 kernel: RPC: Registered udp transport module.
Feb 27 12:58:31 mds1 kernel: RPC: Registered tcp transport module.
Feb 27 12:58:31 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Feb 27 12:58:31 mds1 kernel: libcfs: loading out-of-tree module taints kernel.
Feb 27 12:58:31 mds1 kernel: libcfs: module verification failed: signature and/or required key missing - tainting kernel
Feb 27 12:58:31 mds1 kernel: LNet: HW NUMA nodes: 1, HW CPU cores: 32, npartitions: 2
Feb 27 12:58:31 mds1 kernel: alg: No test for adler32 (adler32-zlib)
Feb 27 12:58:32 mds1 kernel: Key type ._llcrypt registered
Feb 27 12:58:32 mds1 kernel: Key type .llcrypt registered
Feb 27 12:58:32 mds1 kernel: LNet: Added LNI 10.0.152.229@tcp [256/2048/0/180]
Feb 27 12:58:32 mds1 kernel: LNet: Accept secure, port 988
Feb 27 12:58:32 mds1 kernel: Lustre: Lustre: Build Version: 2.15.5
Feb 27 12:59:39 mds1 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Feb 27 13:06:05 mds1 kernel: LNet: Added LNI 10.0.179.185@tcp [256/2048/0/180]
Feb 27 13:06:09 mds1 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Feb 27 13:06:12 mds1 kernel: LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Feb 27 13:06:13 mds1 kernel: LustreError: 137-5: lustrefs-MDT0000_UUID: not available for connect from 10.0.44.75@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Feb 27 13:06:13 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: No data found on store. Initialize space: rc = -61
Feb 27 13:06:14 mds1 kernel: Lustre: lustrefs-MDT0000: new disk, initializing
Feb 27 13:06:14 mds1 kernel: Lustre: lustrefs-MDT0000: Imperative Recovery not enabled, recovery window 300-900
Feb 27 13:06:14 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
Feb 27 13:06:14 mds1 kernel: SELinux: (dev lustre, type lustre) has no xattr support
Feb 27 13:06:14 mds1 kernel: SELinux: (dev lustre, type lustre) falling back to genfs
Feb 27 13:06:15 mds1 kernel: Lustre: 336743:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS  req@0000000038b751bc x1825215466180544/t0(0) o700->lustrefs-MDT000c-osp-MDT0000@10.0.116.131@tcp:30/10 lens 264/248 e 0 to 0 dl 1740661587 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:''
Feb 27 13:06:15 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:e:mdt
Feb 27 13:06:16 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:f:mdt
Feb 27 13:06:16 mds1 kernel: Lustre: Skipped 3 previous similar messages
Feb 27 13:06:18 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000480000400-0x00000004c0000400]:3:mdt
Feb 27 13:06:18 mds1 kernel: Lustre: Skipped 4 previous similar messages
Feb 27 13:06:30 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000600000400-0x0000000640000400]:25:ost
Feb 27 13:06:30 mds1 kernel: Lustre: Skipped 5 previous similar messages
Feb 27 13:06:39 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000000a40000400-0x0000000a80000400]:16a:ost
Feb 27 13:06:39 mds1 kernel: Lustre: Skipped 16 previous similar messages
Feb 27 13:06:55 mds1 kernel: Lustre: ctl-lustrefs-MDT0000: super-sequence allocation rc = 0 [0x0000002140000400-0x0000002180000400]:70b:ost
Feb 27 13:06:55 mds1 kernel: Lustre: Skipped 91 previous similar messages
Feb 27 13:07:07 mds1 kernel: Lustre: 337764:0:(osd_io.c:2114:osd_ldiskfs_write_record()) lustrefs-MDT0000/: adding bh without locking off 28320 (block 6, size 32, offs 28320)
Feb 27 13:07:15 mds1 kernel: Lustre: 332519:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740661628/real 1740661628]  req@000000000024dd01 x1825215466334848/t0(0) o41->lustrefs-MDT0001-osp-MDT0000@10.0.64.120@tcp:24/4 lens 224/368 e 0 to 1 dl 1740661635 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:''
Feb 27 13:07:15 mds1 kernel: Lustre: lustrefs-MDT0001-osp-MDT0000: Connection to lustrefs-MDT0001 (at 10.0.64.120@tcp) was lost; in progress operations using this service will wait for recovery to complete
[33763.860611] LDISKFS-fs error (device dm-10): ldiskfs_getblk:1014: inode #166: block 14072026: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 1994, credits 5/0, errcode -28
Feb 27 13:09:06 mds1-primary-vni[33763.863335] LDISKFS-fs (dm-10): Remounting filesystem read-only
c-924205 kernel:[33763.864085] LDISKFS-fs error (device dm-10) in osd_trans_stop:2104: error 28
 Lustre: ctl-lus[33763.865059] LDISKFS-fs error (device dm-10) in osd_trans_stop:2104: IO failure
trefs-MDT0000: super-sequence allocation rc = 0 [0x0000005800000400-0x0000005840000400]:33:ost
Feb 27 13:09:06 mds1 kernel: Lustre: Skipped 218 previous similar messages
Feb 27 13:09:06 mds1 kernel: Lustre: 339425:0:(osd_io.c:2114:osd_ldiskfs_write_record()) lustrefs-MDT0000/: adding bh without locking off 99200 (block 24, size 32, offs 99200)
Feb 27 13:09:06 mds1 kernel: WARNING: CPU: 25 PID: 339425 at fs/jbd2/transaction.c:1526 jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) sunrpc nft_reject_inet nf_reject_ipv6 nft_reject nft_ct nft_limit nft_counter ipt_REJECT nf_reject_ipv4 xt_AUDIT xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink xfs libcrc32c dm_queue_length overlay vfat fat drm_vram_helper intel_rapl_msr drm_ttm_helper intel_rapl_common ttm drm_kms_helper kvm_amd syscopyarea ccp sysfillrect mlx5_ib iTCO_wdt sysimgblt iTCO_vendor_support kvm ib_uverbs irqbypass drm ib_core mlx5_vdpa pcspkr joydev i2c_i801 vringh lpc_ich vhost_iotlb vdpa binfmt_misc nvme_tcp(X) nvme_fabrics nvme nvme_core ext4 mbcache jbd2 sd_mod t10_pi sg mlx5_core ahci libahci libata crc32_pclmul serio_raw virtio_scsi mlxfw psample pci_hyperv_intf dm_multipath dm_mirror dm_region_hash dm_log dm_mod
Feb 27 13:09:06 mds1 kernel: crypto_user ansi_cprng cmac ccm xts ecdh_generic dh_generic des3_ede_x86_64 des_generic ghash_clmulni_intel crct10dif_pclmul crc32c_intel sha3_generic be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi [last unloaded: kheaders]
Feb 27 13:09:06 mds1 kernel: Red Hat flags: eBPF/event
Feb 27 13:09:06 mds1 kernel: CPU: 25 PID: 339425 Comm: llog_process_th Tainted: G           OE  X  -------- -  - 4.18.0-553.5.1.el8_10_lustre.x86_64 #1
Feb 27 13:09:06 mds1 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.4 02/27/2023
Feb 27 13:09:06 mds1 kernel: RIP: 0010:jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: Code: 80 00 75 f4 e9 26 ff ff ff 41 bd 8b ff ff ff e9 32 fe ff ff 4c 8b 4e 70 4c 8d 73 02 4d 39 cc 0f 84 e1 fe ff ff e9 42 9c 00 00 <0f> 0b 41 bd e4 ff ff ff 4c 8d 73 02 e9 cb fe ff ff 0f 0b 66 0f 1f
Feb 27 13:09:06 mds1 kernel: RSP: 0018:ff564dfee130f850 EFLAGS: 00010246
Feb 27 13:09:06 mds1 kernel: RAX: 0000000000000001 RBX: ff4ed98784d40d68 RCX: 0000000000000000
Feb 27 13:09:06 mds1 kernel: RDX: 0000000000000007 RSI: ff4ed982a1b8d800 RDI: ff4ed98429f33ee0
Feb 27 13:09:06 mds1 kernel: RBP: ff4ed98784d41f00 R08: 0000000000000000 R09: ff4ed98340bae000
Feb 27 13:09:06 mds1 kernel: R10: 0000000000000000 R11: 0000000000000100 R12: ff4ed985412b1100
Feb 27 13:09:06 mds1 kernel: R13: 0000000000000000 R14: ffffffffc1a2fc50 R15: 00000000000003f6
Feb 27 13:09:06 mds1 kernel: FS:  0000000000000000(0000) GS:ff4ed99a9ba40000(0000) knlGS:0000000000000000
Feb 27 13:09:06 mds1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 27 13:09:06 mds1 kernel: CR2: 000055c5e7123540 CR3: 0000001589610003 CR4: 0000000000771ee0
Feb 27 13:09:06 mds1 kernel: PKRU: 55555554
Feb 27 13:09:06 mds1 kernel: Call Trace:
Feb 27 13:09:06 mds1 kernel: ? __warn+0x94/0xe0
Feb 27 13:09:06 mds1 kernel: ? jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: ? jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: ? report_bug+0xb1/0xe0
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: ? do_error_trap+0x9e/0xd0
Feb 27 13:09:06 mds1 kernel: ? do_invalid_op+0x36/0x40
Feb 27 13:09:06 mds1 kernel: ? jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: ? invalid_op+0x14/0x20
Feb 27 13:09:06 mds1 kernel: ? jbd2_journal_dirty_metadata+0x247/0x260 [jbd2]
Feb 27 13:09:06 mds1 kernel: __ldiskfs_handle_dirty_metadata+0x4f/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ldiskfs_getblk+0x112/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ldiskfs_bread+0x1f/0xc0 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: osd_ldiskfs_write_record+0x515/0x6c0 [osd_ldiskfs]
Feb 27 13:09:06 mds1 kernel: ? __irqentry_text_end+0x101463/0x101467
Feb 27 13:09:06 mds1 kernel: osd_write+0x12e/0x670 [osd_ldiskfs]
Feb 27 13:09:06 mds1 kernel: dt_record_write+0x32/0x110 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_osd_put_cat_list+0x79d/0x930 [obdclass]
Feb 27 13:09:06 mds1 kernel: osp_sync_llog_init+0x66f/0xb20 [osp]
Feb 27 13:09:06 mds1 kernel: ? osp_sync_init+0x262/0x770 [osp]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: osp_sync_init+0x262/0x770 [osp]
Feb 27 13:09:06 mds1 kernel: ? osp_init_precreate+0x35/0x2b0 [osp]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: osp_init0.isra.19+0x16ad/0x19f0 [osp]
Feb 27 13:09:06 mds1 kernel: osp_device_alloc+0xcb/0x180 [osp]
Feb 27 13:09:06 mds1 kernel: obd_setup+0x119/0x2e0 [obdclass]
Feb 27 13:09:06 mds1 kernel: class_setup+0x587/0x790 [obdclass]
Feb 27 13:09:06 mds1 kernel: class_process_config+0xfc8/0x2080 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? class_config_llog_handler+0x6b1/0x1250 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: ? __kmalloc+0x15f/0x2d0
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: class_config_llog_handler+0x846/0x1250 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_process_thread+0xf99/0x1a30 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: ? lu_context_init+0xa5/0x1b0 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? llog_backup+0x540/0x540 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_process_thread_daemonize+0x9b/0xe0 [obdclass]
Feb 27 13:09:06 mds1 kernel: kthread+0x134/0x150
Feb 27 13:09:06 mds1 kernel: ? set_kthread_struct+0x50/0x50
Feb 27 13:09:06 mds1 kernel: ret_from_fork+0x35/0x40
Feb 27 13:09:06 mds1 kernel: ---[ end trace 788d043ba0e0534b ]---
Feb 27 13:09:06 mds1 kernel: WARNING: CPU: 25 PID: 339425 at /tmp/rpmbuild-lustre-root-gEXrBD4w/BUILD/lustre-2.15.5/ldiskfs/ext4_jbd2.c:288 __ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) sunrpc nft_reject_inet nf_reject_ipv6 nft_reject nft_ct nft_limit nft_counter ipt_REJECT nf_reject_ipv4 xt_AUDIT xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink xfs libcrc32c dm_queue_length overlay vfat fat drm_vram_helper intel_rapl_msr drm_ttm_helper intel_rapl_common ttm drm_kms_helper kvm_amd syscopyarea ccp sysfillrect mlx5_ib iTCO_wdt sysimgblt iTCO_vendor_support kvm ib_uverbs irqbypass drm ib_core mlx5_vdpa pcspkr joydev i2c_i801 vringh lpc_ich vhost_iotlb vdpa binfmt_misc nvme_tcp(X) nvme_fabrics nvme nvme_core ext4 mbcache jbd2 sd_mod t10_pi sg mlx5_core ahci libahci libata crc32_pclmul serio_raw virtio_scsi mlxfw psample pci_hyperv_intf dm_multipath dm_mirror dm_region_hash dm_log dm_mod
Feb 27 13:09:06 mds1 kernel: crypto_user ansi_cprng cmac ccm xts ecdh_generic dh_generic des3_ede_x86_64 des_generic ghash_clmulni_intel crct10dif_pclmul crc32c_intel sha3_generic be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi fuse scsi_transport_iscsi [last unloaded: kheaders]
Feb 27 13:09:06 mds1 kernel: Red Hat flags: eBPF/event
Feb 27 13:09:06 mds1 kernel: CPU: 25 PID: 339425 Comm: llog_process_th Tainted: G        W  OE  X  -------- -  - 4.18.0-553.5.1.el8_10_lustre.x86_64 #1
Feb 27 13:09:06 mds1 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.4 02/27/2023
Feb 27 13:09:06 mds1 kernel: RIP: 0010:__ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: Code: 80 81 ff ff eb 93 f0 80 4b 01 80 e9 4f ff ff ff f0 80 4b 01 40 e9 39 ff ff ff 48 89 df 45 31 e4 e8 ff b4 f9 db e9 6f ff ff ff <0f> 0b 48 c7 c2 80 1c a3 c1 45 89 e0 48 89 e9 44 89 fe 4c 89 f7 e8
Feb 27 13:09:06 mds1 kernel: RSP: 0018:ff564dfee130f880 EFLAGS: 00010286
Feb 27 13:09:06 mds1 kernel: RAX: ff4ed982a1b8d800 RBX: ff4ed98784d40d68 RCX: 0000000000000000
Feb 27 13:09:06 mds1 kernel: RDX: 0000000000000007 RSI: ff4ed982a1b8d800 RDI: ff4ed98429f33ee0
Feb 27 13:09:06 mds1 kernel: RBP: ff4ed98429f33ee0 R08: 0000000000000000 R09: ff4ed98340bae000
Feb 27 13:09:06 mds1 kernel: R10: 0000000000000000 R11: 0000000000000100 R12: 00000000ffffffe4
Feb 27 13:09:06 mds1 kernel: R13: ff4ed9849a64d5f0 R14: ffffffffc1a2fc50 R15: 00000000000003f6
Feb 27 13:09:06 mds1 kernel: FS:  0000000000000000(0000) GS:ff4ed99a9ba40000(0000) knlGS:0000000000000000
Feb 27 13:09:06 mds1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 27 13:09:06 mds1 kernel: CR2: 000055c5e7123540 CR3: 0000001589610003 CR4: 0000000000771ee0
Feb 27 13:09:06 mds1 kernel: PKRU: 55555554
Feb 27 13:09:06 mds1 kernel: Call Trace:
Feb 27 13:09:06 mds1 kernel: ? __warn+0x94/0xe0
Feb 27 13:09:06 mds1 kernel: ? __ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ? __ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ? report_bug+0xb1/0xe0
Feb 27 13:09:06 mds1 kernel: ? do_error_trap+0x9e/0xd0
Feb 27 13:09:06 mds1 kernel: ? do_invalid_op+0x36/0x40
Feb 27 13:09:06 mds1 kernel: ? __ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ? invalid_op+0x14/0x20
Feb 27 13:09:06 mds1 kernel: ? __ldiskfs_handle_dirty_metadata+0x106/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ldiskfs_getblk+0x112/0x190 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: ldiskfs_bread+0x1f/0xc0 [ldiskfs]
Feb 27 13:09:06 mds1 kernel: osd_ldiskfs_write_record+0x515/0x6c0 [osd_ldiskfs]
Feb 27 13:09:06 mds1 kernel: ? __irqentry_text_end+0x101463/0x101467
Feb 27 13:09:06 mds1 kernel: osd_write+0x12e/0x670 [osd_ldiskfs]
Feb 27 13:09:06 mds1 kernel: dt_record_write+0x32/0x110 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_osd_put_cat_list+0x79d/0x930 [obdclass]
Feb 27 13:09:06 mds1 kernel: osp_sync_llog_init+0x66f/0xb20 [osp]
Feb 27 13:09:06 mds1 kernel: ? osp_sync_init+0x262/0x770 [osp]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: osp_sync_init+0x262/0x770 [osp]
Feb 27 13:09:06 mds1 kernel: ? osp_init_precreate+0x35/0x2b0 [osp]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: osp_init0.isra.19+0x16ad/0x19f0 [osp]
Feb 27 13:09:06 mds1 kernel: osp_device_alloc+0xcb/0x180 [osp]
Feb 27 13:09:06 mds1 kernel: obd_setup+0x119/0x2e0 [obdclass]
Feb 27 13:09:06 mds1 kernel: class_setup+0x587/0x790 [obdclass]
Feb 27 13:09:06 mds1 kernel: class_process_config+0xfc8/0x2080 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? class_config_llog_handler+0x6b1/0x1250 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: ? __kmalloc+0x15f/0x2d0
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: class_config_llog_handler+0x846/0x1250 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_process_thread+0xf99/0x1a30 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? srso_alias_return_thunk+0x5/0xfcdfd
Feb 27 13:09:06 mds1 kernel: ? lu_context_init+0xa5/0x1b0 [obdclass]
Feb 27 13:09:06 mds1 kernel: ? llog_backup+0x540/0x540 [obdclass]
Feb 27 13:09:06 mds1 kernel: llog_process_thread_daemonize+0x9b/0xe0 [obdclass]
Feb 27 13:09:06 mds1 kernel: kthread+0x134/0x150
Feb 27 13:09:06 mds1 kernel: ? set_kthread_struct+0x50/0x50
Feb 27 13:09:06 mds1 kernel: ret_from_fork+0x35/0x40
Feb 27 13:09:06 mds1 kernel: ---[ end trace 788d043ba0e0534c ]---
Feb 27 13:09:06 mds1 kernel: LDISKFS-fs: ldiskfs_getblk:1014: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
Feb 27 13:09:06 mds1 kernel: LDISKFS-fs error (device dm-10): ldiskfs_getblk:1014: inode #166: block 14072026: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 1994, credits 5/0, errcode -28
Feb 27 13:09:06 mds1 kernel: Aborting journal on device dm-10-8.
Feb 27 13:09:06 mds1 kernel: LDISKFS-fs (dm-10): Remounting filesystem read-only
Feb 27 13:09:06 mds1 kernel: LustreError: 339425:0:(osd_io.c:2148:osd_ldiskfs_write_record()) lustrefs-MDT0000/: error reading offset 99200 (block 24, size 32, offs 99200), credits 5/1: rc = -28
Feb 27 13:09:06 mds1 kernel: LDISKFS-fs error (device dm-10) in osd_trans_stop:2104: error 28
Feb 27 13:09:06 mds1 kernel: LustreError: 336540:0:(osd_handler.c:1796:osd_trans_commit_cb()) transaction @0x00000000e7a781cb commit error: 2
Feb 27 13:09:06 mds1 kernel: LDISKFS-fs error (device dm-10) in osd_trans_stop:2104: IO failure
Feb 27 13:09:06 mds1 kernel: LustreError: 339425:0:(osd_handler.c:2107:osd_trans_stop()) lustrefs-MDT0000: failed to stop transaction: rc = -28
Feb 27 13:09:06 mds1 kernel: LustreError: 339425:0:(osp_sync.c:1553:osp_sync_init()) lustrefs-OST0c1c-osc-MDT0000: can't initialize llog: rc = -28
Feb 27 13:09:06 mds1 kernel: LustreError: 339425:0:(obd_config.c:774:class_setup()) setup lustrefs-OST0c1c-osc-MDT0000 failed (-28)
Feb 27 13:09:06 mds1 kernel: LustreError: 339425:0:(obd_config.c:1999:class_config_llog_handler()) MGC10.0.104.241@tcp: cfg command failed: rc = -28
Feb 27 13:09:06 mds1 kernel: Lustre:    cmd=cf003 0:lustrefs-OST0c1c-osc-MDT0000  1:lustrefs-OST0c1c_UUID  2:10.0.11.105@tcp  
Feb 27 13:09:06 mds1 kernel: LustreError: 336635:0:(mgc_request.c:612:do_requeue()) failed processing log: -28
Feb 27 13:09:09 mds1 kernel: LustreError: 339543:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustrefs-OST04f1-osc-MDT0000: fail to cancel 1 llog-records: rc = -30
Feb 27 13:09:09 mds1 kernel: LustreError: 339543:0:(llog_cat.c:789:llog_cat_cancel_records()) lustrefs-OST04f1-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30
Feb 27 13:09:09 mds1 kernel: LustreError: 339544:0:(osp_precreate.c:1178:osp_init_pre_fid()) lustrefs-OST04f1-osc-MDT0000: write fid error: rc = -30
Feb 27 13:09:09 mds1 kernel: LustreError: 339544:0:(osp_precreate.c:1257:osp_precreate_thread()) lustrefs-OST04f1-osc-MDT0000: init pre fid error: rc = -30
[33769.145337] LustreError: 340094:0:(lod_lov.c:169:lod_add_device()) ASSERTION( obd->obd_lu_dev->ld_site == lod->lod_dt_dev.dd_lu_dev.ld_site ) failed: 
[33769.146853] LustreError: 340094:0:(lod_lov.c:169:lod_add_device()) LBUG
[33769.147827] Kernel panic - not syncing: LBUG
[33769.148258] CPU: 7 PID: 340094 Comm: llog_process_th Tainted: G        W  OE  X  -------- -  - 4.18.0-553.5.1.el8_10_lustre.x86_64 #1
[33769.149601] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.4 02/27/2023
[33769.150412] Call Trace:
[33769.150701]  dump_stack+0x41/0x60
[33769.151110]  panic+0xe7/0x2ac
[3F3eb7 267 913.:019:5121 4m7ds21-]pr im ar?y- vnriet_c-f92r42o05m k_erfneol:rk +Lu0stxre3Er5ro/r:0 3x440009
4:
0:(obd_conf[33769.152360]  lbug_with_loc.cold.8+0x18/0x18 [libcfs]
[33769.152865]  lod_add_device+0x1033/0x15f0 [lod]
[33769.153328]  ? srso_alias_return_thunk+0x5/0xfcdfd
[33769.153879]  ? simple_strntoull+0x8c/0xa0
[33769.154264]  lod_process_config+0x1111/0x12b0 [lod]
[33769.154734]  obd_process_config.constprop.37+0x107/0x1f0 [obdclass]
[33769.155363]  class_process_config+0x15c9/0x2080 [obdclass]
[33769.155913]  ? class_config_llog_handler+0x6b1/0x1250 [obdclass]
[33769.156525]  ? srso_alias_return_thunk+0x5/0xfcdfd
[33769.156982]  ? __kmalloc+0x15f/0x2d0
[33769.157318]  ? srso_alias_return_thunk+0x5/0xfcdfd
[33769.157773]  class_config_llog_handler+0x846/0x1250 [obdclass]
[33769.158349]  llog_process_thread+0xf99/0x1a30 [obdclass]
[33769.158871]  ? srso_alias_return_thunk+0x5/0xfcdfd
[33769.159322]  ? lu_context_init+0xa5/0x1b0 [obdclass]
[33769.159816]  ? llog_backup+0x540/0x540 [obdclass]
[33769.160277]  llog_process_thread_daemonize+0x9b/0xe0 [obdclass]
[33769.160862]  kthread+0x134/0x150
[33769.161186]  ? set_kthread_struct+0x50/0x50
[33769.161596]  ret_from_fork+0x35/0x40
[33769.162195] Kernel Offset: 0x1c600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[33769.163187] ---[ end Kernel panic - not syncing: LBUG ]---