[ 0.000000] microcode: microcode updated early to revision 0x71a, date = 2020-03-24 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1160.49.1.el7_lustre.x86_64 (jenkins@onyx-201-el7-x8664-3.onyx.whamcloud.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Sun Apr 3 16:20:30 UTC 2022 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.49.1.el7_lustre.x86_64 root=UUID=af55e6d6-cb1d-46c7-a478-fce49fa6f327 ro crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable [ 0.000000] BIOS-e820: [mem 0x000000000008f000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bb3c6fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bb3c7000-0x00000000bdd2efff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdf2f000-0x00000000bdfabfff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdfac000-0x00000000bdffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000cfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed19000-0x00000000fed19fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.6 present. [ 0.000000] DMI: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x840000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-DBFFF write-through [ 0.000000] DC000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask 3FFF80000000 write-back [ 0.000000] 1 base 000080000000 mask 3FFFC0000000 write-back [ 0.000000] 2 base 000100000000 mask 3FFF00000000 write-back [ 0.000000] 3 base 000200000000 mask 3FFE00000000 write-back [ 0.000000] 4 base 000400000000 mask 3FFC00000000 write-back [ 0.000000] 5 base 000800000000 mask 3FFFC0000000 write-back [ 0.000000] 6 base 0000FF800000 mask 3FFFFF800000 write-protect [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbe000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fcd70-0x000fcd7f] mapped at [ffffffffff200d70] [ 0.000000] Base memory trampoline at [ffff99e440089000] 89000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x272473000, 0x272473fff] PGTABLE [ 0.000000] BRK [0x272474000, 0x272474fff] PGTABLE [ 0.000000] BRK [0x272475000, 0x272475fff] PGTABLE [ 0.000000] BRK [0x272476000, 0x272476fff] PGTABLE [ 0.000000] BRK [0x272477000, 0x272477fff] PGTABLE [ 0.000000] BRK [0x272478000, 0x272478fff] PGTABLE [ 0.000000] BRK [0x272479000, 0x272479fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x3596b000-0x36cadfff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f0410 00024 (v02 INTEL ) [ 0.000000] ACPI: XSDT 00000000bdfa9d98 000BC (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: FACP 00000000bdfa9918 000F4 (v04 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20130517/tbfadt-672) [ 0.000000] ACPI: DSDT 00000000bdf8f018 18762 (v02 INTEL S2600GZ 00000008 INTL 20100331) [ 0.000000] ACPI: FACS 00000000bdfa9f40 00040 [ 0.000000] ACPI: APIC 00000000bdfa8018 00BAA (v03 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SPMI 00000000bdfabb18 00040 (v05 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: MCFG 00000000bdfaba98 0003C (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: WDDT 00000000bdfabf18 00040 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SRAT 00000000bdf8cc18 002A8 (v03 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SLIT 00000000bdfabe98 00030 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: MSCT 00000000bdfaae18 00090 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: HPET 00000000bdfabe18 00038 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SSDT 00000000bdfabd18 0002B (v02 INTEL S2600GZ 00001000 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bddcd018 D30C8 (v02 INTEL S2600GZ 00004000 INTL 20100331) [ 0.000000] ACPI: SPCR 00000000bdfabd98 00050 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: HEST 00000000bdf8ef18 000A8 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: BERT 00000000bdfabc18 00030 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: ERST 00000000bdf8ec98 00230 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: EINJ 00000000bdfa9798 00130 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: SSDT 00000000bdf8a018 01729 (v02 INTEL S2600GZ 00000002 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdfabc98 00045 (v02 INTEL S2600GZ 00000001 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdf8de18 00181 (v02 INTEL S2600GZ 00000003 INTL 20100331) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0d -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0e -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0f -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x22 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x23 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x24 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x25 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x26 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x27 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2a -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2b -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2c -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2d -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2e -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2f -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0xbfffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x43fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x440000000-0x83fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x43ffd9000-0x43fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x83ffd8000-0x83fffefff] [ 0.000000] Reserving 162MB of memory at 688MB for crashkernel (System RAM: 32691MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x83fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0008efff] [ 0.000000] node 0: [mem 0x00100000-0xbb3c6fff] [ 0.000000] node 0: [mem 0xbdfac000-0xbdffffff] [ 0.000000] node 0: [mem 0x100000000-0x43fffffff] [ 0.000000] node 1: [mem 0x440000000-0x83fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff] [ 0.000000] On node 0 totalpages: 4174761 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3982 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 11921 pages used for memmap [ 0.000000] DMA32 zone: 762907 pages, LIFO batch:31 [ 0.000000] Normal zone: 53248 pages used for memmap [ 0.000000] Normal zone: 3407872 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff] [ 0.000000] On node 1 totalpages: 4194304 [ 0.000000] Normal zone: 65536 pages used for memmap [ 0.000000] Normal zone: 4194304 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x0d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x0f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x27] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x2b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x2d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x2f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x58] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x59] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0xff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x40] uid[0x40] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x41] uid[0x41] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x42] uid[0x42] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x43] uid[0x43] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x44] uid[0x44] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x45] uid[0x45] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x46] uid[0x46] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x47] uid[0x47] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x48] uid[0x48] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x49] uid[0x49] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4a] uid[0x4a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4b] uid[0x4b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4c] uid[0x4c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4d] uid[0x4d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4e] uid[0x4e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4f] uid[0x4f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x50] uid[0x50] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x51] uid[0x51] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x52] uid[0x52] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x53] uid[0x53] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x54] uid[0x54] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x55] uid[0x55] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x56] uid[0x56] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x57] uid[0x57] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x58] uid[0x58] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x59] uid[0x59] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5a] uid[0x5a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5b] uid[0x5b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5c] uid[0x5c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5d] uid[0x5d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5e] uid[0x5e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5f] uid[0x5f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x60] uid[0x60] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x61] uid[0x61] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x62] uid[0x62] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x63] uid[0x63] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x64] uid[0x64] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x65] uid[0x65] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x66] uid[0x66] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x67] uid[0x67] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x68] uid[0x68] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x69] uid[0x69] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6a] uid[0x6a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6b] uid[0x6b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6c] uid[0x6c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6d] uid[0x6d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6e] uid[0x6e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6f] uid[0x6f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x70] uid[0x70] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x71] uid[0x71] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x72] uid[0x72] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x73] uid[0x73] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x74] uid[0x74] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x75] uid[0x75] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x76] uid[0x76] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x77] uid[0x77] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 152 CPUs, 120 hotplug CPUs [ 0.000000] smpboot: Ignoring 88 unusable CPUs in ACPI table [ 0.000000] PM: Registered nosave memory: [mem 0x0008f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbb3c7000-0xbdd2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdd2f000-0xbddccfff] [ 0.000000] PM: Registered nosave memory: [mem 0xbddcd000-0xbdea0fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdea1000-0xbdf2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdf2f000-0xbdfabfff] [ 0.000000] PM: Registered nosave memory: [mem 0xbe000000-0xcfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xd0000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfed18fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed19000-0xfed19fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1a000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xffa1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xffa20000-0xffffffff] [ 0.000000] e820: [mem 0xd0000000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:152 nr_cpu_ids:152 nr_node_ids:2 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 [ 0.000000] pcpu-alloc: [0] 016 017 018 019 020 021 022 023 [ 0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 [ 0.000000] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 [ 0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 [ 0.000000] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 [ 0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 [ 0.000000] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 [ 0.000000] pcpu-alloc: [0] 128 130 132 134 136 138 140 142 [ 0.000000] pcpu-alloc: [0] 144 146 148 150 --- --- --- --- [ 0.000000] pcpu-alloc: [1] 008 009 010 011 012 013 014 015 [ 0.000000] pcpu-alloc: [1] 024 025 026 027 028 029 030 031 [ 0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 [ 0.000000] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 [ 0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 [ 0.000000] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 [ 0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 [ 0.000000] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 [ 0.000000] pcpu-alloc: [1] 129 131 133 135 137 139 141 143 [ 0.000000] pcpu-alloc: [1] 145 147 149 151 --- --- --- --- [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 8238275 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.49.1.el7_lustre.x86_64 root=UUID=af55e6d6-cb1d-46c7-a478-fce49fa6f327 ro crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 7010596k/34603008k available (7796k kernel code, 1126748k absent, 818764k reserved, 5947k data, 1980k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=152, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=152. [ 0.000000] NR_IRQS:327936 nr_irqs:2456 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 268435456 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2693.251 MHz processor [ 0.000141] Calibrating delay loop (skipped), value calculated using timer frequency.. 5386.50 BogoMIPS (lpj=2693251) [ 0.012018] pid_max: default: 155648 minimum: 1216 [ 0.017841] Security Framework initialized [ 0.022465] SELinux: Initializing. [ 0.026462] SELinux: Starting in permissive mode [ 0.026464] Yama: becoming mindful. [ 0.035889] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes) [ 0.057677] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.071418] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.079187] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.088396] Initializing cgroup subsys memory [ 0.093315] Initializing cgroup subsys devices [ 0.098284] Initializing cgroup subsys freezer [ 0.103253] Initializing cgroup subsys net_cls [ 0.108221] Initializing cgroup subsys blkio [ 0.112995] Initializing cgroup subsys perf_event [ 0.118275] Initializing cgroup subsys hugetlb [ 0.123243] Initializing cgroup subsys pids [ 0.127920] Initializing cgroup subsys net_prio [ 0.133166] CPU0: Thermal monitoring enabled (TM1) [ 0.138601] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0 [ 0.144734] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32 [ 0.151089] tlb_flushall_shift: 6 [ 0.154883] FEATURE SPEC_CTRL Present [ 0.158977] FEATURE IBPB_SUPPORT Present [ 0.163362] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.174149] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.181521] Spectre V2 : Mitigation: Full retpoline [ 0.187003] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.197738] MDS: Mitigation: Clear CPU buffers [ 0.205204] Freeing SMP alternatives: 28k freed [ 0.216377] ACPI: Core revision 20130517 [ 0.312744] ACPI: All ACPI Tables successfully acquired [ 0.339989] ftrace: allocating 29690 entries in 116 pages [ 0.402771] IRQ remapping doesn't support X2APIC mode, disable x2apic. [ 0.410158] Switched APIC routing to physical flat. [ 0.416329] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.433041] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (fam: 06, model: 2d, stepping: 07) [ 0.443487] TSC deadline timer enabled [ 0.443612] Performance Events: PEBS fmt1+, SandyBridge events, 16-deep LBR, full-width counters, Intel PMU driver. [ 0.455341] ... version: 3 [ 0.459818] ... bit width: 48 [ 0.464392] ... generic registers: 4 [ 0.468859] ... value mask: 0000ffffffffffff [ 0.474782] ... max period: 00007fffffffffff [ 0.480715] ... fixed-purpose events: 3 [ 0.485191] ... event mask: 000000070000000f [ 0.515180] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.496171] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.578432] smpboot: CPU 8 Converting physical 0 to logical die 1 [ 0.568287] smpboot: Booting Node 1, Processors #8 #9 #10 #11 #12 #13 #14 #15 OK [ 0.713637] smpboot: Booting Node 0, Processors #16 [ 0.723742] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 0.739506] #17 #18 #19 #20 #21 #22 #23 OK [ 0.772924] smpboot: Booting Node 1, Processors #24 #25 #26 #27 #28 #29 #30 #31 [ 0.813085] Brought up 32 CPUs [ 0.816702] smpboot: Max logical packages: 10 [ 0.821574] smpboot: Total of 32 processors activated (172578.78 BogoMIPS) [ 1.072710] node 0 initialised, 2810308 pages in 133ms [ 1.109001] node 1 initialised, 3601415 pages in 164ms [ 1.115530] devtmpfs: initialized [ 1.119410] x86/mm: Memory block size: 128MB [ 1.135658] EVM: security.selinux [ 1.139362] EVM: security.ima [ 1.142676] EVM: security.capability [ 1.146898] PM: Registering ACPI NVS region [mem 0xbdd2f000-0xbddccfff] (647168 bytes) [ 1.155771] PM: Registering ACPI NVS region [mem 0xbdea1000-0xbdf2efff] (581632 bytes) [ 1.167937] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.175621] pinctrl core: initialized pinctrl subsystem [ 1.181613] RTC time: 3:32:12, date: 05/04/22 [ 1.186916] NET: Registered protocol family 16 [ 1.192353] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.200811] ACPI: bus type PCI registered [ 1.205291] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.212764] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xc0000000-0xcfffffff] (base 0xc0000000) [ 1.223163] PCI: MMCONFIG at [mem 0xc0000000-0xcfffffff] reserved in E820 [ 1.230797] PCI: Using configuration type 1 for base access [ 1.237068] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 1.256788] ACPI: Added _OSI(Module Device) [ 1.261464] ACPI: Added _OSI(Processor Device) [ 1.266428] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.271681] ACPI: Added _OSI(Processor Aggregator Device) [ 1.277712] ACPI: Added _OSI(Linux-Dell-Video) [ 1.302218] ACPI: EC: Look up EC in DSDT [ 1.322002] ACPI: Executed 1 blocks of module-level executable AML code [ 1.619250] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 1.652834] ACPI: Interpreter enabled [ 1.656959] ACPI: (supports S0 S1 S5) [ 1.661050] ACPI: Using IOAPIC for interrupt routing [ 1.666714] HEST: Table parsing has been initialized. [ 1.672361] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.682637] ACPI: GPE 0x1E active on init [ 1.687152] ACPI: GPE 0x24 active on init [ 1.691653] ACPI: Enabled 9 GPEs in block 00 to 3F [ 1.758394] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 1.765309] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.774984] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug AER] [ 1.783483] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.792037] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.801489] acpi PNP0A08:00: host bridge window expanded to [io 0x0000-0xbfff]; [io 0x0000-0xbfff window] ignored [ 1.813153] acpi PNP0A08:00: ignoring host bridge window [mem 0x000cc000-0x000cffff window] (conflicts with Adapter ROM [mem 0x000c8000-0x000cdbff]) [ 1.828014] acpi PNP0A08:00: ignoring host bridge window [mem 0x000d4000-0x000d7fff window] (conflicts with Adapter ROM [mem 0x000ce000-0x000d7bff]) [ 1.843262] PCI host bridge to bus 0000:00 [ 1.847842] pci_bus 0000:00: root bus resource [io 0x0000-0xbfff] [ 1.854745] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.863105] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] [ 1.871464] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] [ 1.879823] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] [ 1.888182] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] [ 1.896542] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] [ 1.904900] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] [ 1.913257] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] [ 1.921616] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] [ 1.929973] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] [ 1.938331] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] [ 1.946690] pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] [ 1.955049] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xebffffff window] [ 1.963411] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 1.972548] pci_bus 0000:00: root bus resource [bus 00-7e] [ 1.978697] pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000 [ 1.978811] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold [ 1.979004] pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400 [ 1.979130] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 1.979247] pci 0000:00:01.0: System wakeup disabled by ACPI [ 1.985675] pci 0000:00:01.1: [8086:3c03] type 01 class 0x060400 [ 1.985799] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold [ 1.985913] pci 0000:00:01.1: System wakeup disabled by ACPI [ 1.992336] pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400 [ 1.992462] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 1.992577] pci 0000:00:02.0: System wakeup disabled by ACPI [ 1.998991] pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400 [ 1.999117] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold [ 1.999232] pci 0000:00:02.2: System wakeup disabled by ACPI [ 2.005660] pci 0000:00:03.0: [8086:3c08] type 01 class 0x060400 [ 2.005785] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 2.005899] pci 0000:00:03.0: System wakeup disabled by ACPI [ 2.012314] pci 0000:00:04.0: [8086:3c20] type 00 class 0x088000 [ 2.012342] pci 0000:00:04.0: reg 0x10: [mem 0x38007ff90000-0x38007ff93fff 64bit] [ 2.012584] pci 0000:00:04.1: [8086:3c21] type 00 class 0x088000 [ 2.012610] pci 0000:00:04.1: reg 0x10: [mem 0x38007ff80000-0x38007ff83fff 64bit] [ 2.012848] pci 0000:00:04.2: [8086:3c22] type 00 class 0x088000 [ 2.012873] pci 0000:00:04.2: reg 0x10: [mem 0x38007ff70000-0x38007ff73fff 64bit] [ 2.013114] pci 0000:00:04.3: [8086:3c23] type 00 class 0x088000 [ 2.013140] pci 0000:00:04.3: reg 0x10: [mem 0x38007ff60000-0x38007ff63fff 64bit] [ 2.013384] pci 0000:00:04.4: [8086:3c24] type 00 class 0x088000 [ 2.013410] pci 0000:00:04.4: reg 0x10: [mem 0x38007ff50000-0x38007ff53fff 64bit] [ 2.013642] pci 0000:00:04.5: [8086:3c25] type 00 class 0x088000 [ 2.013668] pci 0000:00:04.5: reg 0x10: [mem 0x38007ff40000-0x38007ff43fff 64bit] [ 2.013903] pci 0000:00:04.6: [8086:3c26] type 00 class 0x088000 [ 2.013928] pci 0000:00:04.6: reg 0x10: [mem 0x38007ff30000-0x38007ff33fff 64bit] [ 2.014172] pci 0000:00:04.7: [8086:3c27] type 00 class 0x088000 [ 2.014197] pci 0000:00:04.7: reg 0x10: [mem 0x38007ff20000-0x38007ff23fff 64bit] [ 2.014440] pci 0000:00:05.0: [8086:3c28] type 00 class 0x088000 [ 2.014663] pci 0000:00:05.2: [8086:3c2a] type 00 class 0x088000 [ 2.014883] pci 0000:00:05.4: [8086:3c2c] type 00 class 0x080020 [ 2.014902] pci 0000:00:05.4: reg 0x10: [mem 0xd0f60000-0xd0f60fff] [ 2.015142] pci 0000:00:11.0: [8086:1d3e] type 01 class 0x060400 [ 2.015286] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold [ 2.015487] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000 [ 2.015515] pci 0000:00:16.0: reg 0x10: [mem 0xd0f50000-0xd0f5000f 64bit] [ 2.015607] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 2.015767] pci 0000:00:16.1: [8086:1d3b] type 00 class 0x078000 [ 2.015795] pci 0000:00:16.1: reg 0x10: [mem 0xd0f40000-0xd0f4000f 64bit] [ 2.015886] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 2.016063] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320 [ 2.016089] pci 0000:00:1a.0: reg 0x10: [mem 0xd0f20000-0xd0f203ff] [ 2.016206] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 2.016335] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 2.022733] pci 0000:00:1c.0: [8086:1d10] type 01 class 0x060400 [ 2.022884] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 2.022997] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 2.029416] pci 0000:00:1c.7: [8086:1d1e] type 01 class 0x060400 [ 2.029536] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 2.029639] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 2.036046] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320 [ 2.036072] pci 0000:00:1d.0: reg 0x10: [mem 0xd0f10000-0xd0f103ff] [ 2.036191] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 2.036312] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 2.042707] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401 [ 2.042868] pci 0000:00:1e.0: System wakeup disabled by ACPI [ 2.049271] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100 [ 2.049554] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601 [ 2.049579] pci 0000:00:1f.2: reg 0x10: [io 0x4070-0x4077] [ 2.049592] pci 0000:00:1f.2: reg 0x14: [io 0x4060-0x4063] [ 2.049605] pci 0000:00:1f.2: reg 0x18: [io 0x4050-0x4057] [ 2.049618] pci 0000:00:1f.2: reg 0x1c: [io 0x4040-0x4043] [ 2.049631] pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f] [ 2.049644] pci 0000:00:1f.2: reg 0x24: [mem 0xd0f00000-0xd0f007ff] [ 2.049708] pci 0000:00:1f.2: PME# supported from D3hot [ 2.049862] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500 [ 2.049887] pci 0000:00:1f.3: reg 0x10: [mem 0x38007ff10000-0x38007ff100ff 64bit] [ 2.049918] pci 0000:00:1f.3: reg 0x20: [io 0x4000-0x401f] [ 2.050170] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.055878] acpiphp: Slot [2] registered [ 2.060323] pci 0000:02:00.0: [8086:1521] type 00 class 0x020000 [ 2.060349] pci 0000:02:00.0: reg 0x10: [mem 0xd0b60000-0xd0b7ffff] [ 2.060370] pci 0000:02:00.0: reg 0x18: [io 0x3060-0x307f] [ 2.060383] pci 0000:02:00.0: reg 0x1c: [mem 0xd0bb0000-0xd0bb3fff] [ 2.060509] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 2.060545] pci 0000:02:00.0: reg 0x184: [mem 0xd0ca0000-0xd0ca3fff] [ 2.060551] pci 0000:02:00.0: VF(n) BAR0 space: [mem 0xd0ca0000-0xd0cbffff] (contains BAR0 for 8 VFs) [ 2.070868] pci 0000:02:00.0: reg 0x190: [mem 0xd0c80000-0xd0c83fff] [ 2.070874] pci 0000:02:00.0: VF(n) BAR3 space: [mem 0xd0c80000-0xd0c9ffff] (contains BAR3 for 8 VFs) [ 2.081226] pci 0000:02:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:00:01.1 (capable of 31.504 Gb/s with 8 GT/s x4 link) [ 2.096794] pci 0000:02:00.1: [8086:1521] type 00 class 0x020000 [ 2.096819] pci 0000:02:00.1: reg 0x10: [mem 0xd0b40000-0xd0b5ffff] [ 2.096840] pci 0000:02:00.1: reg 0x18: [io 0x3040-0x305f] [ 2.096853] pci 0000:02:00.1: reg 0x1c: [mem 0xd0ba0000-0xd0ba3fff] [ 2.096972] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold [ 2.097001] pci 0000:02:00.1: reg 0x184: [mem 0xd0c60000-0xd0c63fff] [ 2.097006] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0xd0c60000-0xd0c7ffff] (contains BAR0 for 8 VFs) [ 2.107326] pci 0000:02:00.1: reg 0x190: [mem 0xd0c40000-0xd0c43fff] [ 2.107331] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0xd0c40000-0xd0c5ffff] (contains BAR3 for 8 VFs) [ 2.117793] pci 0000:02:00.2: [8086:1521] type 00 class 0x020000 [ 2.117817] pci 0000:02:00.2: reg 0x10: [mem 0xd0b20000-0xd0b3ffff] [ 2.117838] pci 0000:02:00.2: reg 0x18: [io 0x3020-0x303f] [ 2.117851] pci 0000:02:00.2: reg 0x1c: [mem 0xd0b90000-0xd0b93fff] [ 2.117968] pci 0000:02:00.2: PME# supported from D0 D3hot D3cold [ 2.117997] pci 0000:02:00.2: reg 0x184: [mem 0xd0c20000-0xd0c23fff] [ 2.118003] pci 0000:02:00.2: VF(n) BAR0 space: [mem 0xd0c20000-0xd0c3ffff] (contains BAR0 for 8 VFs) [ 2.128321] pci 0000:02:00.2: reg 0x190: [mem 0xd0c00000-0xd0c03fff] [ 2.128326] pci 0000:02:00.2: VF(n) BAR3 space: [mem 0xd0c00000-0xd0c1ffff] (contains BAR3 for 8 VFs) [ 2.138783] pci 0000:02:00.3: [8086:1521] type 00 class 0x020000 [ 2.138807] pci 0000:02:00.3: reg 0x10: [mem 0xd0b00000-0xd0b1ffff] [ 2.138828] pci 0000:02:00.3: reg 0x18: [io 0x3000-0x301f] [ 2.138841] pci 0000:02:00.3: reg 0x1c: [mem 0xd0b80000-0xd0b83fff] [ 2.138958] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold [ 2.138988] pci 0000:02:00.3: reg 0x184: [mem 0xd0be0000-0xd0be3fff] [ 2.138993] pci 0000:02:00.3: VF(n) BAR0 space: [mem 0xd0be0000-0xd0bfffff] (contains BAR0 for 8 VFs) [ 2.149304] pci 0000:02:00.3: reg 0x190: [mem 0xd0bc0000-0xd0bc3fff] [ 2.149309] pci 0000:02:00.3: VF(n) BAR3 space: [mem 0xd0bc0000-0xd0bdffff] (contains BAR3 for 8 VFs) [ 2.161636] pci 0000:00:01.1: PCI bridge to [bus 02-03] [ 2.167477] pci 0000:00:01.1: bridge window [io 0x3000-0x3fff] [ 2.167484] pci 0000:00:01.1: bridge window [mem 0xd0b00000-0xd0cfffff] [ 2.167694] acpiphp: Slot [2-2] registered [ 2.172537] pci 0000:04:00.0: [15b3:1003] type 00 class 0x028000 [ 2.172937] pci 0000:04:00.0: reg 0x10: [mem 0xd0e00000-0xd0efffff 64bit] [ 2.173160] pci 0000:04:00.0: reg 0x18: [mem 0x38007f000000-0x38007f7fffff 64bit pref] [ 2.175350] pci 0000:04:00.0: reg 0x134: [mem 0x38007b000000-0x38007b7fffff 64bit pref] [ 2.175356] pci 0000:04:00.0: VF(n) BAR2 space: [mem 0x38007b000000-0x38007effffff 64bit pref] (contains BAR2 for 8 VFs) [ 2.188805] pci 0000:00:02.0: PCI bridge to [bus 04] [ 2.194357] pci 0000:00:02.0: bridge window [mem 0xd0e00000-0xd0efffff] [ 2.194366] pci 0000:00:02.0: bridge window [mem 0x38007b000000-0x38007f7fffff 64bit pref] [ 2.194530] acpiphp: Slot [2-3] registered [ 2.199149] pci 0000:00:02.2: PCI bridge to [bus 05] [ 2.204858] acpiphp: Slot [2-4] registered [ 2.209491] pci 0000:06:00.0: [1000:0087] type 00 class 0x010700 [ 2.209511] pci 0000:06:00.0: reg 0x10: [io 0x2000-0x20ff] [ 2.209527] pci 0000:06:00.0: reg 0x14: [mem 0xd0a40000-0xd0a4ffff 64bit] [ 2.209543] pci 0000:06:00.0: reg 0x1c: [mem 0xd0a00000-0xd0a3ffff 64bit] [ 2.209561] pci 0000:06:00.0: reg 0x30: [mem 0xd0900000-0xd09fffff pref] [ 2.209661] pci 0000:06:00.0: supports D1 D2 [ 2.209695] pci 0000:06:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:03.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) [ 2.225236] pci 0000:00:03.0: PCI bridge to [bus 06] [ 2.230784] pci 0000:00:03.0: bridge window [io 0x2000-0x2fff] [ 2.230791] pci 0000:00:03.0: bridge window [mem 0xd0900000-0xd0afffff] [ 2.230912] pci 0000:07:00.0: [8086:1d6b] type 00 class 0x010700 [ 2.230948] pci 0000:07:00.0: reg 0x10: [mem 0x38007fc00000-0x38007fc03fff 64bit pref] [ 2.230969] pci 0000:07:00.0: reg 0x18: [mem 0x38007f800000-0x38007fbfffff 64bit pref] [ 2.230985] pci 0000:07:00.0: reg 0x20: [io 0x1000-0x10ff] [ 2.231129] pci 0000:07:00.0: reg 0x164: [mem 0x38007fc10000-0x38007fc13fff 64bit pref] [ 2.231135] pci 0000:07:00.0: VF(n) BAR0 space: [mem 0x38007fc10000-0x38007fc8bfff 64bit pref] (contains BAR0 for 31 VFs) [ 2.243538] pci 0000:07:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:11.0 (capable of 7.876 Gb/s with 8 GT/s x1 link) [ 2.259064] pci 0000:07:00.3: [8086:1d70] type 00 class 0x0c0500 [ 2.259092] pci 0000:07:00.3: reg 0x10: [mem 0xd0d00000-0xd0d00fff] [ 2.259145] pci 0000:07:00.3: reg 0x20: [io 0x1100-0x111f] [ 2.259249] pci 0000:07:00.3: PME# supported from D0 D3hot D3cold [ 2.259404] pci 0000:00:11.0: PCI bridge to [bus 07] [ 2.264955] pci 0000:00:11.0: bridge window [io 0x1000-0x1fff] [ 2.264962] pci 0000:00:11.0: bridge window [mem 0xd0d00000-0xd0dfffff] [ 2.264973] pci 0000:00:11.0: bridge window [mem 0x38007f800000-0x38007fcfffff 64bit pref] [ 2.265078] pci 0000:00:1c.0: PCI bridge to [bus 08] [ 2.270751] pci 0000:09:00.0: [102b:0522] type 00 class 0x030000 [ 2.270790] pci 0000:09:00.0: reg 0x10: [mem 0xea000000-0xeaffffff pref] [ 2.270814] pci 0000:09:00.0: reg 0x14: [mem 0xd0810000-0xd0813fff] [ 2.270837] pci 0000:09:00.0: reg 0x18: [mem 0xd0000000-0xd07fffff] [ 2.270913] pci 0000:09:00.0: reg 0x30: [mem 0xd0800000-0xd080ffff pref] [ 2.271071] pci 0000:09:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:1c.7 (capable of 7.876 Gb/s with 8 GT/s x1 link) [ 2.286534] pci 0000:09:00.0: System wakeup disabled by ACPI [ 2.292944] pci 0000:00:1c.7: PCI bridge to [bus 09] [ 2.298498] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 2.298508] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 2.298621] pci 0000:00:1e.0: PCI bridge to [bus 0a] (subtractive decode) [ 2.306215] pci 0000:00:1e.0: bridge window [io 0x0000-0xbfff] (subtractive decode) [ 2.306220] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode) [ 2.306225] pci 0000:00:1e.0: bridge window [mem 0x000c0000-0x000c3fff window] (subtractive decode) [ 2.306229] pci 0000:00:1e.0: bridge window [mem 0x000c4000-0x000c7fff window] (subtractive decode) [ 2.306233] pci 0000:00:1e.0: bridge window [mem 0x000c8000-0x000cbfff window] (subtractive decode) [ 2.306238] pci 0000:00:1e.0: bridge window [mem 0x000d0000-0x000d3fff window] (subtractive decode) [ 2.306242] pci 0000:00:1e.0: bridge window [mem 0x000d8000-0x000dbfff window] (subtractive decode) [ 2.306247] pci 0000:00:1e.0: bridge window [mem 0x000dc000-0x000dffff window] (subtractive decode) [ 2.306251] pci 0000:00:1e.0: bridge window [mem 0x000e0000-0x000e3fff window] (subtractive decode) [ 2.306255] pci 0000:00:1e.0: bridge window [mem 0x000e4000-0x000e7fff window] (subtractive decode) [ 2.306260] pci 0000:00:1e.0: bridge window [mem 0x000e8000-0x000ebfff window] (subtractive decode) [ 2.306264] pci 0000:00:1e.0: bridge window [mem 0x000ec000-0x000effff window] (subtractive decode) [ 2.306268] pci 0000:00:1e.0: bridge window [mem 0x000f0000-0x000fffff window] (subtractive decode) [ 2.306273] pci 0000:00:1e.0: bridge window [mem 0xd0000000-0xebffffff window] (subtractive decode) [ 2.306277] pci 0000:00:1e.0: bridge window [mem 0x380000000000-0x38007fffffff window] (subtractive decode) [ 2.306358] pci_bus 0000:00: on NUMA node 0 [ 2.307655] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 2.315544] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 2.323451] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15) [ 2.331345] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15) [ 2.339232] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15) [ 2.347135] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15) [ 2.355032] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15) [ 2.362912] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15) [ 2.371017] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 2.377927] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.387349] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug AER] [ 2.395679] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.404233] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.413802] PCI host bridge to bus 0000:80 [ 2.418383] pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] [ 2.425971] pci_bus 0000:80: root bus resource [io 0xc000-0xffff window] [ 2.433554] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.441913] pci_bus 0000:80: root bus resource [mem 0xec000000-0xfbffffff window] [ 2.450273] pci_bus 0000:80: root bus resource [mem 0x380080000000-0x3800ffffffff window] [ 2.459408] pci_bus 0000:80: root bus resource [bus 80-fe] [ 2.465556] pci 0000:80:01.0: [8086:3c02] type 01 class 0x060400 [ 2.465692] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold [ 2.465775] pci 0000:80:01.0: System wakeup disabled by ACPI [ 2.472199] pci 0000:80:02.0: [8086:3c04] type 01 class 0x060400 [ 2.472332] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold [ 2.472416] pci 0000:80:02.0: System wakeup disabled by ACPI [ 2.478846] pci 0000:80:03.0: [8086:3c08] type 01 class 0x060400 [ 2.478978] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold [ 2.479057] pci 0000:80:03.0: System wakeup disabled by ACPI [ 2.485484] pci 0000:80:03.2: [8086:3c0a] type 01 class 0x060400 [ 2.485613] pci 0000:80:03.2: PME# supported from D0 D3hot D3cold [ 2.485696] pci 0000:80:03.2: System wakeup disabled by ACPI [ 2.492119] pci 0000:80:04.0: [8086:3c20] type 00 class 0x088000 [ 2.492146] pci 0000:80:04.0: reg 0x10: [mem 0x3800fff70000-0x3800fff73fff 64bit] [ 2.492359] pci 0000:80:04.1: [8086:3c21] type 00 class 0x088000 [ 2.492386] pci 0000:80:04.1: reg 0x10: [mem 0x3800fff60000-0x3800fff63fff 64bit] [ 2.492592] pci 0000:80:04.2: [8086:3c22] type 00 class 0x088000 [ 2.492618] pci 0000:80:04.2: reg 0x10: [mem 0x3800fff50000-0x3800fff53fff 64bit] [ 2.492825] pci 0000:80:04.3: [8086:3c23] type 00 class 0x088000 [ 2.492851] pci 0000:80:04.3: reg 0x10: [mem 0x3800fff40000-0x3800fff43fff 64bit] [ 2.493067] pci 0000:80:04.4: [8086:3c24] type 00 class 0x088000 [ 2.493093] pci 0000:80:04.4: reg 0x10: [mem 0x3800fff30000-0x3800fff33fff 64bit] [ 2.493299] pci 0000:80:04.5: [8086:3c25] type 00 class 0x088000 [ 2.493325] pci 0000:80:04.5: reg 0x10: [mem 0x3800fff20000-0x3800fff23fff 64bit] [ 2.493535] pci 0000:80:04.6: [8086:3c26] type 00 class 0x088000 [ 2.493561] pci 0000:80:04.6: reg 0x10: [mem 0x3800fff10000-0x3800fff13fff 64bit] [ 2.493765] pci 0000:80:04.7: [8086:3c27] type 00 class 0x088000 [ 2.493791] pci 0000:80:04.7: reg 0x10: [mem 0x3800fff00000-0x3800fff03fff 64bit] [ 2.493996] pci 0000:80:05.0: [8086:3c28] type 00 class 0x088000 [ 2.494193] pci 0000:80:05.2: [8086:3c2a] type 00 class 0x088000 [ 2.494382] pci 0000:80:05.4: [8086:3c2c] type 00 class 0x080020 [ 2.494402] pci 0000:80:05.4: reg 0x10: [mem 0xec000000-0xec000fff] [ 2.494696] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.500338] pci 0000:80:02.0: PCI bridge to [bus 82] [ 2.505989] pci 0000:80:03.0: PCI bridge to [bus 83] [ 2.511629] pci 0000:80:03.2: PCI bridge to [bus 84] [ 2.517227] pci_bus 0000:80: on NUMA node 1 [ 2.528604] ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f]) [ 2.535226] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.544420] acpi PNP0A03:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER] [ 2.554097] acpi PNP0A03:00: _OSC: OS now controls [PCIeCapability] [ 2.561090] acpi PNP0A03:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.570362] PCI host bridge to bus 0000:7f [ 2.574941] pci_bus 0000:7f: root bus resource [bus 7f] [ 2.580794] pci 0000:7f:08.0: [8086:3c80] type 00 class 0x088000 [ 2.580931] pci 0000:7f:09.0: [8086:3c90] type 00 class 0x088000 [ 2.581043] pci 0000:7f:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.581148] pci 0000:7f:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.581249] pci 0000:7f:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.581353] pci 0000:7f:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.581456] pci 0000:7f:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.581561] pci 0000:7f:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.581662] pci 0000:7f:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.581771] pci 0000:7f:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.581877] pci 0000:7f:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.581980] pci 0000:7f:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.582084] pci 0000:7f:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.582186] pci 0000:7f:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.582285] pci 0000:7f:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.582388] pci 0000:7f:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.582486] pci 0000:7f:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.582591] pci 0000:7f:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.582692] pci 0000:7f:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.582805] pci 0000:7f:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.582912] pci 0000:7f:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.583033] pci 0000:7f:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.583170] pci 0000:7f:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.583305] pci 0000:7f:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.583439] pci 0000:7f:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.583575] pci 0000:7f:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.583710] pci 0000:7f:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.583845] pci 0000:7f:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.583960] pci 0000:7f:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.584093] pci 0000:7f:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.584228] pci 0000:7f:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.584359] pci 0000:7f:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.584494] pci 0000:7f:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.584633] pci 0000:7f:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.584765] pci 0000:7f:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.584908] pci 0000:7f:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.585037] pci 0000:7f:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.585156] pci 0000:7f:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.585257] pci 0000:7f:13.1: [8086:3c43] type 00 class 0x110100 [ 2.585364] pci 0000:7f:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.585464] pci 0000:7f:13.5: [8086:3c44] type 00 class 0x110100 [ 2.585571] pci 0000:7f:13.6: [8086:3c45] type 00 class 0x088000 [ 2.585783] ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff]) [ 2.592403] acpi PNP0A03:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.601596] acpi PNP0A03:01: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER] [ 2.611266] acpi PNP0A03:01: _OSC: OS now controls [PCIeCapability] [ 2.618267] acpi PNP0A03:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.627536] PCI host bridge to bus 0000:ff [ 2.632106] pci_bus 0000:ff: root bus resource [bus ff] [ 2.637962] pci 0000:ff:08.0: [8086:3c80] type 00 class 0x088000 [ 2.638083] pci 0000:ff:09.0: [8086:3c90] type 00 class 0x088000 [ 2.638201] pci 0000:ff:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.638305] pci 0000:ff:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.638412] pci 0000:ff:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.638517] pci 0000:ff:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.638628] pci 0000:ff:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.638733] pci 0000:ff:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.638842] pci 0000:ff:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.638951] pci 0000:ff:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.639064] pci 0000:ff:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.639167] pci 0000:ff:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.639277] pci 0000:ff:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.639379] pci 0000:ff:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.639486] pci 0000:ff:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.639592] pci 0000:ff:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.639699] pci 0000:ff:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.639802] pci 0000:ff:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.639912] pci 0000:ff:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.640024] pci 0000:ff:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.640139] pci 0000:ff:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.640261] pci 0000:ff:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.640403] pci 0000:ff:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.640543] pci 0000:ff:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.640682] pci 0000:ff:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.640818] pci 0000:ff:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.640963] pci 0000:ff:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.641102] pci 0000:ff:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.641216] pci 0000:ff:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.641361] pci 0000:ff:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.641499] pci 0000:ff:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.641641] pci 0000:ff:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.641781] pci 0000:ff:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.641922] pci 0000:ff:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.642067] pci 0000:ff:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.642207] pci 0000:ff:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.642346] pci 0000:ff:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.642466] pci 0000:ff:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.642573] pci 0000:ff:13.1: [8086:3c43] type 00 class 0x110100 [ 2.642684] pci 0000:ff:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.642793] pci 0000:ff:13.5: [8086:3c44] type 00 class 0x110100 [ 2.642898] pci 0000:ff:13.6: [8086:3c45] type 00 class 0x088000 [ 2.643608] vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.652681] vgaarb: loaded [ 2.655704] vgaarb: bridge control possible 0000:09:00.0 [ 2.661938] SCSI subsystem initialized [ 2.666197] ACPI: bus type USB registered [ 2.670723] usbcore: registered new interface driver usbfs [ 2.676875] usbcore: registered new interface driver hub [ 2.683062] usbcore: registered new device driver usb [ 2.689165] EDAC MC: Ver: 3.0.0 [ 2.693246] PCI: Using ACPI for IRQ routing [ 2.704689] PCI: pci_cache_line_size set to 64 bytes [ 2.705058] e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] [ 2.705062] e820: reserve RAM buffer [mem 0xbb3c7000-0xbbffffff] [ 2.705065] e820: reserve RAM buffer [mem 0xbe000000-0xbfffffff] [ 2.705446] NetLabel: Initializing [ 2.709247] NetLabel: domain hash size = 128 [ 2.714112] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.719699] NetLabel: unlabeled traffic allowed by default [ 2.726222] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.733243] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.741822] amd_nb: Cannot enumerate AMD northbridges [ 2.747611] Switched to clocksource hpet [ 2.765217] pnp: PnP ACPI init [ 2.768660] ACPI: bus type PNP registered [ 2.773433] system 00:00: [io 0x0680-0x069f] has been reserved [ 2.780058] system 00:00: [io 0xffff] has been reserved [ 2.785994] system 00:00: [io 0xffff] has been reserved [ 2.791932] system 00:00: [io 0xffff] has been reserved [ 2.797872] system 00:00: [io 0x0400-0x0453] could not be reserved [ 2.804867] system 00:00: [io 0x0458-0x047f] has been reserved [ 2.811489] system 00:00: [io 0x0500-0x057f] has been reserved [ 2.818105] system 00:00: [io 0x0600-0x061f] has been reserved [ 2.824722] system 00:00: [io 0x0ca2-0x0ca5] could not be reserved [ 2.831727] system 00:00: [io 0x0cf9] could not be reserved [ 2.838054] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.838124] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.838276] system 00:02: [io 0x0454-0x0457] has been reserved [ 2.844899] system 00:02: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 2.845298] pnp 00:03: [dma 0 disabled] [ 2.845432] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.845725] pnp 00:04: [dma 0 disabled] [ 2.845859] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.846485] system 00:05: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 2.853883] system 00:05: [mem 0xebfff000-0xebffffff] has been reserved [ 2.861275] system 00:05: [mem 0xc0000000-0xcfffffff] has been reserved [ 2.868667] system 00:05: [mem 0xfed20000-0xfed3ffff] has been reserved [ 2.876061] system 00:05: [mem 0xfed45000-0xfed8ffff] has been reserved [ 2.883454] system 00:05: [mem 0xff000000-0xffffffff] could not be reserved [ 2.891234] system 00:05: [mem 0xfee00000-0xfeefffff] could not be reserved [ 2.899012] system 00:05: [mem 0xfec00000-0xfecfffff] could not be reserved [ 2.906793] system 00:05: [mem 0xd0f70000-0xd0f70fff] has been reserved [ 2.914187] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.914739] system 00:06: [mem 0x00000000-0x0009cfff] could not be reserved [ 2.922525] system 00:06: Plug and Play ACPI device, IDs PNP0c01 (active) [ 2.923074] pnp: PnP ACPI: found 7 devices [ 2.927659] ACPI: bus type PNP unregistered [ 2.941590] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.947170] pci 0000:00:01.1: PCI bridge to [bus 02-03] [ 2.953010] pci 0000:00:01.1: bridge window [io 0x3000-0x3fff] [ 2.959825] pci 0000:00:01.1: bridge window [mem 0xd0b00000-0xd0cfffff] [ 2.967418] pci 0000:00:02.0: PCI bridge to [bus 04] [ 2.972971] pci 0000:00:02.0: bridge window [mem 0xd0e00000-0xd0efffff] [ 2.980559] pci 0000:00:02.0: bridge window [mem 0x38007b000000-0x38007f7fffff 64bit pref] [ 2.989990] pci 0000:00:02.2: PCI bridge to [bus 05] [ 2.995551] pci 0000:00:03.0: PCI bridge to [bus 06] [ 3.001093] pci 0000:00:03.0: bridge window [io 0x2000-0x2fff] [ 3.007907] pci 0000:00:03.0: bridge window [mem 0xd0900000-0xd0afffff] [ 3.015500] pci 0000:00:11.0: PCI bridge to [bus 07] [ 3.021051] pci 0000:00:11.0: bridge window [io 0x1000-0x1fff] [ 3.027867] pci 0000:00:11.0: bridge window [mem 0xd0d00000-0xd0dfffff] [ 3.035454] pci 0000:00:11.0: bridge window [mem 0x38007f800000-0x38007fcfffff 64bit pref] [ 3.044879] pci 0000:00:1c.0: PCI bridge to [bus 08] [ 3.050444] pci 0000:00:1c.7: PCI bridge to [bus 09] [ 3.055999] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 3.063586] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 3.072247] pci 0000:00:1e.0: PCI bridge to [bus 0a] [ 3.077809] pci_bus 0000:00: resource 4 [io 0x0000-0xbfff] [ 3.077814] pci_bus 0000:00: resource 5 [mem 0x000a0000-0x000bffff window] [ 3.077819] pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] [ 3.077823] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] [ 3.077828] pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] [ 3.077832] pci_bus 0000:00: resource 9 [mem 0x000d0000-0x000d3fff window] [ 3.077836] pci_bus 0000:00: resource 10 [mem 0x000d8000-0x000dbfff window] [ 3.077841] pci_bus 0000:00: resource 11 [mem 0x000dc000-0x000dffff window] [ 3.077846] pci_bus 0000:00: resource 12 [mem 0x000e0000-0x000e3fff window] [ 3.077850] pci_bus 0000:00: resource 13 [mem 0x000e4000-0x000e7fff window] [ 3.077854] pci_bus 0000:00: resource 14 [mem 0x000e8000-0x000ebfff window] [ 3.077859] pci_bus 0000:00: resource 15 [mem 0x000ec000-0x000effff window] [ 3.077863] pci_bus 0000:00: resource 16 [mem 0x000f0000-0x000fffff window] [ 3.077868] pci_bus 0000:00: resource 17 [mem 0xd0000000-0xebffffff window] [ 3.077872] pci_bus 0000:00: resource 18 [mem 0x380000000000-0x38007fffffff window] [ 3.077877] pci_bus 0000:02: resource 0 [io 0x3000-0x3fff] [ 3.077882] pci_bus 0000:02: resource 1 [mem 0xd0b00000-0xd0cfffff] [ 3.077886] pci_bus 0000:04: resource 1 [mem 0xd0e00000-0xd0efffff] [ 3.077891] pci_bus 0000:04: resource 2 [mem 0x38007b000000-0x38007f7fffff 64bit pref] [ 3.077895] pci_bus 0000:06: resource 0 [io 0x2000-0x2fff] [ 3.077899] pci_bus 0000:06: resource 1 [mem 0xd0900000-0xd0afffff] [ 3.077904] pci_bus 0000:07: resource 0 [io 0x1000-0x1fff] [ 3.077908] pci_bus 0000:07: resource 1 [mem 0xd0d00000-0xd0dfffff] [ 3.077913] pci_bus 0000:07: resource 2 [mem 0x38007f800000-0x38007fcfffff 64bit pref] [ 3.077918] pci_bus 0000:09: resource 1 [mem 0xd0000000-0xd08fffff] [ 3.077922] pci_bus 0000:09: resource 2 [mem 0xea000000-0xeaffffff 64bit pref] [ 3.077927] pci_bus 0000:0a: resource 4 [io 0x0000-0xbfff] [ 3.077931] pci_bus 0000:0a: resource 5 [mem 0x000a0000-0x000bffff window] [ 3.077935] pci_bus 0000:0a: resource 6 [mem 0x000c0000-0x000c3fff window] [ 3.077940] pci_bus 0000:0a: resource 7 [mem 0x000c4000-0x000c7fff window] [ 3.077944] pci_bus 0000:0a: resource 8 [mem 0x000c8000-0x000cbfff window] [ 3.077948] pci_bus 0000:0a: resource 9 [mem 0x000d0000-0x000d3fff window] [ 3.077953] pci_bus 0000:0a: resource 10 [mem 0x000d8000-0x000dbfff window] [ 3.077957] pci_bus 0000:0a: resource 11 [mem 0x000dc000-0x000dffff window] [ 3.077962] pci_bus 0000:0a: resource 12 [mem 0x000e0000-0x000e3fff window] [ 3.077966] pci_bus 0000:0a: resource 13 [mem 0x000e4000-0x000e7fff window] [ 3.077970] pci_bus 0000:0a: resource 14 [mem 0x000e8000-0x000ebfff window] [ 3.077975] pci_bus 0000:0a: resource 15 [mem 0x000ec000-0x000effff window] [ 3.077979] pci_bus 0000:0a: resource 16 [mem 0x000f0000-0x000fffff window] [ 3.077984] pci_bus 0000:0a: resource 17 [mem 0xd0000000-0xebffffff window] [ 3.077988] pci_bus 0000:0a: resource 18 [mem 0x380000000000-0x38007fffffff window] [ 3.077999] pci 0000:80:01.0: PCI bridge to [bus 81] [ 3.083559] pci 0000:80:02.0: PCI bridge to [bus 82] [ 3.089109] pci 0000:80:03.0: PCI bridge to [bus 83] [ 3.094669] pci 0000:80:03.2: PCI bridge to [bus 84] [ 3.100232] pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] [ 3.100236] pci_bus 0000:80: resource 5 [io 0xc000-0xffff window] [ 3.100241] pci_bus 0000:80: resource 6 [mem 0x000a0000-0x000bffff window] [ 3.100245] pci_bus 0000:80: resource 7 [mem 0xec000000-0xfbffffff window] [ 3.100250] pci_bus 0000:80: resource 8 [mem 0x380080000000-0x3800ffffffff window] [ 3.100454] NET: Registered protocol family 2 [ 3.106909] TCP established hash table entries: 262144 (order: 9, 2097152 bytes) [ 3.116073] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 3.123839] TCP: Hash tables configured (established 262144 bind 65536) [ 3.131319] TCP: reno registered [ 3.135087] UDP hash table entries: 16384 (order: 7, 524288 bytes) [ 3.142239] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes) [ 3.150533] NET: Registered protocol family 1 [ 4.255740] pci 0000:00:1a.0: EHCI: BIOS handoff failed (BIOS bug?) 01010001 [ 5.363845] pci 0000:00:1d.0: EHCI: BIOS handoff failed (BIOS bug?) 01010001 [ 5.372029] PCI: CLS mismatch (64 != 32), using 64 bytes [ 5.372055] pci 0000:09:00.0: Boot video device [ 5.372350] Unpacking initramfs... [ 5.987453] Freeing initrd memory: 19724k freed [ 6.000165] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.007376] software IO TLB [mem 0xb73c7000-0xbb3c7000] (64MB) mapped at [ffff99e4f73c7000-ffff99e4fb3c6fff] [ 6.018763] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer [ 6.027619] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules [ 6.034049] RAPL PMU: hw unit of domain package 2^-16 Joules [ 6.040377] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 6.055808] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 6.062642] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 6.074582] futex hash table entries: 65536 (order: 10, 4194304 bytes) [ 6.082848] Initialise system trusted keyring [ 6.087820] audit: initializing netlink socket (disabled) [ 6.093907] type=2000 audit(1651635133.587:1): initialized [ 6.156552] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 6.163661] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 6.174115] zpool: loaded [ 6.177055] zbud: loaded [ 6.180842] VFS: Disk quotas dquot_6.6.0 [ 6.185487] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 6.193752] Key type big_key registered [ 6.198042] SELinux: Registering netfilter hooks [ 6.200321] NET: Registered protocol family 38 [ 6.205309] Key type asymmetric registered [ 6.209899] Asymmetric key parser 'x509' registered [ 6.215471] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 6.223966] io scheduler noop registered [ 6.228361] io scheduler deadline registered (default) [ 6.234190] io scheduler cfq registered [ 6.238485] io scheduler mq-deadline registered [ 6.243558] io scheduler kyber registered [ 6.248935] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 6.249295] pcieport 0000:00:01.1: irq 26 for MSI/MSI-X [ 6.249628] pcieport 0000:00:02.0: irq 27 for MSI/MSI-X [ 6.249985] pcieport 0000:00:02.2: irq 28 for MSI/MSI-X [ 6.250361] pcieport 0000:00:03.0: irq 29 for MSI/MSI-X [ 6.250690] pcieport 0000:00:11.0: irq 30 for MSI/MSI-X [ 6.251040] pcieport 0000:00:1c.0: irq 31 for MSI/MSI-X [ 6.251400] pcieport 0000:00:1c.7: irq 32 for MSI/MSI-X [ 6.251991] pcieport 0000:80:01.0: irq 34 for MSI/MSI-X [ 6.252345] pcieport 0000:80:02.0: irq 35 for MSI/MSI-X [ 6.252635] pcieport 0000:80:03.0: irq 36 for MSI/MSI-X [ 6.252930] pcieport 0000:80:03.2: irq 37 for MSI/MSI-X [ 6.253143] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 6.260938] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded [ 6.260971] pcieport 0000:00:01.1: Signaling PME through PCIe PME interrupt [ 6.268755] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 6.276059] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt [ 6.283358] pci 0000:02:00.2: Signaling PME through PCIe PME interrupt [ 6.290660] pci 0000:02:00.3: Signaling PME through PCIe PME interrupt [ 6.297948] pcie_pme 0000:00:01.1:pcie001: service driver pcie_pme loaded [ 6.297987] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 6.305769] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 6.313066] pcie_pme 0000:00:02.0:pcie001: service driver pcie_pme loaded [ 6.313096] pcieport 0000:00:02.2: Signaling PME through PCIe PME interrupt [ 6.320886] pcie_pme 0000:00:02.2:pcie001: service driver pcie_pme loaded [ 6.320920] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 6.328713] pci 0000:06:00.0: Signaling PME through PCIe PME interrupt [ 6.336015] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded [ 6.336048] pcieport 0000:00:11.0: Signaling PME through PCIe PME interrupt [ 6.343834] pci 0000:07:00.0: Signaling PME through PCIe PME interrupt [ 6.351134] pci 0000:07:00.3: Signaling PME through PCIe PME interrupt [ 6.358434] pcie_pme 0000:00:11.0:pcie001: service driver pcie_pme loaded [ 6.358473] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 6.366264] pcie_pme 0000:00:1c.0:pcie001: service driver pcie_pme loaded [ 6.366295] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt [ 6.374082] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt [ 6.381389] pcie_pme 0000:00:1c.7:pcie001: service driver pcie_pme loaded [ 6.381426] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt [ 6.389221] pcie_pme 0000:80:01.0:pcie001: service driver pcie_pme loaded [ 6.389258] pcieport 0000:80:02.0: Signaling PME through PCIe PME interrupt [ 6.397052] pcie_pme 0000:80:02.0:pcie001: service driver pcie_pme loaded [ 6.397089] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt [ 6.404885] pcie_pme 0000:80:03.0:pcie001: service driver pcie_pme loaded [ 6.404919] pcieport 0000:80:03.2: Signaling PME through PCIe PME interrupt [ 6.412709] pcie_pme 0000:80:03.2:pcie001: service driver pcie_pme loaded [ 6.412835] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 6.419086] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 6.426554] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.434247] intel_idle: MWAIT substates: 0x21120 [ 6.434251] intel_idle: v0.4.1 model 0x2D [ 6.434586] intel_idle: lapic_timer_reliable_states 0xffffffff [ 6.434996] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.443276] ACPI: Power Button [PWRF] [ 6.447463] ACPI: Requesting acpi_cpufreq [ 6.534770] ERST: Error Record Serialization Table (ERST) support is initialized. [ 6.543176] pstore: Registered erst as persistent store backend [ 6.550142] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 6.558625] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.586491] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 6.613626] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 6.621013] Non-volatile memory driver v1.3 [ 6.625776] Linux agpgart interface v0.103 [ 6.630976] crash memory driver: version 1.1 [ 6.636519] rdac: device handler registered [ 6.641283] hp_sw: device handler registered [ 6.646083] emc: device handler registered [ 6.650908] alua: device handler registered [ 6.655789] libphy: Fixed MDIO Bus: probed [ 6.660469] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 6.667793] ehci-pci: EHCI PCI platform driver [ 6.673259] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 6.679225] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 6.687525] ehci-pci 0000:00:1a.0: debug port 2 [ 6.696546] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 6.696579] ehci-pci 0000:00:1a.0: irq 22, io mem 0xd0f20000 [ 6.707985] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 6.714523] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 6.723761] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 6.731844] usb usb1: Product: EHCI Host Controller [ 6.737302] usb usb1: Manufacturer: Linux 3.10.0-1160.49.1.el7_lustre.x86_64 ehci_hcd [ 6.746051] usb usb1: SerialNumber: 0000:00:1a.0 [ 6.751478] hub 1-0:1.0: USB hub found [ 6.755697] hub 1-0:1.0: 2 ports detected [ 6.760745] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 6.766707] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 6.774994] ehci-pci 0000:00:1d.0: debug port 2 [ 6.783979] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 6.784019] ehci-pci 0000:00:1d.0: irq 20, io mem 0xd0f10000 [ 6.795987] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 6.802521] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 6.811757] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 6.819841] usb usb2: Product: EHCI Host Controller [ 6.825306] usb usb2: Manufacturer: Linux 3.10.0-1160.49.1.el7_lustre.x86_64 ehci_hcd [ 6.834060] usb usb2: SerialNumber: 0000:00:1d.0 [ 6.839446] hub 2-0:1.0: USB hub found [ 6.843647] hub 2-0:1.0: 2 ports detected [ 6.848414] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 6.855348] ohci-pci: OHCI PCI platform driver [ 6.860368] uhci_hcd: USB Universal Host Controller Interface driver [ 6.867657] usbcore: registered new interface driver usbserial_generic [ 6.874986] usbserial: USB Serial support registered for generic [ 6.881792] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 7.052024] tsc: Refined TSC clocksource calibration: 2693.509 MHz [ 7.934685] i8042: No controller found [ 7.938952] Switched to clocksource tsc [ 7.939046] mousedev: PS/2 mouse device common for all mice [ 7.939232] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 7.939315] rtc_cmos 00:01: RTC can wake from S4 [ 7.939564] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 7.939620] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 7.939764] intel_pstate: Intel P-state driver initializing [ 7.951610] cpuidle: using governor menu [ 7.953158] hidraw: raw HID events driver (C) Jiri Kosina [ 7.953386] usbcore: registered new interface driver usbhid [ 7.953387] usbhid: USB HID core driver [ 7.953667] drop_monitor: Initializing network drop monitor service [ 7.953980] Netfilter messages via NETLINK v0.30. [ 7.954120] TCP: cubic registered [ 7.954127] Initializing XFRM netlink socket [ 7.954551] NET: Registered protocol family 10 [ 7.955791] NET: Registered protocol family 17 [ 7.955803] mpls_gso: MPLS GSO support [ 7.957534] mce: Using 20 MCE banks [ 7.957646] microcode: sig=0x206d7, pf=0x1, revision=0x71a [ 7.962262] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 7.962503] PM: Hibernation image not present or could not be loaded. [ 7.962510] Loading compiled-in X.509 certificates [ 7.962564] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 7.962591] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 7.963673] Loaded X.509 cert 'CentOS Linux kernel signing key: cf7384fb57402fbf0fb223e870f29b5564305c60' [ 7.963714] registered taskstats version 1 [ 7.963758] page_owner is disabled [ 7.968895] Key type trusted registered [ 7.973446] Key type encrypted registered [ 7.973549] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 7.974366] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 7.974508] Magic number: 6:87:512 [ 7.977299] rtc_cmos 00:01: setting system clock to 2022-05-04 03:32:19 UTC (1651635139) [ 7.990130] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 8.064541] usb 1-1: New USB device found, idVendor=8087, idProduct=0024, bcdDevice= 0.00 [ 8.064545] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 8.064945] hub 1-1:1.0: USB hub found [ 8.065154] hub 1-1:1.0: 6 ports detected [ 8.102374] random: fast init done [ 8.114533] usb 2-1: New USB device found, idVendor=8087, idProduct=0024, bcdDevice= 0.00 [ 8.114537] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 8.115112] hub 2-1:1.0: USB hub found [ 8.115284] hub 2-1:1.0: 8 ports detected [ 8.211064] Freeing unused kernel memory: 1980k freed [ 8.217566] Write protecting the kernel read-only data: 12288k [ 8.226530] Freeing unused kernel memory: 384k freed [ 8.234801] Freeing unused kernel memory: 532k freed [ 8.253365] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 8.273819] systemd[1]: Detected architecture x86-64. [ 8.279474] systemd[1]: Running in initial RAM disk. [ 8.295335] systemd[1]: Set hostname to . [ 8.390149] usb 2-1.4: new full-speed USB device number 3 using ehci-pci [ 8.399720] systemd[1]: Reached target Swap. [ 8.409297] systemd[1]: Reached target Timers. [ 8.419313] systemd[1]: Reached target Local File Systems. [ 8.432634] systemd[1]: Created slice Root Slice. [ 8.443377] systemd[1]: Listening on udev Control Socket. [ 8.456386] systemd[1]: Listening on Journal Socket. [ 8.467341] systemd[1]: Listening on udev Kernel Socket. [ 8.480281] systemd[1]: Reached target Sockets. [ 8.490415] systemd[1]: Created slice System Slice. [ 8.502185] systemd[1]: Starting Journal Service... [ 8.509318] usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10, bcdDevice= 1.00 [ 8.518757] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 8.528453] usb 2-1.4: Product: Virtual Keyboard and Mouse [ 8.535125] usb 2-1.4: Manufacturer: American Megatrends Inc. [ 8.541537] usb 2-1.4: SerialNumber: serial [ 8.547154] systemd[1]: Starting Setup Virtual Console... [ 8.547757] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input1 [ 8.572282] systemd[1]: Reached target Slices. [ 8.593327] systemd[1]: Starting dracut cmdline hook... [ 8.599690] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0 [ 8.618927] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input2 [ 8.634280] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1 [ 8.651894] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 8.672981] systemd[1]: Starting Apply Kernel Variables... [ 8.684686] systemd[1]: Started Journal Service. [ 8.899082] mlx_compat: loading out-of-tree module taints kernel. [ 8.908129] mlx_compat: module verification failed: signature and/or required key missing - tainting kernel [ 8.921198] dca service started, version 1.12.1 [ 8.928206] libata version 3.00 loaded. [ 8.928694] Compat-mlnx-ofed backport release: ab010e5 [ 8.935686] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git ab010e5 [ 8.943080] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 8.951869] pps_core: LinuxPPS API ver. 1 registered [ 8.957416] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 8.971090] PTP clock support registered [ 8.977125] isci: Intel(R) C600 SAS Controller Driver - version 1.2.0 [ 8.977134] mpt2sas version 20.103.01.00 loaded [ 8.977159] isci 0000:07:00.0: driver configured for rev: 6 silicon [ 8.977173] resource sanity check: requesting [mem 0x000ce000-0x000d7bff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000d3fff window] [ 8.977178] caller pci_map_biosrom+0x26/0x40 mapping multiple BARs [ 8.977184] isci 0000:07:00.0: OEM parameter table found in OROM [ 8.977187] isci 0000:07:00.0: OEM SAS parameters (version: 1.1) loaded (platform) [ 8.977244] mpt2sas 0000:06:00.0: can't disable ASPM; OS doesn't have ASPM control [ 8.977382] isci 0000:07:00.0: SCU controller 0: phy 3-0 cables: {short, short, short, short} [ 8.977421] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32680144 kB) [ 8.980120] scsi host1: isci [ 8.980678] isci 0000:07:00.0: irq 39 for MSI/MSI-X [ 8.980708] isci 0000:07:00.0: irq 40 for MSI/MSI-X [ 9.031715] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 9.031722] mpt2sas_cm0: MSI-X vectors supported: 16 [ 9.031723] no of cores: 32, max_msix_vectors: -1 [ 9.031724] mpt2sas_cm0: 0 16 [ 9.031783] mpt2sas 0000:06:00.0: irq 41 for MSI/MSI-X [ 9.031800] mpt2sas 0000:06:00.0: irq 42 for MSI/MSI-X [ 9.031817] mpt2sas 0000:06:00.0: irq 43 for MSI/MSI-X [ 9.031834] mpt2sas 0000:06:00.0: irq 44 for MSI/MSI-X [ 9.031851] mpt2sas 0000:06:00.0: irq 45 for MSI/MSI-X [ 9.031868] mpt2sas 0000:06:00.0: irq 46 for MSI/MSI-X [ 9.031884] mpt2sas 0000:06:00.0: irq 47 for MSI/MSI-X [ 9.031913] mpt2sas 0000:06:00.0: irq 48 for MSI/MSI-X [ 9.031939] mpt2sas 0000:06:00.0: irq 49 for MSI/MSI-X [ 9.031957] mpt2sas 0000:06:00.0: irq 50 for MSI/MSI-X [ 9.031976] mpt2sas 0000:06:00.0: irq 51 for MSI/MSI-X [ 9.031996] mpt2sas 0000:06:00.0: irq 52 for MSI/MSI-X [ 9.032014] mpt2sas 0000:06:00.0: irq 53 for MSI/MSI-X [ 9.032033] mpt2sas 0000:06:00.0: irq 54 for MSI/MSI-X [ 9.032051] mpt2sas 0000:06:00.0: irq 55 for MSI/MSI-X [ 9.032069] mpt2sas 0000:06:00.0: irq 56 for MSI/MSI-X [ 9.032200] mpt2sas_cm0: High IOPs queues : disabled [ 9.032201] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 41 [ 9.032202] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 42 [ 9.032202] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 43 [ 9.032203] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 44 [ 9.032204] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 45 [ 9.032204] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 46 [ 9.032205] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 47 [ 9.032206] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 48 [ 9.032206] mpt2sas0-msix8: PCI-MSI-X enabled: IRQ 49 [ 9.032207] mpt2sas0-msix9: PCI-MSI-X enabled: IRQ 50 [ 9.032208] mpt2sas0-msix10: PCI-MSI-X enabled: IRQ 51 [ 9.032208] mpt2sas0-msix11: PCI-MSI-X enabled: IRQ 52 [ 9.032209] mpt2sas0-msix12: PCI-MSI-X enabled: IRQ 53 [ 9.032209] mpt2sas0-msix13: PCI-MSI-X enabled: IRQ 54 [ 9.032210] mpt2sas0-msix14: PCI-MSI-X enabled: IRQ 55 [ 9.032211] mpt2sas0-msix15: PCI-MSI-X enabled: IRQ 56 [ 9.032212] mpt2sas_cm0: iomem(0x00000000d0a40000), mapped(0xffffa92643c60000), size(65536) [ 9.032213] mpt2sas_cm0: ioport(0x0000000000002000), size(256) [ 9.055564] ahci 0000:00:1f.2: version 3.0 [ 9.055782] ahci 0000:00:1f.2: irq 57 for MSI/MSI-X [ 9.066264] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 9.066267] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst [ 9.076842] scsi host2: ahci [ 9.076951] scsi host3: ahci [ 9.077029] scsi host4: ahci [ 9.077104] scsi host5: ahci [ 9.077181] scsi host6: ahci [ 9.077273] scsi host7: ahci [ 9.077317] ata1: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00100 irq 57 [ 9.077318] ata2: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00180 irq 57 [ 9.077321] ata3: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00200 irq 57 [ 9.077324] ata4: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00280 irq 57 [ 9.077327] ata5: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00300 irq 57 [ 9.077330] ata6: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00380 irq 57 [ 9.086714] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 9.086723] mpt2sas_cm0: sending message unit reset !! [ 9.088198] mpt2sas_cm0: message unit reset: SUCCESS [ 9.144995] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k [ 9.144996] igb: Copyright (c) 2007-2014 Intel Corporation. [ 9.206847] mpt2sas_cm0: Allocated physical memory: size(7454 kB) [ 9.206848] mpt2sas_cm0: Current Controller Queue Depth(10104),Max Controller Queue Depth(10240) [ 9.206849] mpt2sas_cm0: Scatter Gather Elements per IO(128) [ 9.208107] igb 0000:02:00.0: irq 59 for MSI/MSI-X [ 9.208160] igb 0000:02:00.0: irq 59 for MSI/MSI-X [ 9.208177] igb 0000:02:00.0: irq 60 for MSI/MSI-X [ 9.208195] igb 0000:02:00.0: irq 61 for MSI/MSI-X [ 9.208231] igb 0000:02:00.0: irq 62 for MSI/MSI-X [ 9.208256] igb 0000:02:00.0: irq 63 for MSI/MSI-X [ 9.208274] igb 0000:02:00.0: irq 64 for MSI/MSI-X [ 9.208291] igb 0000:02:00.0: irq 65 for MSI/MSI-X [ 9.208317] igb 0000:02:00.0: irq 66 for MSI/MSI-X [ 9.208334] igb 0000:02:00.0: irq 67 for MSI/MSI-X [ 9.208366] igb 0000:02:00.0: PHY reset is blocked due to SOL/IDER session. [ 9.208378] mlx4_core: Mellanox ConnectX core driver v4.9-4.1.7 [ 9.208422] mlx4_core: Initializing 0000:04:00.0 [ 9.263813] igb 0000:02:00.0: added PHC on eth0 [ 9.263814] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection [ 9.263816] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:24:6b [ 9.263889] igb 0000:02:00.0: eth0: PBA No: 100000-000 [ 9.263890] igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.264250] igb 0000:02:00.1: irq 70 for MSI/MSI-X [ 9.264293] igb 0000:02:00.1: irq 70 for MSI/MSI-X [ 9.264310] igb 0000:02:00.1: irq 71 for MSI/MSI-X [ 9.264327] igb 0000:02:00.1: irq 72 for MSI/MSI-X [ 9.264344] igb 0000:02:00.1: irq 73 for MSI/MSI-X [ 9.264361] igb 0000:02:00.1: irq 74 for MSI/MSI-X [ 9.264390] igb 0000:02:00.1: irq 75 for MSI/MSI-X [ 9.264406] igb 0000:02:00.1: irq 76 for MSI/MSI-X [ 9.264423] igb 0000:02:00.1: irq 77 for MSI/MSI-X [ 9.264439] igb 0000:02:00.1: irq 78 for MSI/MSI-X [ 9.264467] igb 0000:02:00.1: PHY reset is blocked due to SOL/IDER session. [ 9.276683] mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.29.00.00) [ 9.276686] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 9.276775] scsi host0: Fusion MPT SAS Host [ 9.277121] mpt2sas_cm0: sending port enable !! [ 9.280035] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b005d6e9a0), phys(8) [ 9.287222] mpt2sas_cm0: port enable: SUCCESS [ 9.288023] scsi 0:0:0:0: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.288028] scsi 0:0:0:0: SSP: handle(0x000a), sas_addr(0x50080e52ffd82000), phy(4), device_name(0x50080e52ffd82000) [ 9.288030] scsi 0:0:0:0: enclosure logical id (0x500605b005d6e9a0), slot(7) [ 9.288033] scsi 0:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.318968] igb 0000:02:00.1: added PHC on eth1 [ 9.318969] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection [ 9.318971] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:24:6c [ 9.319044] igb 0000:02:00.1: eth1: PBA No: 100000-000 [ 9.319046] igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.333719] igb 0000:02:00.2: irq 80 for MSI/MSI-X [ 9.333763] igb 0000:02:00.2: irq 80 for MSI/MSI-X [ 9.333781] igb 0000:02:00.2: irq 81 for MSI/MSI-X [ 9.333797] igb 0000:02:00.2: irq 82 for MSI/MSI-X [ 9.333813] igb 0000:02:00.2: irq 83 for MSI/MSI-X [ 9.333830] igb 0000:02:00.2: irq 84 for MSI/MSI-X [ 9.333848] igb 0000:02:00.2: irq 85 for MSI/MSI-X [ 9.333878] igb 0000:02:00.2: irq 86 for MSI/MSI-X [ 9.333895] igb 0000:02:00.2: irq 87 for MSI/MSI-X [ 9.333912] igb 0000:02:00.2: irq 88 for MSI/MSI-X [ 9.333961] [TTM] Zone kernel: Available graphics memory: 16340072 kiB [ 9.333962] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 9.333963] [TTM] Initializing pool allocator [ 9.333968] [TTM] Initializing DMA pool allocator [ 9.379447] fbcon: mgadrmfb (fb0) is primary device [ 9.382261] ata2: SATA link down (SStatus 0 SControl 300) [ 9.382317] ata6: SATA link down (SStatus 0 SControl 300) [ 9.382367] ata3: SATA link down (SStatus 0 SControl 300) [ 9.382416] ata1: SATA link down (SStatus 0 SControl 300) [ 9.382454] ata4: SATA link down (SStatus 0 SControl 300) [ 9.382502] ata5: SATA link down (SStatus 0 SControl 300) [ 9.390324] igb 0000:02:00.2: added PHC on eth2 [ 9.390326] igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection [ 9.390327] igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:24:6d [ 9.390400] igb 0000:02:00.2: eth2: PBA No: 100000-000 [ 9.390402] igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.390871] igb 0000:02:00.3: irq 90 for MSI/MSI-X [ 9.390915] igb 0000:02:00.3: irq 90 for MSI/MSI-X [ 9.390947] igb 0000:02:00.3: irq 91 for MSI/MSI-X [ 9.390963] igb 0000:02:00.3: irq 92 for MSI/MSI-X [ 9.390980] igb 0000:02:00.3: irq 93 for MSI/MSI-X [ 9.390996] igb 0000:02:00.3: irq 94 for MSI/MSI-X [ 9.391013] igb 0000:02:00.3: irq 95 for MSI/MSI-X [ 9.391031] igb 0000:02:00.3: irq 96 for MSI/MSI-X [ 9.391049] igb 0000:02:00.3: irq 97 for MSI/MSI-X [ 9.391066] igb 0000:02:00.3: irq 98 for MSI/MSI-X [ 9.402344] scsi 0:0:0:1: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.402349] scsi 0:0:0:1: SSP: handle(0x000a), sas_addr(0x50080e52ffd82000), phy(4), device_name(0x50080e52ffd82000) [ 9.402350] scsi 0:0:0:1: enclosure logical id (0x500605b005d6e9a0), slot(7) [ 9.402353] scsi 0:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.402641] scsi 0:0:0:1: Power-on or device reset occurred [ 9.426690] scsi 0:0:0:2: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.426695] scsi 0:0:0:2: SSP: handle(0x000a), sas_addr(0x50080e52ffd82000), phy(4), device_name(0x50080e52ffd82000) [ 9.426696] scsi 0:0:0:2: enclosure logical id (0x500605b005d6e9a0), slot(7) [ 9.426699] scsi 0:0:0:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.426870] scsi 0:0:0:2: Power-on or device reset occurred [ 9.438676] scsi 0:0:0:3: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.438694] scsi 0:0:0:3: SSP: handle(0x000a), sas_addr(0x50080e52ffd82000), phy(4), device_name(0x50080e52ffd82000) [ 9.438697] scsi 0:0:0:3: enclosure logical id (0x500605b005d6e9a0), slot(7) [ 9.438700] scsi 0:0:0:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.438880] scsi 0:0:0:3: Power-on or device reset occurred [ 9.447558] igb 0000:02:00.3: added PHC on eth3 [ 9.447559] igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection [ 9.447561] igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:24:6e [ 9.447634] igb 0000:02:00.3: eth3: PBA No: 100000-000 [ 9.447636] igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.459615] scsi 0:0:0:7: Direct-Access LSI Universal Xport 0820 PQ: 0 ANSI: 5 [ 9.459620] scsi 0:0:0:7: SSP: handle(0x000a), sas_addr(0x50080e52ffd82000), phy(4), device_name(0x50080e52ffd82000) [ 9.459621] scsi 0:0:0:7: enclosure logical id (0x500605b005d6e9a0), slot(7) [ 9.459624] scsi 0:0:0:7: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.459826] scsi 0:0:0:7: Power-on or device reset occurred [ 9.520256] scsi 0:0:1:0: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.520260] scsi 0:0:1:0: SSP: handle(0x0009), sas_addr(0x50080e52ff4f0004), phy(0), device_name(0x50080e52ff4f0004) [ 9.520262] scsi 0:0:1:0: enclosure logical id (0x500605b005d6e9a0), slot(3) [ 9.520264] scsi 0:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.535345] scsi 0:0:1:1: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.535358] scsi 0:0:1:1: SSP: handle(0x0009), sas_addr(0x50080e52ff4f0004), phy(0), device_name(0x50080e52ff4f0004) [ 9.535360] scsi 0:0:1:1: enclosure logical id (0x500605b005d6e9a0), slot(3) [ 9.535363] scsi 0:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.535645] scsi 0:0:1:1: Power-on or device reset occurred [ 9.551706] scsi 0:0:1:2: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.551725] scsi 0:0:1:2: SSP: handle(0x0009), sas_addr(0x50080e52ff4f0004), phy(0), device_name(0x50080e52ff4f0004) [ 9.551727] scsi 0:0:1:2: enclosure logical id (0x500605b005d6e9a0), slot(3) [ 9.551730] scsi 0:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.551913] scsi 0:0:1:2: Power-on or device reset occurred [ 9.571858] scsi 0:0:1:3: Direct-Access LSI INF-01-00 0820 PQ: 0 ANSI: 5 [ 9.571865] scsi 0:0:1:3: SSP: handle(0x0009), sas_addr(0x50080e52ff4f0004), phy(0), device_name(0x50080e52ff4f0004) [ 9.571868] scsi 0:0:1:3: enclosure logical id (0x500605b005d6e9a0), slot(3) [ 9.571872] scsi 0:0:1:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.572129] scsi 0:0:1:3: Power-on or device reset occurred [ 9.589836] scsi 0:0:1:7: Direct-Access LSI Universal Xport 0820 PQ: 0 ANSI: 5 [ 9.589844] scsi 0:0:1:7: SSP: handle(0x0009), sas_addr(0x50080e52ff4f0004), phy(0), device_name(0x50080e52ff4f0004) [ 9.589847] scsi 0:0:1:7: enclosure logical id (0x500605b005d6e9a0), slot(3) [ 9.589852] scsi 0:0:1:7: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.590087] scsi 0:0:1:7: Power-on or device reset occurred [ 9.612556] Console: switching to colour frame buffer device 128x48 [ 9.662818] mgag200 0000:09:00.0: fb0: mgadrmfb frame buffer device [ 9.678342] [drm] Initialized mgag200 1.0.0 20110418 for 0000:09:00.0 on minor 0 [ 10.492432] sas: phy-1:0 added to port-1:0, phy_mask:0x1 (5001e676b246b000) [ 10.492494] sas: phy-1:1 added to port-1:1, phy_mask:0x2 (5001e676b246b001) [ 10.492516] sas: DOING DISCOVERY on port 0, pid:229 [ 10.492607] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 10.492669] sas: ata7: end_device-1:0: dev error handler [ 10.653310] ata7.00: ATA-8: ST500NM0011, SN02, max UDMA/133 [ 10.659560] ata7.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 10.667839] ata7.00: configured for UDMA/133 [ 10.672681] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 10.683878] scsi 1:0:0:0: Direct-Access ATA ST500NM0011 SN02 PQ: 0 ANSI: 5 [ 15.005226] mlx4_core: device is working in RoCE mode: Roce V1 [ 15.011738] mlx4_core: UD QP Gid type is: V1 [ 16.651459] mlx4_core 0000:04:00.0: DMFS high rate steer mode is: default performance [ 16.660475] mlx4_core 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link) [ 16.670147] mlx4_core 0000:04:00.0: irq 99 for MSI/MSI-X [ 16.670164] mlx4_core 0000:04:00.0: irq 100 for MSI/MSI-X [ 16.670179] mlx4_core 0000:04:00.0: irq 101 for MSI/MSI-X [ 16.670194] mlx4_core 0000:04:00.0: irq 102 for MSI/MSI-X [ 16.670209] mlx4_core 0000:04:00.0: irq 103 for MSI/MSI-X [ 16.670223] mlx4_core 0000:04:00.0: irq 104 for MSI/MSI-X [ 16.670238] mlx4_core 0000:04:00.0: irq 105 for MSI/MSI-X [ 16.670253] mlx4_core 0000:04:00.0: irq 106 for MSI/MSI-X [ 16.670268] mlx4_core 0000:04:00.0: irq 107 for MSI/MSI-X [ 16.670283] mlx4_core 0000:04:00.0: irq 108 for MSI/MSI-X [ 16.670298] mlx4_core 0000:04:00.0: irq 109 for MSI/MSI-X [ 16.670313] mlx4_core 0000:04:00.0: irq 110 for MSI/MSI-X [ 16.670327] mlx4_core 0000:04:00.0: irq 111 for MSI/MSI-X [ 16.670342] mlx4_core 0000:04:00.0: irq 112 for MSI/MSI-X [ 16.670371] mlx4_core 0000:04:00.0: irq 113 for MSI/MSI-X [ 16.670385] mlx4_core 0000:04:00.0: irq 114 for MSI/MSI-X [ 16.670400] mlx4_core 0000:04:00.0: irq 115 for MSI/MSI-X [ 16.670415] mlx4_core 0000:04:00.0: irq 116 for MSI/MSI-X [ 16.670430] mlx4_core 0000:04:00.0: irq 117 for MSI/MSI-X [ 16.670445] mlx4_core 0000:04:00.0: irq 118 for MSI/MSI-X [ 16.670459] mlx4_core 0000:04:00.0: irq 119 for MSI/MSI-X [ 16.670474] mlx4_core 0000:04:00.0: irq 120 for MSI/MSI-X [ 16.670488] mlx4_core 0000:04:00.0: irq 121 for MSI/MSI-X [ 16.670503] mlx4_core 0000:04:00.0: irq 122 for MSI/MSI-X [ 16.670518] mlx4_core 0000:04:00.0: irq 123 for MSI/MSI-X [ 16.670533] mlx4_core 0000:04:00.0: irq 124 for MSI/MSI-X [ 16.670548] mlx4_core 0000:04:00.0: irq 125 for MSI/MSI-X [ 16.670562] mlx4_core 0000:04:00.0: irq 126 for MSI/MSI-X [ 16.670577] mlx4_core 0000:04:00.0: irq 127 for MSI/MSI-X [ 16.670592] mlx4_core 0000:04:00.0: irq 128 for MSI/MSI-X [ 16.670621] mlx4_core 0000:04:00.0: irq 129 for MSI/MSI-X [ 16.670635] mlx4_core 0000:04:00.0: irq 130 for MSI/MSI-X [ 16.670650] mlx4_core 0000:04:00.0: irq 131 for MSI/MSI-X [ 16.670666] mlx4_core 0000:04:00.0: irq 132 for MSI/MSI-X [ 16.670680] mlx4_core 0000:04:00.0: irq 133 for MSI/MSI-X [ 16.670695] mlx4_core 0000:04:00.0: irq 134 for MSI/MSI-X [ 16.670710] mlx4_core 0000:04:00.0: irq 135 for MSI/MSI-X [ 16.670724] mlx4_core 0000:04:00.0: irq 136 for MSI/MSI-X [ 16.670739] mlx4_core 0000:04:00.0: irq 137 for MSI/MSI-X [ 16.670753] mlx4_core 0000:04:00.0: irq 138 for MSI/MSI-X [ 16.670768] mlx4_core 0000:04:00.0: irq 139 for MSI/MSI-X [ 16.670784] mlx4_core 0000:04:00.0: irq 140 for MSI/MSI-X [ 16.670799] mlx4_core 0000:04:00.0: irq 141 for MSI/MSI-X [ 16.670813] mlx4_core 0000:04:00.0: irq 142 for MSI/MSI-X [ 16.670828] mlx4_core 0000:04:00.0: irq 143 for MSI/MSI-X [ 16.670843] mlx4_core 0000:04:00.0: irq 144 for MSI/MSI-X [ 16.670871] mlx4_core 0000:04:00.0: irq 145 for MSI/MSI-X [ 16.670885] mlx4_core 0000:04:00.0: irq 146 for MSI/MSI-X [ 16.670900] mlx4_core 0000:04:00.0: irq 147 for MSI/MSI-X [ 16.670921] mlx4_core 0000:04:00.0: irq 148 for MSI/MSI-X [ 16.670936] mlx4_core 0000:04:00.0: irq 149 for MSI/MSI-X [ 16.670951] mlx4_core 0000:04:00.0: irq 150 for MSI/MSI-X [ 16.670966] mlx4_core 0000:04:00.0: irq 151 for MSI/MSI-X [ 16.670981] mlx4_core 0000:04:00.0: irq 152 for MSI/MSI-X [ 16.670995] mlx4_core 0000:04:00.0: irq 153 for MSI/MSI-X [ 16.671010] mlx4_core 0000:04:00.0: irq 154 for MSI/MSI-X [ 16.671025] mlx4_core 0000:04:00.0: irq 155 for MSI/MSI-X [ 16.671040] mlx4_core 0000:04:00.0: irq 156 for MSI/MSI-X [ 16.671055] mlx4_core 0000:04:00.0: irq 157 for MSI/MSI-X [ 16.671069] mlx4_core 0000:04:00.0: irq 158 for MSI/MSI-X [ 16.671084] mlx4_core 0000:04:00.0: irq 159 for MSI/MSI-X [ 16.671099] mlx4_core 0000:04:00.0: irq 160 for MSI/MSI-X [ 16.671127] mlx4_core 0000:04:00.0: irq 161 for MSI/MSI-X [ 16.671143] mlx4_core 0000:04:00.0: irq 162 for MSI/MSI-X [ 16.671158] mlx4_core 0000:04:00.0: irq 163 for MSI/MSI-X [ 16.928957] sas: DONE DISCOVERY on port 0, pid:229, result:0 [ 16.928991] sas: DOING DISCOVERY on port 1, pid:229 [ 16.929102] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 16.929125] sas: ata7: end_device-1:0: dev error handler [ 16.929178] sas: ata8: end_device-1:1: dev error handler [ 16.933744] mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.9-4.1.7 [ 16.944120] mlx4_ib_add: counter index 0 for port 1 allocated 0 [ 16.951706] mlx4_ib_add: counter index 1 for port 2 allocated 0 [ 17.089869] ata8.00: ATA-8: ST500NM0011, SN02, max UDMA/133 [ 17.096116] ata8.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 17.104516] ata8.00: configured for UDMA/133 [ 17.109376] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 17.120685] scsi 1:0:1:0: Direct-Access ATA ST500NM0011 SN02 PQ: 0 ANSI: 5 [ 17.139970] sas: DONE DISCOVERY on port 1, pid:229, result:0 [ 17.140403] sd 1:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 17.141033] sd 1:0:1:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 17.141380] sd 1:0:1:0: [sdb] Write Protect is off [ 17.141384] sd 1:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [ 17.141582] sd 1:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 17.141878] scsi 0:0:0:0: rdac: LUN 0 (RDAC) (owned) [ 17.142430] sd 0:0:0:0: [sdc] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.142866] scsi 0:0:0:1: rdac: LUN 1 (RDAC) (unowned) [ 17.143320] sd 0:0:0:0: [sdc] Write Protect is off [ 17.143338] sd 0:0:0:0: [sdc] Mode Sense: 83 00 10 08 [ 17.143423] sd 0:0:0:1: [sdd] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.143764] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.144266] scsi 0:0:0:2: rdac: LUN 2 (RDAC) (owned) [ 17.144601] sd 0:0:0:1: [sdd] Write Protect is off [ 17.144604] sd 0:0:0:1: [sdd] Mode Sense: 83 00 10 08 [ 17.144784] sd 0:0:0:2: [sde] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.144934] sd 0:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.145711] scsi 0:0:0:3: rdac: LUN 3 (RDAC) (unowned) [ 17.146322] sd 0:0:0:2: [sde] Write Protect is off [ 17.146326] sd 0:0:0:2: [sde] Mode Sense: 83 00 10 08 [ 17.146565] sd 0:0:0:3: [sdf] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.146897] sd 0:0:0:2: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.146949] scsi 0:0:1:0: rdac: LUN 0 (RDAC) (unowned) [ 17.147364] sd 0:0:1:0: [sdg] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.147958] scsi 0:0:1:1: rdac: LUN 1 (RDAC) (owned) [ 17.148101] sd 0:0:0:3: [sdf] Write Protect is off [ 17.148104] sd 0:0:0:3: [sdf] Mode Sense: 83 00 10 08 [ 17.148227] sd 0:0:1:0: [sdg] Write Protect is off [ 17.148238] sd 0:0:1:0: [sdg] Mode Sense: 83 00 10 08 [ 17.148469] Dev sdd: unable to read RDB block 0 [ 17.148480] sd 0:0:0:3: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.148496] sdd: unable to read partition table [ 17.148566] sd 0:0:1:1: [sdh] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.148571] sd 0:0:1:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.149315] scsi 0:0:1:2: rdac: LUN 2 (RDAC) (unowned) [ 17.149654] sd 0:0:1:1: [sdh] Write Protect is off [ 17.149659] sd 0:0:1:1: [sdh] Mode Sense: 83 00 10 08 [ 17.149981] sd 0:0:1:1: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.149984] sd 0:0:1:2: [sdi] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.150384] sdb: [ 17.150718] sd 0:0:0:1: [sdd] Attached SCSI disk [ 17.151118] sd 0:0:1:2: [sdi] Write Protect is off [ 17.151121] sd 0:0:1:2: [sdi] Mode Sense: 83 00 10 08 [ 17.151163] sd 1:0:1:0: [sdb] Attached SCSI disk [ 17.151224] Dev sdf: unable to read RDB block 0 [ 17.151237] sdf: unable to read partition table [ 17.151511] scsi 0:0:1:3: rdac: LUN 3 (RDAC) (owned) [ 17.151676] sd 0:0:1:2: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.152105] Dev sdg: unable to read RDB block 0 [ 17.152119] sdg: unable to read partition table [ 17.152215] sd 0:0:1:3: [sdj] 6986547200 512-byte logical blocks: (3.57 TB/3.25 TiB) [ 17.152500] sd 0:0:0:3: [sdf] Attached SCSI disk [ 17.153153] sd 0:0:1:3: [sdj] Write Protect is off [ 17.153156] sd 0:0:1:3: [sdj] Mode Sense: 83 00 10 08 [ 17.153595] sd 0:0:1:3: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.154783] sd 0:0:1:0: [sdg] Attached SCSI disk [ 17.155422] Dev sdi: unable to read RDB block 0 [ 17.155435] sdi: unable to read partition table [ 17.157075] sd 0:0:1:2: [sdi] Attached SCSI disk [ 17.157812] sd 0:0:1:3: [sdj] Attached SCSI disk [ 17.158702] sd 0:0:0:0: [sdc] Attached SCSI disk [ 17.161414] sd 0:0:0:2: [sde] Attached SCSI disk [ 17.231635] random: crng init done [ 17.494139] sd 1:0:0:0: [sda] Write Protect is off [ 17.499518] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 17.499595] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 17.531341] sda: sda1 sda2 [ 17.535191] sd 1:0:0:0: [sda] Attached SCSI disk [ 17.665713] sd 0:0:1:1: [sdh] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 17.676029] sd 0:0:1:1: [sdh] tag#2 Sense Key : Hardware Error [current] [ 17.683621] sd 0:0:1:1: [sdh] tag#2 <>ASC=0x84 ASCQ=0x0 [ 17.690332] sd 0:0:1:1: [sdh] tag#2 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 17.700241] blk_update_request: critical target error, dev sdh, sector 0 [ 17.707734] Buffer I/O error on dev sdh, logical block 0, async page read [ 18.215691] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 18.226006] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 18.233583] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 18.240294] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 18.250207] blk_update_request: critical target error, dev sdh, sector 0 [ 18.257692] Buffer I/O error on dev sdh, logical block 0, async page read [ 18.765739] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 18.776048] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 18.783641] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 18.790357] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 18.800271] blk_update_request: critical target error, dev sdh, sector 0 [ 18.807753] Buffer I/O error on dev sdh, logical block 0, async page read [ 19.315765] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 19.326075] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 19.333658] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 19.340374] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 19.350288] blk_update_request: critical target error, dev sdh, sector 0 [ 19.357787] Buffer I/O error on dev sdh, logical block 0, async page read [ 19.865789] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 19.876102] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 19.883686] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 19.890403] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 19.900313] blk_update_request: critical target error, dev sdh, sector 0 [ 19.907794] Buffer I/O error on dev sdh, logical block 0, async page read [ 20.415858] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 20.426167] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 20.433750] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 20.440467] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 20.450371] blk_update_request: critical target error, dev sdh, sector 0 [ 20.457855] Buffer I/O error on dev sdh, logical block 0, async page read [ 20.965876] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 20.976184] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 20.983777] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 20.990488] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 21.000403] blk_update_request: critical target error, dev sdh, sector 0 [ 21.007885] Buffer I/O error on dev sdh, logical block 0, async page read [ 21.015490] Dev sdh: unable to read RDB block 0 [ 21.532589] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 21.542899] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 21.550490] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 21.557201] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 21.567117] blk_update_request: critical target error, dev sdh, sector 0 [ 21.574607] Buffer I/O error on dev sdh, logical block 0, async page read [ 22.082614] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 22.092921] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 22.100512] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 22.107227] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 22.117137] blk_update_request: critical target error, dev sdh, sector 0 [ 22.124629] Buffer I/O error on dev sdh, logical block 0, async page read [ 22.132229] sdh: unable to read partition table [ 22.138789] sd 0:0:1:1: [sdh] Attached SCSI disk [ 22.649293] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 22.659611] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 22.667198] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 22.673918] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 22.683840] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 23.199324] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 23.209646] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 23.217232] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 23.223955] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 23.233864] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 23.242221] Buffer I/O error on dev sdh, logical block 873318384, async page read [ 23.483438] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 23.898428] systemd-journald[283]: Received SIGTERM from PID 1 (systemd). [ 24.282603] SELinux: Disabled at runtime. [ 24.287275] SELinux: Unregistering netfilter hooks [ 24.336699] type=1404 audit(1651635155.859:2): selinux=0 auid=4294967295 ses=4294967295 [ 24.592934] ip_tables: (C) 2000-2006 Netfilter Core Team [ 24.599645] systemd[1]: Inserted module 'ip_tables' [ 25.816476] RPC: Registered named UNIX socket transport module. [ 25.823644] RPC: Registered udp transport module. [ 25.828909] RPC: Registered tcp transport module. [ 25.834166] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 25.838986] EXT4-fs (sda1): re-mounted. Opts: (null) [ 26.005941] systemd-journald[672]: Received request to flush runtime journal from PID 1 [ 26.140153] device-mapper: uevent: version 1.0.3 [ 26.145495] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 27.410400] IPMI message handler: version 39.2 [ 27.427831] ipmi device interface [ 27.457205] ipmi_si: IPMI System Interface driver [ 27.462482] ipmi_si dmi-ipmi-si.0: probing via SMBIOS [ 27.468131] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 27.476208] ipmi_si: Adding SMBIOS-specified kcs state machine [ 27.482804] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 27.482807] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 27.506546] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt [ 27.513320] ioatdma: Intel(R) QuickData Technology Driver 4.00 [ 27.515280] i801_smbus 0000:07:00.3: Enabling SMBus device [ 27.515313] i801_smbus 0000:07:00.3: SMBus using PCI interrupt [ 27.532856] ioatdma 0000:00:04.0: irq 165 for MSI/MSI-X [ 27.533145] igb 0000:02:00.0: DCA enabled [ 27.537667] igb 0000:02:00.1: DCA enabled [ 27.542390] ioatdma 0000:00:04.1: irq 167 for MSI/MSI-X [ 27.542603] igb 0000:02:00.2: DCA enabled [ 27.549615] igb 0000:02:00.3: DCA enabled [ 27.554368] ioatdma 0000:00:04.2: irq 168 for MSI/MSI-X [ 27.554731] ioatdma 0000:00:04.3: irq 169 for MSI/MSI-X [ 27.555066] ioatdma 0000:00:04.4: irq 170 for MSI/MSI-X [ 27.555445] ioatdma 0000:00:04.5: irq 171 for MSI/MSI-X [ 27.555806] ioatdma 0000:00:04.6: irq 172 for MSI/MSI-X [ 27.556168] ioatdma 0000:00:04.7: irq 173 for MSI/MSI-X [ 27.556642] ioatdma 0000:80:04.0: irq 175 for MSI/MSI-X [ 27.557066] ioatdma 0000:80:04.1: irq 177 for MSI/MSI-X [ 27.557369] ioatdma 0000:80:04.2: irq 178 for MSI/MSI-X [ 27.557667] ioatdma 0000:80:04.3: irq 179 for MSI/MSI-X [ 27.557942] ioatdma 0000:80:04.4: irq 180 for MSI/MSI-X [ 27.558241] ioatdma 0000:80:04.5: irq 181 for MSI/MSI-X [ 27.558535] ioatdma 0000:80:04.6: irq 182 for MSI/MSI-X [ 27.558811] ioatdma 0000:80:04.7: irq 183 for MSI/MSI-X [ 27.571025] ipmi_si dmi-ipmi-si.0: Found new BMC (man_id: 0x000157, prod_id: 0x0049, dev_id: 0x21) [ 27.643896] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized [ 27.717006] ipmi_ssif: IPMI SSIF Interface driver [ 28.236663] sd 1:0:0:0: Attached scsi generic sg0 type 0 [ 28.242773] sd 1:0:1:0: Attached scsi generic sg1 type 0 [ 28.248784] sd 0:0:0:0: Attached scsi generic sg2 type 0 [ 28.254806] sd 0:0:0:1: Attached scsi generic sg3 type 0 [ 28.260813] sd 0:0:0:2: Attached scsi generic sg4 type 0 [ 28.266817] sd 0:0:0:3: Attached scsi generic sg5 type 0 [ 28.272862] scsi 0:0:0:7: Attached scsi generic sg6 type 0 [ 28.279160] sd 0:0:1:0: Attached scsi generic sg7 type 0 [ 28.285161] sd 0:0:1:1: Attached scsi generic sg8 type 0 [ 28.291181] sd 0:0:1:2: Attached scsi generic sg9 type 0 [ 28.297201] sd 0:0:1:3: Attached scsi generic sg10 type 0 [ 28.303308] scsi 0:0:1:7: Attached scsi generic sg11 type 0 [ 28.310616] acpi PNP0C14:00: duplicate WMI GUID 0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61 (first instance was on PNP0C14:00) [ 28.322665] acpi PNP0C14:00: duplicate WMI GUID 0E7AF9F2-44A1-4C6F-A4B0-A7678480DA61 (first instance was on PNP0C14:00) [ 28.343488] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 28.381158] cryptd: max_cpu_qlen set to 1000 [ 28.506816] device-mapper: multipath round-robin: version 1.2.0 loaded [ 28.551968] sd 0:0:1:0: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [ 28.630201] iTCO_vendor_support: vendor-support=0 [ 28.691150] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 28.697472] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 28.733239] AVX version of gcm_enc/dec engaged. [ 28.738302] AES CTR mode by8 optimization enabled [ 28.746587] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 28.753857] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 28.861013] sd 0:0:1:1: [sdh] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 28.871311] sd 0:0:1:1: [sdh] tag#2 Sense Key : Hardware Error [current] [ 28.878891] sd 0:0:1:1: [sdh] tag#2 <>ASC=0x84 ASCQ=0x0 [ 28.885598] sd 0:0:1:1: [sdh] tag#2 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 28.895509] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 28.995999] sd 0:0:1:0: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [ 29.004796] sd 0:0:0:1: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [ 29.057631] kvm: disabled by bios [ 29.071541] kvm: disabled by bios [ 29.082277] kvm: disabled by bios [ 29.093200] kvm: disabled by bios [ 29.103286] kvm: disabled by bios [ 29.113179] kvm: disabled by bios [ 29.126250] kvm: disabled by bios [ 29.136188] kvm: disabled by bios [ 29.146155] kvm: disabled by bios [ 29.159288] kvm: disabled by bios [ 29.173215] kvm: disabled by bios [ 29.183251] kvm: disabled by bios [ 29.187072] intel_rapl: Found RAPL domain package [ 29.192329] intel_rapl: Found RAPL domain core [ 29.197295] intel_rapl: Found RAPL domain dram [ 29.202304] intel_rapl: Found RAPL domain package [ 29.207579] intel_rapl: Found RAPL domain core [ 29.212561] intel_rapl: Found RAPL domain dram [ 29.224109] kvm: disabled by bios [ 29.236216] kvm: disabled by bios [ 29.249329] kvm: disabled by bios [ 29.259260] kvm: disabled by bios [ 29.269260] kvm: disabled by bios [ 29.282241] kvm: disabled by bios [ 29.292228] kvm: disabled by bios [ 29.302232] kvm: disabled by bios [ 29.315197] kvm: disabled by bios [ 29.325188] kvm: disabled by bios [ 29.336167] kvm: disabled by bios [ 29.346373] kvm: disabled by bios [ 29.358049] kvm: disabled by bios [ 29.370350] kvm: disabled by bios [ 29.382483] kvm: disabled by bios [ 29.393309] kvm: disabled by bios [ 29.403232] kvm: disabled by bios [ 29.404962] sd 0:0:1:1: [sdh] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 29.404966] sd 0:0:1:1: [sdh] tag#1 Sense Key : Hardware Error [current] [ 29.404970] sd 0:0:1:1: [sdh] tag#1 <>ASC=0x84 ASCQ=0x0 [ 29.404973] sd 0:0:1:1: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 29.404974] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 29.404977] Buffer I/O error on dev sdh, logical block 873318384, async page read [ 29.458164] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 29.458181] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 29.458189] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 29.458194] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 29.458212] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 29.458217] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 29.458220] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 29.458227] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 29.458232] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 29.458234] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 29.458241] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 29.458246] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 29.458248] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 29.458255] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 29.458260] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 29.458262] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 29.458269] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 29.458274] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 29.458276] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 29.458283] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 29.458288] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 29.458290] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 29.458298] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 29.458303] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 29.458305] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 29.458311] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 29.458316] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 29.458319] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 29.458325] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 29.458330] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 29.458333] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 29.458339] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 29.458344] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 29.458467] EDAC MC0: Giving out device to 'sb_edac.c' 'Sandy Bridge SrcID#0_Ha#0': DEV 0000:7f:0e.0 [ 29.468766] EDAC MC1: Giving out device to 'sb_edac.c' 'Sandy Bridge SrcID#1_Ha#0': DEV 0000:ff:0e.0 [ 29.478962] EDAC sbridge: Ver: 1.1.2 [ 29.489007] kvm: disabled by bios [ 29.500112] kvm: disabled by bios [ 29.512379] kvm: disabled by bios [ 30.030376] device-mapper: table: 253:5: multipath: error getting device [ 30.037865] device-mapper: ioctl: error adding target to table [ 30.282486] Adding 16319484k swap on /dev/sda2. Priority:-2 extents:1 across:16319484k FS [ 30.388437] sd 0:0:0:1: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [ 30.397124] sd 0:0:1:2: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [ 30.750622] sd 0:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 30.760953] sd 0:0:0:0: [sdc] tag#0 Sense Key : Illegal Request [current] [ 30.768644] sd 0:0:0:0: [sdc] tag#0 <>ASC=0x94 ASCQ=0x1 [ 30.775356] sd 0:0:0:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 [ 30.785278] blk_update_request: I/O error, dev sdc, sector 0 [ 30.838886] sd 0:0:1:2: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [ 30.892046] sd 0:0:0:2: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [ 30.908784] sd 0:0:0:1: [sdd] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 30.919103] sd 0:0:0:1: [sdd] tag#3 Sense Key : Hardware Error [current] [ 30.926690] sd 0:0:0:1: [sdd] tag#3 <>ASC=0x84 ASCQ=0x0 [ 30.933407] sd 0:0:0:1: [sdd] tag#3 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 30.943316] blk_update_request: critical target error, dev sdd, sector 6986547072 [ 30.951713] blk_update_request: critical target error, dev dm-1, sector 6986547072 [ 31.244998] sd 0:0:0:2: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [ 31.301427] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [ 31.311752] sd 0:0:1:1: [sdh] tag#0 Sense Key : Illegal Request [current] [ 31.319454] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x94 ASCQ=0x1 [ 31.326165] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 [ 31.336073] blk_update_request: I/O error, dev sdh, sector 0 [ 31.342454] sd 0:0:1:3: [sdj] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [ 31.352769] sd 0:0:1:3: [sdj] tag#1 Sense Key : Illegal Request [current] [ 31.360458] sd 0:0:1:3: [sdj] tag#1 <>ASC=0x94 ASCQ=0x1 [ 31.367174] sd 0:0:1:3: [sdj] tag#1 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 [ 31.377091] blk_update_request: I/O error, dev sdj, sector 0 [ 31.475492] sd 0:0:0:1: [sdd] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 31.485813] sd 0:0:0:1: [sdd] tag#3 Sense Key : Hardware Error [current] [ 31.493405] sd 0:0:0:1: [sdd] tag#3 <>ASC=0x84 ASCQ=0x0 [ 31.500125] sd 0:0:0:1: [sdd] tag#3 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 31.500129] blk_update_request: critical target error, dev sdd, sector 6986547072 [ 31.500154] blk_update_request: critical target error, dev dm-1, sector 6986547072 [ 31.500158] Buffer I/O error on dev dm-1, logical block 873318384, async page read [ 31.559443] type=1305 audit(1651635163.081:3): audit_pid=1067 old=0 auid=4294967295 ses=4294967295 res=1 [ 32.008796] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 32.019219] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 32.028330] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 32.036586] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [ 32.048042] blk_update_request: critical target error, dev sdd, sector 0 [ 32.558809] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 32.569870] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 32.569890] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 32.569894] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 32.569917] Buffer I/O error on dev dm-1, logical block 0, async page read [ 33.075473] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 33.085784] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 33.093366] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 33.100077] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [ 33.524439] IPv6: ADDRCONF(NETDEV_UP): ens2f0: link is not ready [ 33.545798] igb 0000:02:00.0 ens2f0: igb: ens2f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 33.556695] IPv6: ADDRCONF(NETDEV_UP): ens2f0: link is not ready [ 33.563440] IPv6: ADDRCONF(NETDEV_CHANGE): ens2f0: link becomes ready [ 33.576885] IPv6: ADDRCONF(NETDEV_UP): ens2f1: link is not ready [ 33.604752] IPv6: ADDRCONF(NETDEV_UP): ens2f1: link is not ready [ 33.616520] IPv6: ADDRCONF(NETDEV_UP): ens2f2: link is not ready [ 33.625645] Buffer I/O error on dev dm-1, logical block 873318399, async page read [ 33.675381] IPv6: ADDRCONF(NETDEV_UP): ens2f2: link is not ready [ 33.686973] IPv6: ADDRCONF(NETDEV_UP): ens2f3: link is not ready [ 33.745690] IPv6: ADDRCONF(NETDEV_UP): ens2f3: link is not ready [ 34.020996] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.9-4.1.7 [ 34.142205] scsi_io_completion: 1 callbacks suppressed [ 34.147954] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 34.158268] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 34.165853] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 34.172565] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 34.182490] blk_update_request: 7 callbacks suppressed [ 34.188231] blk_update_request: critical target error, dev sdd, sector 0 [ 34.196524] blk_update_request: critical target error, dev dm-1, sector 0 [ 34.204125] Buffer I/O error on dev dm-1, logical block 0, async page read [ 34.725600] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 34.735915] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 34.743502] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 34.750218] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 34.760127] blk_update_request: critical target error, dev sdd, sector 0 [ 34.767625] blk_update_request: critical target error, dev dm-1, sector 0 [ 34.775209] Buffer I/O error on dev dm-1, logical block 0, async page read [ 34.824798] card: mlx4_0, QP: 0x220, inline size: 120 [ 34.842592] card: mlx4_0, QP: 0x300, inline size: 120 [ 34.888136] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 34.906030] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 34.915038] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 34.931702] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 34.970690] ib0: enabling connected mode will cause multicast packet drops [ 34.978448] ib0: mtu > 4092 will cause multicast packet drops. [ 34.999174] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 35.292367] sd 0:0:0:1: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 35.302696] sd 0:0:0:1: [sdd] tag#0 Sense Key : Hardware Error [current] [ 35.310293] sd 0:0:0:1: [sdd] tag#0 <>ASC=0x84 ASCQ=0x0 [ 35.317004] sd 0:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [ 35.326918] blk_update_request: critical target error, dev sdd, sector 24 [ 35.334543] blk_update_request: critical target error, dev dm-1, sector 24 [ 35.342240] Buffer I/O error on dev dm-1, logical block 3, async page read [ 35.364641] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [ 35.656528] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [ 36.183659] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 36.193969] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 36.201577] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 36.208295] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 36.218202] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 36.226643] blk_update_request: critical target error, dev dm-1, sector 6986547072 [ 36.747121] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 36.757429] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 36.765014] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 36.771742] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 36.781652] blk_update_request: critical target error, dev sdh, sector 6986547072 [ 36.790029] blk_update_request: critical target error, dev dm-1, sector 6986547072 [ 36.798510] Buffer I/O error on dev dm-1, logical block 873318384, async page read [ 37.333737] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 37.344045] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 37.351638] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 37.358355] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 37.883718] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 37.894051] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 37.901635] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 37.908351] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [ 37.918686] Buffer I/O error on dev dm-1, logical block 873318384, async page read [ 38.433767] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 38.444076] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 38.451656] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 38.458371] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [ 38.983809] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 38.994127] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 39.001713] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 39.008436] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 39.018367] Buffer I/O error on dev dm-1, logical block 0, async page read [ 39.533815] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 39.544124] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 39.551712] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 39.558422] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [ 39.568328] blk_update_request: 8 callbacks suppressed [ 39.574079] blk_update_request: critical target error, dev sdh, sector 6986547192 [ 39.582761] blk_update_request: critical target error, dev dm-1, sector 6986547192 [ 39.854981] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [ 40.092800] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 40.103107] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 40.110700] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 40.117416] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [ 40.127331] blk_update_request: critical target error, dev sdh, sector 6986547192 [ 40.135714] blk_update_request: critical target error, dev dm-1, sector 6986547192 [ 40.139515] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [ 40.152824] Buffer I/O error on dev dm-1, logical block 873318399, async page read [ 40.667300] sd 0:0:1:1: [sdh] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 40.677612] sd 0:0:1:1: [sdh] tag#1 Sense Key : Hardware Error [current] [ 40.685205] sd 0:0:1:1: [sdh] tag#1 <>ASC=0x84 ASCQ=0x0 [ 40.691916] sd 0:0:1:1: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 40.701824] blk_update_request: critical target error, dev sdh, sector 0 [ 40.709814] blk_update_request: critical target error, dev dm-1, sector 0 [ 40.717412] Buffer I/O error on dev dm-1, logical block 0, async page read [ 40.959444] sd 0:0:0:3: [sdf] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 40.970905] sd 0:0:0:3: [sdf] tag#0 Sense Key : Illegal Request [current] [ 40.978616] sd 0:0:0:3: [sdf] tag#0 <>ASC=0x94 ASCQ=0x1 [ 40.985344] sd 0:0:0:3: [sdf] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [ 40.994794] device-mapper: multipath: Failing path 8:80. [ 41.233879] sd 0:0:1:1: [sdh] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 41.244188] sd 0:0:1:1: [sdh] tag#1 Sense Key : Hardware Error [current] [ 41.251775] sd 0:0:1:1: [sdh] tag#1 <>ASC=0x84 ASCQ=0x0 [ 41.258485] sd 0:0:1:1: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [ 41.268394] blk_update_request: critical target error, dev sdh, sector 0 [ 41.275898] blk_update_request: critical target error, dev dm-1, sector 0 [ 41.283487] Buffer I/O error on dev dm-1, logical block 0, async page read [ 41.800595] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 41.810902] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [ 41.818484] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [ 41.825193] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [ 41.835101] blk_update_request: critical target error, dev sdh, sector 24 [ 41.842712] blk_update_request: critical target error, dev dm-1, sector 24 [ 41.850399] Buffer I/O error on dev dm-1, logical block 3, async page read [ 42.009359] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 42.052940] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 42.061929] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 42.438992] FS-Cache: Loaded [ 42.539485] FS-Cache: Netfs 'nfs' registered for caching [ 42.559180] Key type dns_resolver registered [ 42.594145] NFS: Registering the id_resolver key type [ 42.599780] Key type id_resolver registered [ 42.604451] Key type id_legacy registered [ 45.246540] device-mapper: multipath: Reinstating path 8:80. [ 45.253033] device-mapper: multipath: Failing path 8:80. [ 50.253349] device-mapper: multipath: Reinstating path 8:80. [ 50.259852] device-mapper: multipath: Failing path 8:80. [ 55.261759] device-mapper: multipath: Reinstating path 8:80. [ 55.268377] device-mapper: multipath: Failing path 8:80. [ 64.486372] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 121.237074] LNet: HW NUMA nodes: 2, HW CPU cores: 32, npartitions: 2 [ 121.247741] alg: No test for adler32 (adler32-zlib) [ 122.110525] Lustre: Lustre: Build Version: 2.15.0_RC3_3_gf161c9d [ 122.318341] LNet: Using FMR for registration [ 122.385025] LNetError: 253:0:(o2iblnd_cb.c:2519:kiblnd_passive_connect()) Can't accept conn from 192.168.1.121@o2ib on NA (ib0:0:192.168.1.108): bad dst nid 192.168.1.108@o2ib [ 122.401234] LNet: Added LNI 192.168.1.108@o2ib [8/256/0/180] [ 122.637694] LDISKFS-fs warning (device dm-3): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [ 168.021962] LDISKFS-fs (dm-3): recovery complete [ 168.027560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 171.867547] Lustre: soaked-MDT0000: Imperative Recovery not enabled, recovery window 300-900 [ 178.600059] Lustre: soaked-MDT0000: Will be in recovery for at least 5:00, or until 22 clients reconnect [ 202.713067] Lustre: soaked-MDT0000: Recovery over after 0:24, of 22 clients 22 recovered and 0 were evicted. [ 3073.667827] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638197/real 0] req@ffff99ec3ce6d580 x1731865206416704/t0(0) o13->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651638204 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3073.699512] Lustre: soaked-OST0009-osc-MDT0000: Connection to soaked-OST0009 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 3075.172962] Lustre: 3507:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638199/real 0] req@ffff99ec4380f500 x1731865206429568/t0(0) o13->soaked-OST0005-osc-MDT0000@192.168.1.105@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651638206 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3075.204629] Lustre: soaked-OST0005-osc-MDT0000: Connection to soaked-OST0005 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 3076.220103] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638199/real 0] req@ffff99e81a179b00 x1731865206429632/t0(0) o13->soaked-OST000d-osc-MDT0000@192.168.1.105@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651638207 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3076.251765] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3076.262485] Lustre: soaked-OST000d-osc-MDT0000: Connection to soaked-OST000d (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 3076.281149] Lustre: Skipped 1 previous similar message [ 3078.660327] Lustre: 3519:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651638198/real 0] req@ffff99ec3b172880 x1731865206417728/t0(0) o6->soaked-OST0001-osc-MDT0000@192.168.1.105@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651638210 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 3082.191702] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [ 3082.203948] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.105@o2ib (18): c: 0, oc: 0, rc: 8 [ 3086.192100] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3090.192531] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 8 seconds [ 3090.203904] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 6 previous similar messages [ 3094.192917] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 12 seconds [ 3094.204380] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 3114.194870] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3114.206243] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 2 previous similar messages [ 3122.195642] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 2 seconds [ 3134.196817] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 3 seconds [ 3134.208196] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [ 3150.198255] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 4 seconds [ 3150.209619] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [ 3186.201480] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 2 seconds [ 3186.212853] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 3250.206786] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 11 seconds [ 3250.218258] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 20 previous similar messages [ 3289.090975] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0009_UUID (at 192.168.1.105@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec476e1000, cur 1651638420 expire 1651638270 last 1651638193 [ 3378.216333] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib: 11 seconds [ 3378.227805] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 40 previous similar messages [ 3647.310943] Lustre: soaked-OST0009-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3682.495305] Lustre: soaked-OST0001-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3728.127038] Lustre: soaked-OST0005-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3774.565549] Lustre: soaked-OST000d-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [ 3828.061021] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3828.075426] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3828.575359] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3828.589750] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 66 previous similar messages [ 3828.601052] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3828.615918] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 66 previous similar messages [ 3829.599072] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3829.613459] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 159 previous similar messages [ 3829.624849] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3829.639711] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 159 previous similar messages [ 3831.602684] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3831.617073] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 317 previous similar messages [ 3831.628449] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3831.643311] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 317 previous similar messages [ 3835.632958] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 3835.647360] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 525 previous similar messages [ 3835.658755] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 3835.673618] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 525 previous similar messages [ 4204.234178] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4204.248571] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 643 previous similar messages [ 4204.259948] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4204.274812] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 643 previous similar messages [ 4580.417042] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4580.431435] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [ 4580.442932] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4580.457799] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [ 4960.220259] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4960.234657] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1715 previous similar messages [ 4960.246167] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4960.261033] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1715 previous similar messages [ 5335.585773] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 5335.600164] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1719 previous similar messages [ 5335.611649] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 5335.626514] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1719 previous similar messages [ 5711.893902] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 5711.908294] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1718 previous similar messages [ 5711.919767] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 5711.934631] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1718 previous similar messages [ 5814.198337] LustreError: 12981:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5814.698710] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5814.721440] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) Skipped 103 previous similar messages [ 5815.704685] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5815.727436] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) Skipped 215 previous similar messages [ 5819.937782] LustreError: 13023:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5819.960625] LustreError: 13023:0:(out_handler.c:910:out_tx_end()) Skipped 399 previous similar messages [ 5824.625183] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5824.647910] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 411 previous similar messages [ 5832.681030] LustreError: 12988:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5832.703848] LustreError: 12988:0:(out_handler.c:910:out_tx_end()) Skipped 1067 previous similar messages [ 5848.682997] LustreError: 12980:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5848.705824] LustreError: 12980:0:(out_handler.c:910:out_tx_end()) Skipped 3735 previous similar messages [ 6084.656079] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 6084.670472] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1718 previous similar messages [ 6084.681959] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 6084.696818] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1718 previous similar messages [ 6830.447411] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 6830.461838] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 3439 previous similar messages [ 6830.473345] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 6830.488216] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 3439 previous similar messages [ 6954.330738] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651642074/real 0] req@ffff99e80d9bba80 x1731865430302976/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651642081 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 6954.362399] Lustre: 3496:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6954.373341] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6955.016776] Lustre: 3503:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651642075/real 0] req@ffff99e810b85580 x1731865430317440/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651642082 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 6955.048442] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6959.378955] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 19 seconds [ 6959.391201] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (20): c: 0, oc: 0, rc: 8 [ 6959.404983] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642071/real 1651642090] req@ffff99ec43130000 x1731865429951872/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642397 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6959.405034] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6959.456680] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6963.379120] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 7 seconds [ 6963.390485] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 54 previous similar messages [ 6963.400967] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642075/real 1651642094] req@ffff99ebfd0d3f00 x1731865430320000/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642401 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6963.433976] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6967.379454] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 6971.379547] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651642075/real 1651642102] req@ffff99e811077980 x1731865430359424/t0(0) o6->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651642401 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 6971.412554] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6995.380561] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [ 6995.391937] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 27 previous similar messages [ 7071.384047] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [ 7071.395426] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 17 previous similar messages [ 7141.693801] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0008_UUID (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e817084000, cur 1651642273 expire 1651642123 last 1651642046 [ 7141.718076] Lustre: Skipped 4 previous similar messages [ 7151.019626] Lustre: MGS: haven't heard from client 564d6672-71d3-4129-859c-bf1077267f7c (at 192.168.1.104@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e5b55f5c00, cur 1651642282 expire 1651642132 last 1651642053 [ 7151.043321] Lustre: Skipped 3 previous similar messages [ 7199.389962] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 5 seconds [ 7199.401336] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 32 previous similar messages [ 7575.844709] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 7575.859097] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 3436 previous similar messages [ 7575.870570] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 7575.885438] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 3436 previous similar messages [ 7760.568167] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7798.085168] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7843.083518] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 7889.797579] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [ 8800.125040] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 8800.139478] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [ 8800.150969] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8800.165837] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [ 8940.885815] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 8940.900233] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 221 previous similar messages [ 8940.911616] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8940.926473] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 221 previous similar messages [ 9175.577672] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [ 9175.591884] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 215 previous similar messages [ 9175.603262] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [ 9175.617938] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 215 previous similar messages [ 9178.625121] LustreError: 3779:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e86cdf8200 [ 9178.636120] LustreError: 3779:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0002-osp-MDT0000: write updates failed: rc = -116 [ 9549.431970] Lustre: 3497:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651644669/real 0] req@ffff99e81333ad00 x1731865561876736/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651644676 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 9549.463638] Lustre: 3497:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9549.474356] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 9554.621168] Lustre: 3509:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651644668/real 0] req@ffff99ec2a970000 x1731865561876672/t0(0) o13->soaked-OST0006-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651644675 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [ 9554.652831] Lustre: 3509:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9554.663549] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 9554.682214] Lustre: Skipped 1 previous similar message [ 9555.501201] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [ 9555.513452] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (21): c: 0, oc: 0, rc: 8 [ 9559.501433] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 5 seconds [ 9559.512802] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 72 previous similar messages [ 9595.503193] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9595.514561] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [ 9671.506897] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9671.518264] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [ 9760.286734] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e8176dcc00, cur 1651644891 expire 1651644741 last 1651644664 [ 9799.514204] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [ 9799.525572] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 33 previous similar messages [10071.526582] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [10071.537956] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 66 previous similar messages [10373.395804] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10400.410141] Lustre: mdt01_018: service thread pid 8315 was inactive for 200.154 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [10400.410145] Pid: 3776, comm: mdt00_006 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [10400.410150] Call Trace: [10400.410188] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [10400.410199] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [10400.410229] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [10400.410249] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [10400.410267] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [10400.410286] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [10400.410304] [<0>] lod_prepare_create+0x230/0x320 [lod] [10400.410322] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [10400.410339] [<0>] lod_declare_create+0x1f5/0x600 [lod] [10400.410360] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [10400.410374] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [10400.410388] [<0>] mdd_declare_create+0x66/0x480 [mdd] [10400.410401] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [10400.410437] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [10400.410463] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [10400.410484] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [10400.410507] [<0>] mdt_intent_open+0x93/0x480 [mdt] [10400.410529] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [10400.410550] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [10400.410620] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [10400.410683] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [10400.410766] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [10400.410844] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [10400.410914] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [10400.410982] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [10400.410988] [<0>] kthread+0xd1/0xe0 [10400.410994] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [10400.411025] [<0>] 0xfffffffffffffffe [10400.607430] Lustre: Skipped 1 previous similar message [10400.613182] Pid: 8315, comm: mdt01_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [10400.624192] Call Trace: [10400.626960] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [10400.633012] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [10400.638785] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [10400.644849] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [10400.651183] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [10400.658119] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [10400.664189] [<0>] lod_prepare_create+0x230/0x320 [lod] [10400.669969] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [10400.676413] [<0>] lod_declare_create+0x1f5/0x600 [lod] [10400.682183] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [10400.689400] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [10400.696522] [<0>] mdd_declare_create+0x66/0x480 [mdd] [10400.702190] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [10400.707302] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [10400.712878] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [10400.718069] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [10400.723845] [<0>] mdt_intent_open+0x93/0x480 [mdt] [10400.729237] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [10400.734615] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [10400.740337] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [10400.746318] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [10400.752699] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [10400.758006] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [10400.764188] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [10400.771235] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [10400.776714] [<0>] kthread+0xd1/0xe0 [10400.780628] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [10400.786404] [<0>] 0xfffffffffffffffe [10408.455288] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10453.530384] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10498.598006] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [10590.982372] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [10590.996772] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 439 previous similar messages [10591.008179] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [10591.023036] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 439 previous similar messages [10862.708702] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [10862.723105] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 220 previous similar messages [10862.734496] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [10862.749360] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 220 previous similar messages [10862.816467] LustreError: 7035:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e86cdf8200 [10862.827461] LustreError: 7035:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0002-osp-MDT0000: write updates failed: rc = -116 [11315.467860] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11315.490583] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) Skipped 1583 previous similar messages [11315.526385] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [11315.540570] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 218 previous similar messages [11315.551941] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [11315.566602] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 218 previous similar messages [11324.458665] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11324.481403] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 1427 previous similar messages [11332.461796] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11332.484549] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 855 previous similar messages [11352.750142] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11352.772888] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 1423 previous similar messages [11788.396746] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [11788.410958] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 742 previous similar messages [11788.422331] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [11788.436998] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 742 previous similar messages [11875.168598] LustreError: 7673:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11875.191340] LustreError: 7673:0:(out_handler.c:910:out_tx_end()) Skipped 2039 previous similar messages [11941.802780] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [11941.825521] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 7487 previous similar messages [11979.038147] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647103/real 0] req@ffff99ec40b08900 x1731865633767488/t0(0) o13->soaked-OST0007-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647110 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11979.069836] Lustre: 3515:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [11979.080671] Lustre: soaked-OST0007-osc-MDT0000: Connection to soaked-OST0007 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11979.099336] Lustre: Skipped 1 previous similar message [11979.958137] Lustre: 3498:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647104/real 0] req@ffff99e80eef1680 x1731865633769024/t0(0) o13->soaked-OST000b-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647111 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11979.989806] Lustre: soaked-OST000b-osc-MDT0000: Connection to soaked-OST000b (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11981.686236] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647105/real 0] req@ffff99ec40b0ad00 x1731865633770624/t0(0) o13->soaked-OST000f-osc-MDT0000@192.168.1.107@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651647112 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [11981.717907] Lustre: 3518:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11981.728622] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [11981.747312] Lustre: Skipped 1 previous similar message [11988.534575] Lustre: 3494:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651647103/real 0] req@ffff99e7f4e60000 x1731865633767616/t0(0) o400->soaked-OST0007-osc-MDT0000@192.168.1.107@o2ib:28/4 lens 224/224 e 0 to 1 dl 1651647119 ref 2 fl Rpc:XNr/0/ffffffff rc 0/-1 job:'' [11991.609724] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [11991.621979] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.107@o2ib (20): c: 0, oc: 0, rc: 8 [11995.609846] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 7 seconds [11995.621218] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 6 previous similar messages [12063.612911] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 2 seconds [12063.624290] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [12184.485380] Lustre: MGS: haven't heard from client 61a79e1f-9f76-41dc-8d35-5d871b03db11 (at 192.168.1.107@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e86b913000, cur 1651647315 expire 1651647165 last 1651647088 [12184.509104] Lustre: Skipped 4 previous similar messages [12191.618627] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.107@o2ib: 13 seconds [12191.630108] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 39 previous similar messages [12198.226254] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0003_UUID (at 192.168.1.107@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec6537b400, cur 1651647329 expire 1651647179 last 1651647102 [12723.832985] Lustre: soaked-OST000b-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12748.382012] Lustre: soaked-OST0003-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12799.756837] Lustre: soaked-OST000f-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12843.191517] Lustre: soaked-OST0007-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [12920.045170] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [12920.059562] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 452 previous similar messages [12920.070938] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [12920.085801] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 452 previous similar messages [13045.808055] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13045.822455] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 450 previous similar messages [13045.833840] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13045.848699] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 450 previous similar messages [13169.831601] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13169.845993] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 553 previous similar messages [13169.857383] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13169.872242] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 553 previous similar messages [13401.515261] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13401.529678] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 551 previous similar messages [13401.541069] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13401.555932] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 551 previous similar messages [13978.949869] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [13978.964259] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1114 previous similar messages [13978.975748] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [13978.990606] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1114 previous similar messages [13979.998125] LustreError: 8315:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [13980.009142] LustreError: 8315:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [14184.243025] LustreError: 3844:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [14184.254038] LustreError: 3844:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [14738.875746] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14738.898478] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 2515 previous similar messages [14754.967381] LustreError: 7630:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14754.990105] LustreError: 7630:0:(out_handler.c:910:out_tx_end()) Skipped 811 previous similar messages [14787.134232] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [14787.156964] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 1919 previous similar messages [15259.056801] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [15259.071213] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1074 previous similar messages [15259.082708] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [15259.097583] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1074 previous similar messages [15832.070105] perf: interrupt took too long (2604 > 2500), lowering kernel.perf_event_max_sample_rate to 76000 [16358.703277] Lustre: 3522:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651651482/real 0] req@ffff99ebfcf5f980 x1731865750022080/t0(0) o13->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651651489 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [16358.734937] Lustre: 3522:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16358.745666] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [16360.490380] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651651484/real 0] req@ffff99e80dce9f80 x1731865750051008/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651651491 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [16360.522050] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [16367.795698] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [16367.807959] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (18): c: 0, oc: 0, rc: 8 [16367.821714] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651651479/real 1651651498] req@ffff99ebdf6ccc80 x1731865749975424/t0(0) o6->soaked-OST0006-osc-MDT0000@192.168.1.102@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651651523 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [16367.854730] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [16556.106656] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST000a_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec6ce69800, cur 1651651687 expire 1651651537 last 1651651460 [16556.130948] Lustre: Skipped 3 previous similar messages [16571.119985] Lustre: MGS: haven't heard from client bfe70c3a-93cf-4893-ab6c-fd51acbe2210 (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80f20b000, cur 1651651702 expire 1651651552 last 1651651475 [16571.143701] Lustre: Skipped 3 previous similar messages [16683.808689] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [16683.820074] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 68 previous similar messages [16715.810133] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 4 seconds [16715.821499] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 15 previous similar messages [16779.812995] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 6 seconds [16779.824361] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 29 previous similar messages [16965.538292] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17009.993306] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17054.930575] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17099.890548] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [17113.228709] LustreError: 11-0: soaked-OST0002-osc-MDT0000: operation ost_destroy to node 192.168.1.107@o2ib failed: rc = -107 [17113.228737] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17113.228740] Lustre: Skipped 2 previous similar messages [17113.265880] LustreError: Skipped 2 previous similar messages [17120.185656] LustreError: 11-0: soaked-OST000a-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [17120.198251] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17124.827923] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17125.624354] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [17129.962124] LustreError: 11-0: soaked-OST000e-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [17129.974718] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [17134.376998] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [17134.388499] Lustre: Skipped 1 previous similar message [17168.942311] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17168.956690] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 974 previous similar messages [17168.968062] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17168.982925] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 974 previous similar messages [17178.353542] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17178.367928] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1235 previous similar messages [17178.379397] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17178.394255] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1235 previous similar messages [17403.601697] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [17403.624412] LustreError: 4920:0:(out_handler.c:910:out_tx_end()) Skipped 1487 previous similar messages [17412.485548] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [17412.508279] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 899 previous similar messages [17553.230285] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17553.244687] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 43 previous similar messages [17553.255988] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17553.270848] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 43 previous similar messages [17556.971065] LustreError: 3775:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17556.982074] LustreError: 3775:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17557.723321] LustreError: 3776:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17557.734329] LustreError: 3776:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17559.111327] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17559.122328] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) Skipped 1 previous similar message [17559.132758] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17559.145871] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) Skipped 1 previous similar message [17561.275415] LustreError: 8312:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99e6b19b6800 [17561.286409] LustreError: 8312:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [17781.962360] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [17781.976765] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1514 previous similar messages [17781.988260] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [17782.003130] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1514 previous similar messages [18001.950533] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [18001.964937] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1237 previous similar messages [18001.976438] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [18001.991307] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1237 previous similar messages [18356.494387] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18356.517127] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 2799 previous similar messages [18358.519508] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18358.542239] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) Skipped 379 previous similar messages [18362.700751] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18362.723515] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 599 previous similar messages [18370.755776] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18370.778517] LustreError: 7993:0:(out_handler.c:910:out_tx_end()) Skipped 419 previous similar messages [18386.818212] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [18386.840955] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 1259 previous similar messages [18818.493369] Lustre: 3492:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651653942/real 0] req@ffff99e7ca8d0000 x1731865846555840/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651653949 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [18818.525050] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [18819.933437] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651653943/real 0] req@ffff99e7ca8d3600 x1731865846624768/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651653950 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [18819.965113] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18819.975853] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [18819.994522] Lustre: Skipped 1 previous similar message [18827.904803] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [18827.917064] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (19): c: 0, oc: 0, rc: 8 [19018.611368] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0008_UUID (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e816611800, cur 1651654149 expire 1651653999 last 1651653922 [19020.073179] Lustre: MGS: haven't heard from client ea8ea456-a9d5-467c-8526-0377b4b867c8 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99ec46611c00, cur 1651654150 expire 1651654000 last 1651653922 [19020.096878] Lustre: Skipped 3 previous similar messages [19022.862584] Lustre: mdt00_015: service thread pid 8312 was inactive for 200.668 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19022.862587] Lustre: mdt00_003: service thread pid 3654 was inactive for 200.694 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19022.862592] Lustre: mdt01_018: service thread pid 8315 was inactive for 200.678 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19022.862597] Pid: 3787, comm: mdt01_011 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19022.862601] Lustre: Skipped 1 previous similar message [19022.862604] Lustre: Skipped 1 previous similar message [19022.862610] Call Trace: [19022.862653] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19022.862664] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19022.862698] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19022.862716] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19022.862734] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19022.862751] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19022.862768] [<0>] lod_prepare_create+0x230/0x320 [lod] [19022.862783] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19022.862799] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19022.862823] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19022.862835] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19022.862848] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19022.862861] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19022.862901] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19022.862925] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19022.862945] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19022.862965] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19022.862985] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19022.863006] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19022.863080] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19022.863141] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19022.863223] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19022.863296] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19022.863361] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19022.863424] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19022.863431] [<0>] kthread+0xd1/0xe0 [19022.863438] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19022.863469] [<0>] 0xfffffffffffffffe [19022.863471] Pid: 8315, comm: mdt01_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19022.863472] Call Trace: [19022.863502] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19022.863512] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19022.863559] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19022.863579] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19022.863598] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19022.863616] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19022.863634] [<0>] lod_prepare_create+0x230/0x320 [lod] [19022.863652] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19022.863669] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19022.863687] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19022.863701] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19022.863714] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19022.863728] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19022.863757] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19022.863783] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19022.863805] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19022.863827] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19022.863849] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19022.863871] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19022.863925] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19022.863988] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19022.864066] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19022.864143] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19022.864213] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19022.864281] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19022.864285] [<0>] kthread+0xd1/0xe0 [19022.864289] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19022.864300] [<0>] 0xfffffffffffffffe [19023.284154] Pid: 8312, comm: mdt00_015 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19023.295135] Call Trace: [19023.297901] [<0>] osp_precreate_reserve+0x490/0x9b0 [osp] [19023.303946] [<0>] osp_declare_create+0x1a7/0x6c0 [osp] [19023.309703] [<0>] lod_sub_declare_create+0xe3/0x280 [lod] [19023.315746] [<0>] lod_qos_declare_object_on+0xf3/0x420 [lod] [19023.322077] [<0>] lod_ost_alloc_rr.constprop.22+0xab8/0x1180 [lod] [19023.328993] [<0>] lod_qos_prep_create+0x121a/0x1aa0 [lod] [19023.335029] [<0>] lod_prepare_create+0x230/0x320 [lod] [19023.340779] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19023.347210] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19023.352961] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19023.360171] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19023.367268] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19023.372919] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19023.378010] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19023.383565] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19023.388732] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19023.394485] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19023.399843] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19023.405206] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19023.410901] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19023.416864] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19023.423224] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19023.428517] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19023.434682] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19023.441712] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19023.447172] [<0>] kthread+0xd1/0xe0 [19023.451070] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19023.456823] [<0>] 0xfffffffffffffffe [19081.841301] INFO: task mdt01_002:3565 blocked for more than 120 seconds. [19081.848805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19081.857558] mdt01_002 D ffff99ec4e59b180 0 3565 2 0x00000080 [19081.865503] Call Trace: [19081.868270] [] schedule+0x29/0x70 [19081.873843] [] rwsem_down_write_failed+0x215/0x3c0 [19081.881056] [] call_rwsem_down_write_failed+0x17/0x30 [19081.888556] [] down_write+0x2d/0x3d [19081.894347] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19081.902921] [] ? osp_statfs+0x1ff/0x530 [osp] [19081.909674] [] ? mutex_lock+0x12/0x2f [19081.915634] [] ? memset+0x22/0xb0 [19081.921209] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19081.928855] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19081.937430] [] lod_prepare_create+0x230/0x320 [lod] [19081.944740] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19081.952730] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19081.960541] [] lod_declare_create+0x1f5/0x600 [lod] [19081.967866] [] ? lod_get_ea+0xc6/0x530 [lod] [19081.974507] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19081.983266] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19081.991945] [] mdd_declare_create+0x66/0x480 [mdd] [19081.999153] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.005807] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.012973] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.020800] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.029176] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.037751] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.044478] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.051805] [] mdt_intent_open+0x93/0x480 [mdt] [19082.058728] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.065660] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.073648] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.080924] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.088424] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.096410] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.103553] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.111496] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.119258] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.126116] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.133847] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.142317] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.150231] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.158794] [] ? task_rq_unlock+0x20/0x20 [19082.165124] [] ? __wake_up+0x13/0x20 [19082.171006] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.178058] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.185747] [] kthread+0xd1/0xe0 [19082.191204] [] ? insert_kthread_work+0x40/0x40 [19082.198030] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.205338] [] ? insert_kthread_work+0x40/0x40 [19082.212165] INFO: task mdt01_004:3769 blocked for more than 120 seconds. [19082.219657] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.228410] mdt01_004 D ffff99ec4769e300 0 3769 2 0x00000080 [19082.236326] Call Trace: [19082.239073] [] schedule+0x29/0x70 [19082.244626] [] rwsem_down_write_failed+0x215/0x3c0 [19082.251829] [] call_rwsem_down_write_failed+0x17/0x30 [19082.259326] [] down_write+0x2d/0x3d [19082.265085] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.273694] [] ? tgt_free_reply_data+0xe7/0x270 [ptlrpc] [19082.281481] [] ? kfree+0x106/0x140 [19082.287142] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19082.294749] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19082.303327] [] lod_prepare_create+0x230/0x320 [lod] [19082.310635] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19082.318621] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19082.326415] [] lod_declare_create+0x1f5/0x600 [lod] [19082.333726] [] ? lod_get_ea+0xc6/0x530 [lod] [19082.340354] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19082.349109] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19082.357775] [] mdd_declare_create+0x66/0x480 [mdd] [19082.364980] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.371622] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.378753] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.386559] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.394936] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.403509] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.410253] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.417556] [] mdt_intent_open+0x93/0x480 [mdt] [19082.424479] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.431402] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.439382] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.446615] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.454115] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.462096] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.469238] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.477163] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.484897] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.491755] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.499489] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.507956] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.515883] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.524443] [] ? task_rq_unlock+0x20/0x20 [19082.530772] [] ? __wake_up+0x13/0x20 [19082.536654] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.543699] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.551378] [] kthread+0xd1/0xe0 [19082.556835] [] ? insert_kthread_work+0x40/0x40 [19082.563650] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.570949] [] ? insert_kthread_work+0x40/0x40 [19082.577771] INFO: task mdt01_006:3774 blocked for more than 120 seconds. [19082.585264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.594015] mdt01_006 D ffff99ec6a4b6300 0 3774 2 0x00000080 [19082.601938] Call Trace: [19082.604685] [] schedule+0x29/0x70 [19082.610240] [] rwsem_down_write_failed+0x215/0x3c0 [19082.617442] [] call_rwsem_down_write_failed+0x17/0x30 [19082.624936] [] down_write+0x2d/0x3d [19082.630696] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.639267] [] ? lod_sub_write+0x1d0/0x440 [lod] [19082.646305] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19082.653905] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19082.662474] [] lod_prepare_create+0x230/0x320 [lod] [19082.669787] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19082.677773] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19082.685568] [] lod_declare_create+0x1f5/0x600 [lod] [19082.692872] [] ? lod_get_ea+0xc6/0x530 [lod] [19082.699499] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19082.708275] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19082.716939] [] mdd_declare_create+0x66/0x480 [mdd] [19082.724143] [] mdd_create+0x9a9/0x1cd0 [mdd] [19082.730785] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19082.737912] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19082.745719] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19082.754094] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19082.762664] [] mdt_reint_rec+0x8a/0x240 [mdt] [19082.769391] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19082.776700] [] mdt_intent_open+0x93/0x480 [mdt] [19082.783622] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19082.790544] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19082.798533] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19082.805777] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19082.813309] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19082.821323] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19082.828497] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19082.836433] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19082.844181] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19082.851054] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19082.858798] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19082.867281] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19082.875227] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19082.883813] [] ? task_rq_unlock+0x20/0x20 [19082.890163] [] ? __wake_up+0x13/0x20 [19082.896061] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19082.903121] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19082.910825] [] kthread+0xd1/0xe0 [19082.916298] [] ? insert_kthread_work+0x40/0x40 [19082.923129] [] ret_from_fork_nospec_begin+0x21/0x21 [19082.930450] [] ? insert_kthread_work+0x40/0x40 [19082.937272] INFO: task mdt00_005:3775 blocked for more than 120 seconds. [19082.944772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19082.953523] mdt00_005 D ffff99ec6a4b2100 0 3775 2 0x00000080 [19082.961460] Call Trace: [19082.964214] [] schedule+0x29/0x70 [19082.969775] [] rwsem_down_write_failed+0x215/0x3c0 [19082.976987] [] call_rwsem_down_write_failed+0x17/0x30 [19082.984490] [] down_write+0x2d/0x3d [19082.990289] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19082.998860] [] ? lod_sub_write+0x1d0/0x440 [lod] [19083.005900] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.013526] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.022119] [] lod_prepare_create+0x230/0x320 [lod] [19083.029435] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.037438] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.045260] [] lod_declare_create+0x1f5/0x600 [lod] [19083.052581] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.059232] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.068017] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.076687] [] mdd_declare_create+0x66/0x480 [mdd] [19083.083909] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.090559] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.097698] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.105513] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.113905] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.122479] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.129208] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.136526] [] mdt_intent_open+0x93/0x480 [mdt] [19083.143472] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.150410] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.158414] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.165655] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.173169] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.181166] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.188313] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.196269] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.204026] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.210907] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.218651] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.227134] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.235060] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.243637] [] ? task_rq_unlock+0x20/0x20 [19083.249975] [] ? __wake_up+0x13/0x20 [19083.255879] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.262949] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.270659] [] kthread+0xd1/0xe0 [19083.276137] [] ? insert_kthread_work+0x40/0x40 [19083.282965] [] ret_from_fork_nospec_begin+0x21/0x21 [19083.290283] [] ? insert_kthread_work+0x40/0x40 [19083.297113] INFO: task mdt00_007:3780 blocked for more than 120 seconds. [19083.304612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19083.313370] mdt00_007 D ffff99ec5d1ab180 0 3780 2 0x00000080 [19083.321335] Call Trace: [19083.324090] [] schedule+0x29/0x70 [19083.329652] [] rwsem_down_write_failed+0x215/0x3c0 [19083.336862] [] call_rwsem_down_write_failed+0x17/0x30 [19083.344365] [] down_write+0x2d/0x3d [19083.350138] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19083.358706] [] ? __radix_tree_lookup+0x84/0xf0 [19083.365545] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.373154] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.381738] [] lod_prepare_create+0x230/0x320 [lod] [19083.389066] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.397075] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.404875] [] lod_declare_create+0x1f5/0x600 [lod] [19083.412195] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.418841] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.427608] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.436302] [] mdd_declare_create+0x66/0x480 [mdd] [19083.443533] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.450193] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.457324] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.465131] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.473519] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.482100] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.488828] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.496154] [] mdt_intent_open+0x93/0x480 [mdt] [19083.503099] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.510030] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.518041] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.525308] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.532839] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.540836] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.547992] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.555928] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.563675] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.570557] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.578314] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.586789] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.594715] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.603290] [] ? task_rq_unlock+0x20/0x20 [19083.609617] [] ? __wake_up+0x13/0x20 [19083.615498] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.622544] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.630238] [] kthread+0xd1/0xe0 [19083.635701] [] ? insert_kthread_work+0x40/0x40 [19083.642516] [] ret_from_fork_nospec_begin+0x21/0x21 [19083.649822] [] ? insert_kthread_work+0x40/0x40 [19083.656635] INFO: task mdt00_009:4645 blocked for more than 120 seconds. [19083.664133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19083.672890] mdt00_009 D ffff99ec6941d280 0 4645 2 0x00000080 [19083.680817] Call Trace: [19083.683552] [] schedule+0x29/0x70 [19083.689113] [] rwsem_down_write_failed+0x215/0x3c0 [19083.696330] [] call_rwsem_down_write_failed+0x17/0x30 [19083.703821] [] down_write+0x2d/0x3d [19083.709566] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19083.718110] [] ? __radix_tree_lookup+0x84/0xf0 [19083.724939] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19083.732538] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19083.741114] [] lod_prepare_create+0x230/0x320 [lod] [19083.748428] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19083.756428] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19083.764215] [] lod_declare_create+0x1f5/0x600 [lod] [19083.771514] [] ? lod_get_ea+0xc6/0x530 [lod] [19083.778140] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19083.786922] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19083.795579] [] mdd_declare_create+0x66/0x480 [mdd] [19083.802780] [] mdd_create+0x9a9/0x1cd0 [mdd] [19083.809421] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19083.816547] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19083.824366] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19083.832753] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19083.841328] [] mdt_reint_rec+0x8a/0x240 [mdt] [19083.848079] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19083.855404] [] mdt_intent_open+0x93/0x480 [mdt] [19083.862322] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19083.869239] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19083.877214] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19083.884441] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19083.891936] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19083.899917] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19083.907054] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19083.914969] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19083.922697] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19083.929551] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19083.937280] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19083.945745] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19083.953660] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19083.962218] [] ? task_rq_unlock+0x20/0x20 [19083.968541] [] ? __wake_up+0x13/0x20 [19083.974418] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19083.981460] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19083.989149] [] kthread+0xd1/0xe0 [19083.994600] [] ? insert_kthread_work+0x40/0x40 [19084.001423] [] ret_from_fork_nospec_begin+0x21/0x21 [19084.008719] [] ? insert_kthread_work+0x40/0x40 [19107.917457] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [19107.928833] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 60 previous similar messages [19131.918500] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [19131.929865] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [19157.524661] Lustre: mdt00_009: service thread pid 4645 was inactive for 242.485 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19159.060697] Lustre: mdt01_002: service thread pid 3565 was inactive for 244.022 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19163.919958] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 30 seconds [19163.931421] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 19 previous similar messages [19164.180963] Lustre: mdt00_007: service thread pid 3780 was inactive for 242.018 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19204.020806] INFO: task mdt01_002:3565 blocked for more than 120 seconds. [19204.028319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.037080] mdt01_002 D ffff99ec4e59b180 0 3565 2 0x00000080 [19204.045020] Call Trace: [19204.047786] [] schedule+0x29/0x70 [19204.053364] [] rwsem_down_write_failed+0x215/0x3c0 [19204.060594] [] call_rwsem_down_write_failed+0x17/0x30 [19204.068100] [] down_write+0x2d/0x3d [19204.073887] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.082461] [] ? osp_statfs+0x1ff/0x530 [osp] [19204.089206] [] ? mutex_lock+0x12/0x2f [19204.095171] [] ? memset+0x22/0xb0 [19204.100745] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.108386] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.116956] [] lod_prepare_create+0x230/0x320 [lod] [19204.124270] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.132286] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.140083] [] lod_declare_create+0x1f5/0x600 [lod] [19204.147399] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.154052] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.162795] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.171467] [] mdd_declare_create+0x66/0x480 [mdd] [19204.178672] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.185366] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.192545] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.200373] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.208761] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.217333] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.224060] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.231362] [] mdt_intent_open+0x93/0x480 [mdt] [19204.238287] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.245209] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.253193] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.260470] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.267972] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19204.275959] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19204.283093] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19204.291020] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19204.298753] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19204.305616] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19204.313325] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19204.321790] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19204.329722] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19204.338307] [] ? task_rq_unlock+0x20/0x20 [19204.344635] [] ? __wake_up+0x13/0x20 [19204.350512] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19204.357561] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19204.365264] [] kthread+0xd1/0xe0 [19204.370718] [] ? insert_kthread_work+0x40/0x40 [19204.377555] [] ret_from_fork_nospec_begin+0x21/0x21 [19204.384852] [] ? insert_kthread_work+0x40/0x40 [19204.391674] INFO: task mdt01_004:3769 blocked for more than 120 seconds. [19204.399171] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.407931] mdt01_004 D ffff99ec4769e300 0 3769 2 0x00000080 [19204.415867] Call Trace: [19204.418607] [] schedule+0x29/0x70 [19204.424160] [] rwsem_down_write_failed+0x215/0x3c0 [19204.431374] [] call_rwsem_down_write_failed+0x17/0x30 [19204.438866] [] down_write+0x2d/0x3d [19204.444635] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.453232] [] ? tgt_free_reply_data+0xe7/0x270 [ptlrpc] [19204.461022] [] ? kfree+0x106/0x140 [19204.466678] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.474283] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.482863] [] lod_prepare_create+0x230/0x320 [lod] [19204.490168] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.498164] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.505958] [] lod_declare_create+0x1f5/0x600 [lod] [19204.513275] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.519899] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.528656] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.537314] [] mdd_declare_create+0x66/0x480 [mdd] [19204.544519] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.551157] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.558276] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.566074] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.574447] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.583000] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.589726] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.597031] [] mdt_intent_open+0x93/0x480 [mdt] [19204.603950] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.610868] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.618854] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.626077] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.633573] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19204.641553] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19204.648681] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19204.656588] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19204.664299] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19204.671144] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19204.678863] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19204.687328] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19204.695240] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19204.703798] [] ? task_rq_unlock+0x20/0x20 [19204.710117] [] ? __wake_up+0x13/0x20 [19204.715986] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19204.723023] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19204.730706] [] kthread+0xd1/0xe0 [19204.736161] [] ? insert_kthread_work+0x40/0x40 [19204.742970] [] ret_from_fork_nospec_begin+0x21/0x21 [19204.750281] [] ? insert_kthread_work+0x40/0x40 [19204.757096] INFO: task mdt01_005:3770 blocked for more than 120 seconds. [19204.764600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19204.773358] mdt01_005 D ffff99ec6a533180 0 3770 2 0x00000080 [19204.781312] Call Trace: [19204.784036] [] schedule+0x29/0x70 [19204.789602] [] rwsem_down_write_failed+0x215/0x3c0 [19204.796810] [] call_rwsem_down_write_failed+0x17/0x30 [19204.804307] [] down_write+0x2d/0x3d [19204.810063] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19204.818622] [] ? osp_statfs+0x1ff/0x530 [osp] [19204.825359] [] ? mutex_lock+0x12/0x2f [19204.831324] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19204.838915] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19204.847493] [] lod_prepare_create+0x230/0x320 [lod] [19204.854799] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19204.862777] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19204.870578] [] lod_declare_create+0x1f5/0x600 [lod] [19204.877884] [] ? lod_get_ea+0xc6/0x530 [lod] [19204.884509] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19204.893262] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19204.901920] [] mdd_declare_create+0x66/0x480 [mdd] [19204.909125] [] mdd_create+0x9a9/0x1cd0 [mdd] [19204.915760] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19204.922884] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19204.930692] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19204.939051] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19204.947615] [] mdt_reint_rec+0x8a/0x240 [mdt] [19204.954338] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19204.961663] [] mdt_intent_open+0x93/0x480 [mdt] [19204.968577] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19204.975508] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19204.983490] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19204.990729] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19204.998222] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19205.006214] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19205.013336] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19205.021250] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19205.028967] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19205.035806] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19205.043539] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19205.052010] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19205.059904] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19205.068461] [] ? task_rq_unlock+0x20/0x20 [19205.074791] [] ? __wake_up+0x13/0x20 [19205.080655] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19205.087710] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19205.095393] [] kthread+0xd1/0xe0 [19205.100853] [] ? insert_kthread_work+0x40/0x40 [19205.107668] [] ret_from_fork_nospec_begin+0x21/0x21 [19205.114979] [] ? insert_kthread_work+0x40/0x40 [19205.121791] INFO: task mdt01_006:3774 blocked for more than 120 seconds. [19205.129289] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19205.138040] mdt01_006 D ffff99ec6a4b6300 0 3774 2 0x00000080 [19205.145928] Call Trace: [19205.148680] [] schedule+0x29/0x70 [19205.154231] [] rwsem_down_write_failed+0x215/0x3c0 [19205.161430] [] call_rwsem_down_write_failed+0x17/0x30 [19205.168923] [] down_write+0x2d/0x3d [19205.174685] [] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19205.183255] [] ? lod_sub_write+0x1d0/0x440 [lod] [19205.190285] [] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19205.197882] [] ? osd_declare_inode_qid+0x283/0x450 [osd_ldiskfs] [19205.206444] [] lod_prepare_create+0x230/0x320 [lod] [19205.213759] [] lod_declare_striped_create+0xf8/0xa40 [lod] [19205.221732] [] ? lod_sub_declare_create+0xe3/0x280 [lod] [19205.229525] [] lod_declare_create+0x1f5/0x600 [lod] [19205.236827] [] ? lod_get_ea+0xc6/0x530 [lod] [19205.243463] [] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19205.252208] [] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19205.260873] [] mdd_declare_create+0x66/0x480 [mdd] [19205.268078] [] mdd_create+0x9a9/0x1cd0 [mdd] [19205.274726] [] mdt_reint_open+0x208a/0x2e90 [mdt] [19205.281837] [] ? check_unlink_entry+0x19/0xe0 [obdclass] [19205.289644] [] ? upcall_cache_get_entry+0x227/0x900 [obdclass] [19205.298012] [] ? ucred_set_audit_enabled.isra.12+0x22/0x60 [mdt] [19205.306589] [] mdt_reint_rec+0x8a/0x240 [mdt] [19205.313324] [] mdt_reint_internal+0x76c/0xb50 [mdt] [19205.320628] [] mdt_intent_open+0x93/0x480 [mdt] [19205.327566] [] mdt_intent_opc+0x1e0/0xc10 [mdt] [19205.334487] [] ? mdt_intent_fixup_resent+0x240/0x240 [mdt] [19205.342490] [] mdt_intent_policy+0x1a1/0x360 [mdt] [19205.349708] [] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19205.357194] [] ? cfs_hash_bd_add_locked+0x67/0x90 [libcfs] [19205.365166] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [19205.372297] [] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19205.380190] [] ? lustre_msg_buf_v2+0x150/0x1e0 [ptlrpc] [19205.387916] [] tgt_enqueue+0x64/0x240 [ptlrpc] [19205.394781] [] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19205.402533] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [19205.410991] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [19205.418896] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19205.427475] [] ? task_rq_unlock+0x20/0x20 [19205.433809] [] ? __wake_up+0x13/0x20 [19205.439702] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19205.446732] [] ? ptlrpc_wait_event+0x5c0/0x5c0 [ptlrpc] [19205.454433] [] kthread+0xd1/0xe0 [19205.459900] [] ? insert_kthread_work+0x40/0x40 [19205.466740] [] ret_from_fork_nospec_begin+0x21/0x21 [19205.474031] [] ? insert_kthread_work+0x40/0x40 [19227.922847] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 94 seconds [19227.934308] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [19330.076436] ptlrpc_watchdog_fire: 10 callbacks suppressed [19330.082524] Lustre: mdt01_010: service thread pid 3785 was inactive for 344.744 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19330.103418] Pid: 3785, comm: mdt01_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19330.114410] Call Trace: [19330.117181] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19330.123160] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19330.130193] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19330.136253] [<0>] lod_prepare_create+0x230/0x320 [lod] [19330.142030] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19330.148487] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19330.154289] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19330.161496] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19330.168606] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19330.174254] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19330.179368] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19330.184928] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19330.190112] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19330.195877] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19330.201239] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19330.206607] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19330.212340] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19330.218313] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19330.224698] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19330.230013] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19330.236208] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19330.243237] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19330.248690] [<0>] kthread+0xd1/0xe0 [19330.252588] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19330.258347] [<0>] 0xfffffffffffffffe [19416.628368] Lustre: 22106:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec2ba3b180 x1731763470721664/t0(0) o101->00629187-5c6d-4754-a616-e35b213076e9@192.168.1.121@o2ib:697/0 lens 1312/4072 e 23 to 0 dl 1651654552 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19417.630406] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebe87a0900 x1731763170365440/t0(0) o101->9a274bc2-9a2c-4748-bcc9-e7338953ee88@192.168.1.125@o2ib:698/0 lens 1312/4072 e 23 to 0 dl 1651654553 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19417.662054] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [19423.260103] Lustre: soaked-MDT0000: Client 24816f25-f323-4fbc-a2f1-9bdf8a245040 (at 192.168.1.128@o2ib) reconnecting [19423.271875] Lustre: Skipped 2 previous similar messages [19467.298589] Lustre: mdt01_015: service thread pid 7035 was inactive for 444.876 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [19467.298591] Lustre: mdt01_014: service thread pid 5289 was inactive for 444.875 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [19467.298594] Pid: 22107, comm: mdt01_021 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19467.298596] Lustre: Skipped 5 previous similar messages [19467.298597] Call Trace: [19467.298643] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19467.298684] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19467.298700] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19467.298712] [<0>] lod_prepare_create+0x230/0x320 [lod] [19467.298723] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19467.298734] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19467.298757] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19467.298775] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19467.298796] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19467.298804] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19467.298836] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19467.298851] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19467.298863] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19467.298876] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19467.298889] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19467.298901] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19467.298971] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19467.299019] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19467.299098] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19467.299148] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19467.299193] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19467.299253] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19467.299258] [<0>] kthread+0xd1/0xe0 [19467.299263] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19467.299312] [<0>] 0xfffffffffffffffe [19467.499733] Lustre: Skipped 1 previous similar message [19467.505483] Pid: 7035, comm: mdt01_015 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [19467.516460] Call Trace: [19467.519206] [<0>] call_rwsem_down_write_failed+0x17/0x30 [19467.525154] [<0>] lod_ost_alloc_qos.constprop.21+0x2cd/0x1020 [lod] [19467.532165] [<0>] lod_qos_prep_create+0x11de/0x1aa0 [lod] [19467.538203] [<0>] lod_prepare_create+0x230/0x320 [lod] [19467.543949] [<0>] lod_declare_striped_create+0xf8/0xa40 [lod] [19467.550376] [<0>] lod_declare_create+0x1f5/0x600 [lod] [19467.556131] [<0>] mdd_declare_create_object_internal+0xd6/0x3b0 [mdd] [19467.563330] [<0>] mdd_declare_create_object.isra.35+0x51/0xb60 [mdd] [19467.570439] [<0>] mdd_declare_create+0x66/0x480 [mdd] [19467.576088] [<0>] mdd_create+0x9a9/0x1cd0 [mdd] [19467.581178] [<0>] mdt_reint_open+0x208a/0x2e90 [mdt] [19467.586736] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [19467.591902] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [19467.597656] [<0>] mdt_intent_open+0x93/0x480 [mdt] [19467.603019] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [19467.608383] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [19467.614091] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [19467.620064] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [19467.626430] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [19467.631726] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [19467.637878] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [19467.644908] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [19467.650362] [<0>] kthread+0xd1/0xe0 [19467.654263] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [19467.660024] [<0>] 0xfffffffffffffffe [19509.668553] Lustre: 3784:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebf29b3f00 x1731763737452864/t0(0) o101->cb2a3a98-2672-400e-b628-aaad60526b53@192.168.1.137@o2ib:35/0 lens 1304/4072 e 5 to 0 dl 1651654645 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19516.107655] Lustre: soaked-MDT0000: Client cb2a3a98-2672-400e-b628-aaad60526b53 (at 192.168.1.137@o2ib) reconnecting [19516.119588] Lustre: Skipped 2 previous similar messages [19516.686837] Lustre: 3693:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ebe73e8d80 x1731770169837504/t0(0) o101->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:42/0 lens 1312/4072 e 4 to 0 dl 1651654652 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19516.718285] Lustre: 3693:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [19523.232257] Lustre: soaked-MDT0000: Client 2b5be8a8-9023-43d6-8098-538f23791711 (at 192.168.1.122@o2ib) reconnecting [19523.244038] Lustre: Skipped 1 previous similar message [19617.961465] Lustre: 35179:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb78c4f980 x1731764682509312/t0(0) o101->26c1c6f4-8570-40b8-af2d-5b5a5e2c346b@192.168.1.123@o2ib:143/0 lens 1312/4072 e 2 to 0 dl 1651654753 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19617.993112] Lustre: 35179:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [19623.428848] Lustre: soaked-MDT0000: Client 26c1c6f4-8570-40b8-af2d-5b5a5e2c346b (at 192.168.1.123@o2ib) reconnecting [19623.440640] Lustre: Skipped 1 previous similar message [19662.734178] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19662.745672] Lustre: Skipped 1 previous similar message [19781.399916] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19866.517068] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [19898.590439] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.117@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19900.277931] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19900.297745] LustreError: Skipped 1 previous similar message [19900.953207] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [19900.965462] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (21): c: 5, oc: 0, rc: 8 [19900.979228] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655030/real 1651655031] req@ffff99e7db728480 x1731865889557824/t0(0) o400->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/224 e 0 to 1 dl 1651655074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'' [19900.979233] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655014/real 1651655031] req@ffff99e8117a0d80 x1731865889050560/t0(0) o41->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651655058 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [19900.979239] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19900.979260] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [19900.979261] Lustre: Skipped 1 previous similar message [19902.608674] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19902.628490] LustreError: Skipped 1 previous similar message [19905.293654] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19905.313487] LustreError: Skipped 7 previous similar messages [19910.301836] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19910.321663] LustreError: Skipped 12 previous similar messages [19916.953971] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.109@o2ib: 1 seconds [19916.965340] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 51 previous similar messages [19917.879072] Lustre: 7037:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/-295), not sending early reply req@ffff99eb78c0da00 x1731763470852416/t0(0) o101->00629187-5c6d-4754-a616-e35b213076e9@192.168.1.121@o2ib:442/0 lens 1312/4072 e 1 to 0 dl 1651655052 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [19917.910928] Lustre: 7037:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [19918.629183] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.117@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19918.649009] LustreError: Skipped 30 previous similar messages [19935.343042] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.129@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19935.362885] LustreError: Skipped 95 previous similar messages [19967.506691] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.102@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [19967.526519] LustreError: Skipped 190 previous similar messages [19972.247610] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [20031.575005] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.135@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [20031.594857] LustreError: Skipped 381 previous similar messages [20079.857446] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20080.375714] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20080.386240] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20080.896518] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20080.907027] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.431028] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20081.441555] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.542903] sd 0:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [20081.553289] sd 0:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current] [20081.553304] sd 0:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553311] sd 0:0:0:0: [sdc] tag#2 Sense Key : Illegal Request [current] [20081.553319] sd 0:0:0:0: [sdc] tag#2 <>ASC=0x94 ASCQ=0x1 [20081.553325] sd 0:0:0:0: [sdc] tag#2 CDB: Write(16) 8a 00 00 00 00 00 40 00 00 48 00 00 00 08 00 00 [20081.553330] blk_update_request: I/O error, dev sdc, sector 1073741896 [20081.553350] sd 0:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553354] sd 0:0:0:0: [sdc] tag#3 Sense Key : Illegal Request [current] [20081.553359] sd 0:0:0:0: [sdc] tag#3 <>ASC=0x94 ASCQ=0x1 [20081.553363] sd 0:0:0:0: [sdc] tag#3 CDB: Write(16) 8a 00 00 00 00 00 40 40 00 50 00 00 00 08 00 00 [20081.553366] blk_update_request: I/O error, dev sdc, sector 1077936208 [20081.553376] sd 0:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553380] sd 0:0:0:0: [sdc] tag#4 Sense Key : Illegal Request [current] [20081.553385] sd 0:0:0:0: [sdc] tag#4 <>ASC=0x94 ASCQ=0x1 [20081.553388] sd 0:0:0:0: [sdc] tag#4 CDB: Write(16) 8a 00 00 00 00 00 40 40 00 80 00 00 00 08 00 00 [20081.553390] blk_update_request: I/O error, dev sdc, sector 1077936256 [20081.553400] sd 0:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553404] sd 0:0:0:0: [sdc] tag#5 Sense Key : Illegal Request [current] [20081.553408] sd 0:0:0:0: [sdc] tag#5 <>ASC=0x94 ASCQ=0x1 [20081.553412] sd 0:0:0:0: [sdc] tag#5 CDB: Write(16) 8a 00 00 00 00 00 40 00 00 80 00 00 00 08 00 00 [20081.553414] blk_update_request: I/O error, dev sdc, sector 1073741952 [20081.553423] sd 0:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553427] sd 0:0:0:0: [sdc] tag#6 Sense Key : Illegal Request [current] [20081.553432] sd 0:0:0:0: [sdc] tag#6 <>ASC=0x94 ASCQ=0x1 [20081.553435] sd 0:0:0:0: [sdc] tag#6 CDB: Write(16) 8a 00 00 00 00 00 40 00 03 c8 00 00 00 08 00 00 [20081.553437] blk_update_request: I/O error, dev sdc, sector 1073742792 [20081.553446] sd 0:0:0:0: [sdc] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553449] sd 0:0:0:0: [sdc] tag#7 Sense Key : Illegal Request [current] [20081.553454] sd 0:0:0:0: [sdc] tag#7 <>ASC=0x94 ASCQ=0x1 [20081.553458] sd 0:0:0:0: [sdc] tag#7 CDB: Write(16) 8a 00 00 00 00 00 3f c0 00 80 00 00 00 08 00 00 [20081.553459] blk_update_request: I/O error, dev sdc, sector 1069547648 [20081.553469] sd 0:0:0:0: [sdc] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553472] sd 0:0:0:0: [sdc] tag#8 Sense Key : Illegal Request [current] [20081.553477] sd 0:0:0:0: [sdc] tag#8 <>ASC=0x94 ASCQ=0x1 [20081.553481] sd 0:0:0:0: [sdc] tag#8 CDB: Write(16) 8a 00 00 00 00 00 3f c0 03 c8 00 00 00 08 00 00 [20081.553482] blk_update_request: I/O error, dev sdc, sector 1069548488 [20081.553491] sd 0:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553494] sd 0:0:0:0: [sdc] tag#9 Sense Key : Illegal Request [current] [20081.553499] sd 0:0:0:0: [sdc] tag#9 <>ASC=0x94 ASCQ=0x1 [20081.553503] sd 0:0:0:0: [sdc] tag#9 CDB: Write(16) 8a 00 00 00 00 00 3f c0 00 20 00 00 00 08 00 00 [20081.553504] blk_update_request: I/O error, dev sdc, sector 1069547552 [20081.553514] sd 0:0:0:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s [20081.553517] sd 0:0:0:0: [sdc] tag#10 Sense Key : Illegal Request [current] [20081.553522] sd 0:0:0:0: [sdc] tag#10 <>ASC=0x94 ASCQ=0x1 [20081.553526] sd 0:0:0:0: [sdc] tag#10 CDB: Write(16) 8a 00 00 00 00 00 3f 2c 10 40 00 00 00 08 00 00 [20081.553528] blk_update_request: I/O error, dev sdc, sector 1059852352 [20081.553535] blk_update_request: I/O error, dev sdc, sector 1059852224 [20081.553846] device-mapper: multipath: Failing path 8:32. [20081.951288] sd 0:0:0:0: [sdc] tag#1 <>ASC=0x94 ASCQ=0x1 [20081.956794] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [20081.956804] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20081.977827] sd 0:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 b1 23 78 00 00 00 08 00 00 [20082.290399] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [20082.299082] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [20082.879213] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [20083.899568] Buffer I/O error on dev dm-1, logical block 873318384, async page read [20084.915892] Buffer I/O error on dev dm-1, logical block 0, async page read [20085.935954] Buffer I/O error on dev dm-1, logical block 873318399, async page read [20086.429416] device-mapper: multipath: Reinstating path 8:32. [20086.445146] Buffer I/O error on dev dm-1, logical block 0, async page read [20086.784054] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [20086.954225] scsi_io_completion: 127 callbacks suppressed [20086.960184] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20086.970483] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20086.978085] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20086.984798] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20086.994707] blk_update_request: 133 callbacks suppressed [20087.000636] blk_update_request: critical target error, dev sdh, sector 0 [20087.008139] blk_update_request: critical target error, dev dm-1, sector 0 [20087.015727] Buffer I/O error on dev dm-1, logical block 0, async page read [20087.525001] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20087.535334] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20087.542936] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20087.549659] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [20087.559577] blk_update_request: critical target error, dev sdh, sector 24 [20087.567226] blk_update_request: critical target error, dev dm-1, sector 24 [20087.574933] Buffer I/O error on dev dm-1, logical block 3, async page read [20088.094706] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20088.105030] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20088.112622] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20088.119349] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [20088.129258] blk_update_request: critical target error, dev sdh, sector 6986547072 [20088.137647] blk_update_request: critical target error, dev dm-1, sector 6986547072 [20088.647360] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20088.657698] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20088.665312] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20088.672034] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [20088.681946] blk_update_request: critical target error, dev sdh, sector 6986547072 [20088.690348] blk_update_request: critical target error, dev dm-1, sector 6986547072 [20088.698810] Buffer I/O error on dev dm-1, logical block 873318384, async page read [20089.214077] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20089.224399] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20089.231983] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20089.238706] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [20089.248630] blk_update_request: critical target error, dev sdh, sector 0 [20089.256141] blk_update_request: critical target error, dev dm-1, sector 0 [20089.765608] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20089.775930] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20089.783516] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20089.790233] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20089.800186] Buffer I/O error on dev dm-1, logical block 0, async page read [20090.308795] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20090.319113] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20090.326698] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20090.333422] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [20090.844084] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20090.854407] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20090.861999] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20090.868710] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [20090.878660] Buffer I/O error on dev dm-1, logical block 873318399, async page read [20091.388299] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20091.398621] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20091.406206] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20091.412923] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20091.422873] Buffer I/O error on dev dm-1, logical block 0, async page read [20091.927919] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [20091.937454] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20091.947766] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20091.955352] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20091.962068] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [20091.972298] Buffer I/O error on dev dm-1, logical block 0, async page read [20092.484368] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [20092.494685] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [20092.502278] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [20092.508988] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [20092.518895] blk_update_request: 10 callbacks suppressed [20092.524727] blk_update_request: critical target error, dev sdh, sector 24 [20092.532852] blk_update_request: critical target error, dev dm-1, sector 24 [20092.540584] Buffer I/O error on dev dm-1, logical block 3, async page read [20106.441599] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e818726000, cur 1651655237 expire 1651655087 last 1651655010 [20107.352696] Lustre: MGS: haven't heard from client 2d17adbe-08f3-468b-b9e3-3494dfdac5b3 (at 192.168.1.109@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e5b5523400, cur 1651655238 expire 1651655088 last 1651655010 [20118.769773] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [20159.738184] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.107@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [20159.758039] LustreError: Skipped 839 previous similar messages [20166.326630] LDISKFS-fs (dm-0): recovery complete [20166.332330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [20168.843465] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.120@o2ib (not set up) [20169.329832] Lustre: soaked-MDT0000: Received LWP connection from 0@lo, removing former export from 192.168.1.109@o2ib [20169.338519] Lustre: soaked-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [20169.342672] Lustre: soaked-MDT0001: in recovery but waiting for the first client to connect [20169.410362] Lustre: soaked-MDT0001: Will be in recovery for at least 2:30, or until 22 clients reconnect [20220.375840] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.108@o2ib (at 0@lo) [20220.377523] Lustre: soaked-MDT0001: Recovery over after 0:51, of 22 clients 22 recovered and 0 were evicted. [20221.905877] Lustre: Failing over soaked-MDT0001 [20221.996382] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.122@o2ib (stopping) [20222.008582] LustreError: 36531:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) soaked-OST0009-osc-MDT0001: cannot cleanup orphans: rc = -5 [20222.023267] LustreError: 36531:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [20224.004917] LustreError: 3512:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff99ebf580ad00 x1731865914476544/t0(0) o41->soaked-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [20224.029236] LustreError: 3512:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message [20224.433939] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20224.433942] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20224.433970] Lustre: Skipped 5 previous similar messages [20224.459449] Lustre: Skipped 1 previous similar message [20224.916983] LustreError: 3517:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff99eb47cdd100 x1731865914477440/t0(0) o41->soaked-MDT0003-osp-MDT0001@192.168.1.111@o2ib:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [20225.413215] LustreError: 11-0: soaked-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [20225.424418] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20226.923954] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.136@o2ib (stopping) [20226.933771] Lustre: Skipped 8 previous similar messages [20227.883710] LustreError: 36836:0:(ldlm_resource.c:1124:ldlm_resource_complain()) mdt-soaked-MDT0001_UUID: namespace resource [0x240002353:0x1cd4c:0x0].0x54cb7172 (ffff99ec6452dc80) refcount nonzero (1) after lock cleanup; forcing cleanup. [20228.193029] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20228.705026] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20228.717677] LustreError: Skipped 1 previous similar message [20229.724097] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20229.736743] LustreError: Skipped 3 previous similar messages [20231.743198] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20231.755838] LustreError: Skipped 7 previous similar messages [20231.938373] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.117@o2ib (stopping) [20231.948217] Lustre: Skipped 34 previous similar messages [20235.763389] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20235.776035] LustreError: Skipped 15 previous similar messages [20240.452653] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.110@o2ib (stopping) [20240.462519] Lustre: Skipped 65 previous similar messages [20243.782715] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20243.795371] LustreError: Skipped 31 previous similar messages [20256.873251] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.126@o2ib (stopping) [20256.883094] Lustre: Skipped 109 previous similar messages [20259.804443] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20259.817084] LustreError: Skipped 63 previous similar messages [20278.499861] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20278.522587] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) Skipped 1039 previous similar messages [20282.521971] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20282.544696] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) Skipped 959 previous similar messages [20289.569510] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.105@o2ib (stopping) [20289.579338] Lustre: Skipped 248 previous similar messages [20290.986148] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [20291.008877] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 1139 previous similar messages [20291.826893] LustreError: 0-0: Forced cleanup waiting for mdt-soaked-MDT0001_UUID namespace with 6 resources in use, (rc=-110) [20291.839540] LustreError: Skipped 127 previous similar messages [20320.719349] LustreError: 3776:0:(lod_qos.c:115:lod_statfs_and_check()) soaked-MDT0001-mdtlov: statfs: rc = -108 [20320.731100] LustreError: 3776:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff99ec4190e800 ns: mdt-soaked-MDT0001_UUID lock: ffff99e798808d80/0x322afadd04c8f0d3 lrc: 3/0,0 mode: CR/CR res: [0x24000235e:0x5575:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.1.136@o2ib remote: 0x527c91d1be39cad9 expref: 3 pid: 3776 timeout: 0 lvb_type: 0 [20320.770586] Lustre: 3776:0:(service.c:2327:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (13/87s); client may timeout req@ffff99e7998f1f80 x1731763240218368/t25769805658(0) o101->d85fd948-cdf6-4a2e-8806-76ec090bacdc@192.168.1.136@o2ib:754/0 lens 1304/600 e 0 to 0 dl 1651655364 ref 1 fl Complete:/0/0 rc -19/-19 job:'' [20321.441237] LustreError: 3564:0:(lod_qos.c:115:lod_statfs_and_check()) soaked-MDT0001-mdtlov: statfs: rc = -108 [20321.452526] LustreError: 3564:0:(lod_qos.c:115:lod_statfs_and_check()) Skipped 83 previous similar messages [20321.463706] LustreError: 3564:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff99ec4190e400 ns: mdt-soaked-MDT0001_UUID lock: ffff99ebe85ea640/0x322afadd04c9c397 lrc: 3/0,0 mode: CR/CR res: [0x24000234d:0x11022:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.1.137@o2ib remote: 0x6ba7b49558674dbf expref: 3 pid: 3564 timeout: 0 lvb_type: 0 [20321.503200] LustreError: 3564:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [20321.514435] Lustre: 3564:0:(service.c:2327:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/81s); client may timeout req@ffff99ec3c9cc800 x1731763737640064/t25769805661(0) o101->cb2a3a98-2672-400e-b628-aaad60526b53@192.168.1.137@o2ib:6/0 lens 1304/600 e 0 to 0 dl 1651655371 ref 1 fl Complete:/0/0 rc -19/-19 job:'' [20332.860212] Lustre: server umount soaked-MDT0001 complete [20402.199412] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [20433.977365] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 18 seconds [20433.989612] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.111@o2ib (20): c: 7, oc: 0, rc: 8 [20434.003360] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651655545/real 1651655564] req@ffff99e7bb524800 x1731865920634304/t0(0) o41->soaked-MDT0003-osp-MDT0000@192.168.1.111@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651655589 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [20434.036501] Lustre: soaked-MDT0003-osp-MDT0000: Connection to soaked-MDT0003 (at 192.168.1.111@o2ib) was lost; in progress operations using this service will wait for recovery to complete [20437.977540] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.111@o2ib: 6 seconds [20437.988903] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 7 previous similar messages [20627.423556] Lustre: MGS: haven't heard from client c4de137e-ae46-4871-918a-a490e9e834d3 (at 192.168.1.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e5b54db800, cur 1651655758 expire 1651655608 last 1651655531 [20640.574836] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e818726800, cur 1651655771 expire 1651655621 last 1651655544 [20740.630155] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.111@o2ib, keep former export from same NID [20811.500672] Lustre: soaked-MDT0003-osp-MDT0000: Connection restored to 192.168.1.111@o2ib (at 192.168.1.111@o2ib) [21339.418695] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21339.441443] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) Skipped 1599 previous similar messages [21341.847396] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21341.870195] LustreError: 3772:0:(out_handler.c:910:out_tx_end()) Skipped 179 previous similar messages [21345.854415] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21345.877170] LustreError: 7672:0:(out_handler.c:910:out_tx_end()) Skipped 459 previous similar messages [21353.912541] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21353.935274] LustreError: 7670:0:(out_handler.c:910:out_tx_end()) Skipped 959 previous similar messages [21370.059057] LustreError: 34266:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [21370.081913] LustreError: 34266:0:(out_handler.c:910:out_tx_end()) Skipped 719 previous similar messages [22137.376555] LustreError: 5290:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99ec644d6400 [22137.387546] LustreError: 5290:0:(llog_cat.c:604:llog_cat_add_rec()) Skipped 2 previous similar messages [22137.398084] LustreError: 5290:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0003-osp-MDT0000: write updates failed: rc = -116 [22137.411209] LustreError: 5290:0:(update_trans.c:1062:top_trans_stop()) Skipped 2 previous similar messages [22138.279594] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [22138.293980] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1235 previous similar messages [22138.305466] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [22138.320331] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1235 previous similar messages [22157.231150] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [22157.245541] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 519 previous similar messages [22157.256953] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [22157.271820] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 519 previous similar messages [22446.657561] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657570/real 0] req@ffff99e7a140e780 x1731866015376256/t0(0) o13->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657577 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22446.689230] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22447.618612] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657571/real 0] req@ffff99e7a140ec00 x1731866015436992/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657578 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22447.650272] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22449.924750] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651657573/real 0] req@ffff99ec3cc3cc80 x1731866015509312/t0(0) o13->soaked-OST000a-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651657580 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [22449.925723] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [22449.975086] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [22456.068997] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [22456.081253] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (18): c: 0, oc: 0, rc: 8 [22662.392489] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e8119c5000, cur 1651657793 expire 1651657643 last 1651657566 [22663.896838] Lustre: MGS: haven't heard from client 4c64ca59-6fb9-4c18-a09c-2e3212225e10 (at 192.168.1.102@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99ec41f2f000, cur 1651657794 expire 1651657644 last 1651657566 [22663.920554] Lustre: Skipped 3 previous similar messages [22738.726954] perf: interrupt took too long (3398 > 3255), lowering kernel.perf_event_max_sample_rate to 58000 [22776.083497] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [22776.094880] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [22844.086578] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 3 seconds [22844.097948] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 20 previous similar messages [22976.092605] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [22976.103999] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 48 previous similar messages [23054.591961] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23195.172669] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23245.998801] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.107@o2ib (at 192.168.1.107@o2ib) [23303.759245] LustreError: 11-0: soaked-OST0002-osc-MDT0000: operation ost_destroy to node 192.168.1.107@o2ib failed: rc = -19 [23303.759462] Lustre: soaked-OST0002-osc-MDT0000: Connection to soaked-OST0002 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23303.759465] Lustre: Skipped 1 previous similar message [23303.796192] LustreError: Skipped 3 previous similar messages [23303.934052] LustreError: 3608:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST0002-osc-MDT0000: can't precreate: rc = -19 [23303.947019] LustreError: 3608:0:(osp_precreate.c:1339:osp_precreate_thread()) soaked-OST0002-osc-MDT0000: cannot precreate objects: rc = -19 [23321.137063] LustreError: 11-0: soaked-OST000a-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23321.149611] LustreError: Skipped 1 previous similar message [23321.155876] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23322.657135] LustreError: 11-0: soaked-OST0006-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23322.669744] Lustre: soaked-OST0006-osc-MDT0000: Connection to soaked-OST0006 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23328.338954] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [23328.350453] Lustre: Skipped 1 previous similar message [23330.401476] LustreError: 11-0: soaked-OST000e-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 [23330.414079] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete [23697.284927] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23697.299324] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 307 previous similar messages [23697.310733] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23697.325593] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 307 previous similar messages [23702.125552] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23702.139944] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 36 previous similar messages [23702.151245] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23702.166105] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 36 previous similar messages [23714.425816] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [23714.440231] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 46 previous similar messages [23714.451522] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [23714.466378] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 46 previous similar messages [23735.925214] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [23735.939446] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 205 previous similar messages [23735.950837] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [23735.965543] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 205 previous similar messages [23775.722026] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [23775.736237] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 400 previous similar messages [23775.747620] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [23775.762301] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 400 previous similar messages [24584.011761] Lustre: 3599:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651659702/real 0] req@ffff99eb79a46300 x1731866074594816/t0(0) o1000->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:24/4 lens 488/4320 e 0 to 1 dl 1651659714 ref 3 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [24584.043826] Lustre: soaked-MDT0002-osp-MDT0000: Connection to soaked-MDT0002 (at 192.168.1.110@o2ib) was lost; in progress operations using this service will wait for recovery to complete [24584.740796] Lustre: 3562:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651659703/real 0] req@ffff99e799e9cc80 x1731866074600832/t0(0) o101->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:24/4 lens 328/344 e 0 to 1 dl 1651659715 ref 3 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [24584.772647] Lustre: 3562:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [24591.162074] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: tx_queue(WSQ:001), 19 seconds [24591.174131] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.110@o2ib (24): c: 0, oc: 0, rc: 8 [24591.187909] Lustre: 3502:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659702/real 1651659721] req@ffff99e822bc5100 x1731866074597376/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659749 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24595.162262] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 7 seconds [24595.173618] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [24599.162504] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659707/real 1651659729] req@ffff99ebf7d9d100 x1731866074612608/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659754 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24599.195812] Lustre: 3505:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [24603.162673] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651659707/real 1651659733] req@ffff99ec3e1e3f00 x1731866074614272/t0(0) o103->soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651659754 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [24603.195975] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [24627.163621] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 0 seconds [24627.174985] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 27 previous similar messages [24703.166904] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 2 seconds [24703.178278] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [24772.878865] ptlrpc_watchdog_fire: 3 callbacks suppressed [24772.878881] Lustre: mdt00_007: service thread pid 3780 was inactive for 200.722 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [24772.878884] Lustre: Skipped 3 previous similar messages [24772.906970] Lustre: mdt00_010: service thread pid 5287 was inactive for 200.132 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [24772.927854] Pid: 5287, comm: mdt00_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [24772.938835] Call Trace: [24772.941667] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [24772.947466] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [24772.953372] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [24772.959241] [<0>] osp_md_object_lock+0x160/0x300 [osp] [24772.965019] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [24772.970498] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [24772.975813] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [24772.981963] [<0>] mdt_reint_unlink+0x6d5/0x1b80 [mdt] [24772.987629] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [24772.992810] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [24772.998563] [<0>] mdt_reint+0x67/0x150 [mdt] [24773.003389] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [24773.009584] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [24773.016625] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [24773.022078] [<0>] kthread+0xd1/0xe0 [24773.025978] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [24773.031758] [<0>] 0xfffffffffffffffe [24788.598118] Lustre: MGS: haven't heard from client fe865eff-e137-4afe-a54d-f44c908c3b80 (at 192.168.1.110@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e5b54dd800, cur 1651659919 expire 1651659769 last 1651659692 [24795.607991] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0002-mdtlov_UUID (at 192.168.1.110@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e818726400, cur 1651659926 expire 1651659776 last 1651659697 [25050.142889] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.110@o2ib, keep former export from same NID [25116.355826] Lustre: soaked-MDT0002-osp-MDT0000: Connection restored to 192.168.1.110@o2ib (at 192.168.1.110@o2ib) [25116.367317] Lustre: Skipped 3 previous similar messages [26562.219812] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26562.234210] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 15 previous similar messages [26562.245508] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26562.260375] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 15 previous similar messages [26573.478456] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26573.492866] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 99 previous similar messages [26573.504153] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26573.519021] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 99 previous similar messages [26593.502229] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26593.516630] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 203 previous similar messages [26593.528007] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26593.542897] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 203 previous similar messages [26631.365225] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [26631.379626] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 288 previous similar messages [26631.391028] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [26631.405900] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 288 previous similar messages [26635.159877] Lustre: 3499:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651661758/real 0] req@ffff99ec41c99200 x1731866138744320/t0(0) o13->soaked-OST000e-osc-MDT0000@192.168.1.102@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651661765 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [26635.191530] Lustre: 3499:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [26635.202385] Lustre: soaked-OST000e-osc-MDT0000: Connection to soaked-OST000e (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [26635.876892] Lustre: soaked-OST000a-osc-MDT0000: Connection to soaked-OST000a (at 192.168.1.102@o2ib) was lost; in progress operations using this service will wait for recovery to complete [26635.895558] Lustre: Skipped 2 previous similar messages [26644.251267] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [26644.263527] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.102@o2ib (20): c: 0, oc: 0, rc: 8 [26644.277249] Lustre: 3491:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651661756/real 1651661774] req@ffff99e7ca7a2d00 x1731866138588672/t0(0) o400->soaked-OST0002-osc-MDT0000@192.168.1.102@o2ib:28/4 lens 224/224 e 0 to 1 dl 1651661800 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'' [26644.277724] LustreError: 3632:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST000e-osc-MDT0000: can't precreate: rc = -5 [26644.323385] Lustre: 3491:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [26836.232335] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0002_UUID (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80db19000, cur 1651661966 expire 1651661816 last 1651661739 [26842.778052] Lustre: MGS: haven't heard from client 260df422-b357-49b9-b86c-e462e7402cb7 (at 192.168.1.102@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e80f28d000, cur 1651661973 expire 1651661823 last 1651661746 [26842.801776] Lustre: Skipped 3 previous similar messages [26952.264781] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 0 seconds [26952.276146] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 5 previous similar messages [26968.265451] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 7 seconds [26968.276825] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [27000.266849] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 10 seconds [27000.278332] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 11 previous similar messages [27064.269695] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.102@o2ib: 36 seconds [27064.281167] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 29 previous similar messages [27522.513349] Lustre: soaked-OST0002-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27557.319037] Lustre: soaked-OST000a-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27602.603420] Lustre: soaked-OST0006-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [27647.651494] Lustre: soaked-OST000e-osc-MDT0000: Connection restored to 192.168.1.102@o2ib (at 192.168.1.102@o2ib) [29638.927715] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [29638.950452] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 1379 previous similar messages [30046.955568] Lustre: 3500:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665170/real 0] req@ffff99e7a2ac7980 x1731866190919168/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665177 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30046.987259] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30048.037621] Lustre: 3502:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665171/real 0] req@ffff99e7ca7a3a80 x1731866191046080/t0(0) o13->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665178 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30048.069297] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30050.985748] Lustre: 3521:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665172/real 0] req@ffff99ec3d7e1680 x1731866191099072/t0(0) o13->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665179 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30050.985753] Lustre: 3517:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651665171/real 0] req@ffff99ebef1e5580 x1731866191007168/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651665178 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [30050.985775] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [30056.401033] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [30056.413295] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (19): c: 0, oc: 0, rc: 8 [30056.427197] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651665171/real 1651665186] req@ffff99e7a1106780 x1731866191028416/t0(0) o6->soaked-OST0008-osc-MDT0000@192.168.1.104@o2ib:28/4 lens 544/432 e 0 to 1 dl 1651665258 ref 1 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [30056.460217] Lustre: 3493:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [30243.056606] Lustre: MGS: haven't heard from client 4ae9cc45-77ab-4a37-98e1-c34a7d3e0402 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e813b12000, cur 1651665373 expire 1651665223 last 1651665145 [30324.412772] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 1 seconds [30324.424131] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 61 previous similar messages [30340.413485] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 12 seconds [30340.424952] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 11 previous similar messages [30372.414897] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 44 seconds [30372.426361] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 15 previous similar messages [30436.417709] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 108 seconds [30436.429278] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [30699.393960] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30710.046931] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30757.501971] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [30803.277455] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [31695.412547] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31695.435284] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 31 previous similar messages [31696.244123] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31696.266843] LustreError: 3573:0:(out_handler.c:910:out_tx_end()) Skipped 83 previous similar messages [31697.316034] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31697.338775] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 35 previous similar messages [31699.387208] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31699.409956] LustreError: 7834:0:(out_handler.c:910:out_tx_end()) Skipped 219 previous similar messages [31707.957541] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31707.980308] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 319 previous similar messages [31716.206641] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31716.229370] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 1299 previous similar messages [31732.263752] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [31732.286471] LustreError: 7629:0:(out_handler.c:910:out_tx_end()) Skipped 1179 previous similar messages [31931.407319] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [31931.421712] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 235 previous similar messages [31931.433093] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [31931.447971] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 235 previous similar messages [31941.143574] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [31941.157777] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 89 previous similar messages [31941.169068] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [31941.183751] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 89 previous similar messages [31961.222111] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [31961.236295] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 262 previous similar messages [31961.247665] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [31961.262326] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 262 previous similar messages [32001.067231] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [32001.081445] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 586 previous similar messages [32001.092850] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [32001.107517] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 586 previous similar messages [32879.391297] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32879.414033] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 999 previous similar messages [32883.447814] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32883.470569] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 359 previous similar messages [32891.493754] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32891.516482] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) Skipped 1419 previous similar messages [32907.494860] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [32907.517591] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 1659 previous similar messages [33891.195764] perf: interrupt took too long (4271 > 4247), lowering kernel.perf_event_max_sample_rate to 46000 [33981.104631] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33981.127357] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 699 previous similar messages [33985.389464] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33985.412204] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 667 previous similar messages [33994.813534] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [33994.836280] LustreError: 6643:0:(out_handler.c:910:out_tx_end()) Skipped 991 previous similar messages [34103.193256] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34103.207660] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 394 previous similar messages [34103.219050] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34103.233911] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 394 previous similar messages [34112.688305] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34112.702703] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 128 previous similar messages [34112.714107] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34112.728975] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 128 previous similar messages [34131.471483] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34131.485869] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 269 previous similar messages [34131.497243] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34131.512106] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 269 previous similar messages [34174.377080] Lustre: 3503:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651669297/real 0] req@ffff99e7a2d60000 x1731866385351744/t0(0) o103->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:17/18 lens 328/224 e 0 to 1 dl 1651669304 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [34174.409073] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [34174.427735] Lustre: Skipped 1 previous similar message [34177.219180] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651669300/real 0] req@ffff99e7a1e0cc80 x1731866385354560/t0(0) o400->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/224 e 0 to 1 dl 1651669307 ref 2 fl Rpc:XNr/0/ffffffff rc 0/-1 job:'' [34177.251138] Lustre: 3504:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [34185.083471] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.125@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34185.103287] LustreError: Skipped 191 previous similar messages [34185.583489] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 18 seconds [34185.595732] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (18): c: 2, oc: 0, rc: 8 [34217.106934] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.120@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34217.126761] LustreError: Skipped 150 previous similar messages [34281.779358] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34281.799186] LustreError: Skipped 390 previous similar messages [34352.022230] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34352.535871] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34352.546393] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34353.059581] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34353.070099] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34353.579957] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34353.590499] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34354.100502] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34354.111017] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34354.624230] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34354.634753] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34355.150202] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34355.160718] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34355.676804] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34355.687332] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34356.210282] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [34356.220797] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [34356.332611] sd 0:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s [34356.342940] sd 0:0:0:0: [sdc] tag#0 Sense Key : Illegal Request [current] [34356.350632] sd 0:0:0:0: [sdc] tag#0 <>ASC=0x94 ASCQ=0x1 [34356.357347] sd 0:0:0:0: [sdc] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 a0 45 e0 00 00 00 f8 00 00 [34356.367360] blk_update_request: I/O error, dev sdc, sector 10503648 [34356.374405] device-mapper: multipath: Failing path 8:32. [34356.500803] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [34357.016524] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34357.026858] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34357.034451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34357.041168] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34357.051096] blk_update_request: critical target error, dev sdh, sector 6986547072 [34357.059491] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34357.071164] device-mapper: multipath: Failing path 8:32. [34357.593607] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34357.603907] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34357.611504] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34357.618222] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34357.628132] blk_update_request: critical target error, dev sdh, sector 6986547072 [34357.636542] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34357.645033] Buffer I/O error on dev dm-1, logical block 873318384, async page read [34358.160350] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34358.170669] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34358.178257] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34358.184983] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [34358.194908] blk_update_request: critical target error, dev sdh, sector 0 [34358.202434] blk_update_request: critical target error, dev dm-1, sector 0 [34358.718829] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34358.729141] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34358.736745] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34358.743466] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34358.753379] blk_update_request: critical target error, dev sdh, sector 0 [34358.760897] blk_update_request: critical target error, dev dm-1, sector 0 [34358.768501] Buffer I/O error on dev dm-1, logical block 0, async page read [34359.277144] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34359.287463] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34359.295057] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34359.301769] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34359.311687] blk_update_request: critical target error, dev sdh, sector 6986547192 [34359.827163] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34359.837486] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34359.845079] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34359.851792] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34359.861741] Buffer I/O error on dev dm-1, logical block 873318399, async page read [34360.377194] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34360.387515] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34360.395110] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34360.401834] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34360.411784] Buffer I/O error on dev dm-1, logical block 0, async page read [34360.922243] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34360.932561] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34360.940148] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34360.946858] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34360.956790] Buffer I/O error on dev dm-1, logical block 0, async page read [34361.078071] device-mapper: multipath: Reinstating path 8:32. [34361.129878] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, queueing MODE_SELECT command [34361.466169] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34361.476486] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34361.484079] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34361.490795] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [34361.500710] blk_update_request: 7 callbacks suppressed [34361.506453] blk_update_request: critical target error, dev sdh, sector 24 [34361.514075] blk_update_request: critical target error, dev dm-1, sector 24 [34361.521791] Buffer I/O error on dev dm-1, logical block 3, async page read [34362.042252] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34362.052599] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34362.060196] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34362.066905] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34362.076822] blk_update_request: critical target error, dev sdh, sector 6986547072 [34362.085230] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34362.503808] sd 0:0:0:0: rdac: array soak-netapp2624-1, ctlr 0, MODE_SELECT completed [34362.595955] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34362.606270] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34362.613874] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34362.620589] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [34362.630506] blk_update_request: critical target error, dev sdh, sector 6986547072 [34362.638926] blk_update_request: critical target error, dev dm-1, sector 6986547072 [34362.647412] Buffer I/O error on dev dm-1, logical block 873318384, async page read [34363.170379] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34363.180699] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34363.188285] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34363.194998] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [34363.204908] blk_update_request: critical target error, dev sdh, sector 0 [34363.212430] blk_update_request: critical target error, dev dm-1, sector 0 [34363.727299] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34363.737607] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34363.745200] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34363.751912] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34363.761821] blk_update_request: critical target error, dev sdh, sector 0 [34363.770327] blk_update_request: critical target error, dev dm-1, sector 0 [34363.777923] Buffer I/O error on dev dm-1, logical block 0, async page read [34364.293970] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34364.304284] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34364.311869] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34364.318591] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34364.829346] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34364.839649] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34364.847242] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34364.853953] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [34364.863889] Buffer I/O error on dev dm-1, logical block 873318399, async page read [34365.377421] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34365.387752] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34365.395344] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34365.402054] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34365.412403] Buffer I/O error on dev dm-1, logical block 0, async page read [34365.927407] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34365.937722] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34365.945326] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34365.952047] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [34365.961991] Buffer I/O error on dev dm-1, logical block 0, async page read [34366.477501] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [34366.487810] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [34366.495406] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [34366.502117] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [34366.512019] blk_update_request: 8 callbacks suppressed [34366.517771] blk_update_request: critical target error, dev sdh, sector 24 [34366.525383] blk_update_request: critical target error, dev dm-1, sector 24 [34366.533074] Buffer I/O error on dev dm-1, logical block 3, async page read [34368.179574] Lustre: mdt01_002: service thread pid 3565 was inactive for 200.635 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [34368.200472] Pid: 3565, comm: mdt01_002 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [34368.211474] Call Trace: [34368.214349] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [34368.220162] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [34368.226079] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [34368.231943] [<0>] osp_md_object_lock+0x160/0x300 [osp] [34368.237729] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [34368.243205] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [34368.248510] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [34368.254670] [<0>] mdt_reint_unlink+0x6d5/0x1b80 [mdt] [34368.260334] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [34368.265533] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [34368.271298] [<0>] mdt_reint+0x67/0x150 [mdt] [34368.276152] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [34368.282351] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [34368.289417] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [34368.294876] [<0>] kthread+0xd1/0xe0 [34368.298781] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [34368.304550] [<0>] 0xfffffffffffffffe [34389.509972] Lustre: MGS: haven't heard from client b7526d44-6fcc-4133-a928-8ab0781133c5 (at 192.168.1.109@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff99e7a165f000, cur 1651669519 expire 1651669369 last 1651669289 [34389.533684] Lustre: Skipped 4 previous similar messages [34398.388859] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [34409.865775] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.123@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [34409.885592] LustreError: Skipped 816 previous similar messages [34440.979962] LDISKFS-fs (dm-0): recovery complete [34440.985672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [34442.715058] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.104@o2ib (not set up) [34442.725095] Lustre: Skipped 316 previous similar messages [34442.815341] Lustre: soaked-MDT0000: Received MDS connection from 0@lo, removing former export from 192.168.1.109@o2ib [34442.973240] Lustre: soaked-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [34442.993193] Lustre: soaked-MDT0001: in recovery but waiting for the first client to connect [34443.000237] Lustre: soaked-MDT0001: Will be in recovery for at least 2:30, or until 22 clients reconnect [34448.203256] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [34448.217492] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 533 previous similar messages [34448.228891] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [34448.243579] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 533 previous similar messages [34458.152653] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.108@o2ib (at 0@lo) [34458.153363] Lustre: soaked-MDT0001: Recovery over after 0:15, of 22 clients 22 recovered and 0 were evicted. [34460.002987] Lustre: Failing over soaked-MDT0001 [34460.035786] Lustre: soaked-MDT0001: Not available for connect from 192.168.1.110@o2ib (stopping) [34460.045605] Lustre: Skipped 4 previous similar messages [34460.052751] LustreError: 60248:0:(osp_precreate.c:676:osp_precreate_send()) soaked-OST0001-osc-MDT0001: can't precreate: rc = -5 [34460.053708] LustreError: 60250:0:(osp_precreate.c:966:osp_precreate_cleanup_orphans()) soaked-OST0002-osc-MDT0001: cannot cleanup orphans: rc = -5 [34460.056380] LustreError: 60256:0:(osp_precreate.c:1339:osp_precreate_thread()) soaked-OST0005-osc-MDT0001: cannot precreate objects: rc = -5 [34460.094414] LustreError: 60248:0:(osp_precreate.c:676:osp_precreate_send()) Skipped 1 previous similar message [34460.205679] Lustre: server umount soaked-MDT0001 complete [34463.192063] LustreError: 11-0: soaked-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [34463.203318] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [34489.235172] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [34528.533647] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [34528.548035] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 189 previous similar messages [34528.559413] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0003-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [34528.574294] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 189 previous similar messages [34858.612152] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [34858.624410] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.111@o2ib (19): c: 0, oc: 0, rc: 8 [34858.638231] Lustre: 3560:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1651669972/real 1651669988] req@ffff99e7a173b600 x1731866403973888/t0(0) o101->soaked-MDT0003-osp-MDT0000@192.168.1.111@o2ib:24/4 lens 328/344 e 0 to 1 dl 1651670018 ref 2 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [34858.638285] Lustre: soaked-MDT0003-osp-MDT0000: Connection to soaked-MDT0003 (at 192.168.1.111@o2ib) was lost; in progress operations using this service will wait for recovery to complete [34858.690109] Lustre: 3560:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [35035.856950] Lustre: mdt00_003: service thread pid 3654 was inactive for 200.354 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [35035.856953] Lustre: mdt00_011: service thread pid 5288 was inactive for 200.350 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35035.856956] Pid: 3561, comm: mdt00_001 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35035.856959] Lustre: Skipped 11 previous similar messages [35035.856960] Call Trace: [35035.857066] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35035.857130] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35035.857190] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35035.857208] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35035.857232] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35035.857253] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35035.857286] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35035.857312] [<0>] mdt_create+0xc39/0xe40 [mdt] [35035.857336] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35035.857361] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35035.857382] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35035.857404] [<0>] mdt_reint+0x67/0x150 [mdt] [35035.857487] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35035.857557] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35035.857625] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35035.857631] [<0>] kthread+0xd1/0xe0 [35035.857637] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35035.857667] [<0>] 0xfffffffffffffffe [35035.857671] Pid: 3781, comm: mdt00_008 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35035.857672] Call Trace: [35035.857762] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35035.857827] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35035.857887] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35035.857915] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35035.857936] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35035.857953] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35035.857981] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35035.858006] [<0>] mdt_create+0xc39/0xe40 [mdt] [35035.858030] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35035.858055] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35035.858076] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35035.858098] [<0>] mdt_reint+0x67/0x150 [mdt] [35035.858177] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35035.858248] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35035.858316] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35035.858320] [<0>] kthread+0xd1/0xe0 [35035.858324] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35035.858336] [<0>] 0xfffffffffffffffe [35036.124077] Lustre: Skipped 2 previous similar messages [35036.129922] Pid: 3654, comm: mdt00_003 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35036.140908] Call Trace: [35036.143699] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35036.149480] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35036.155355] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35036.161210] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35036.166971] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35036.172436] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35036.177715] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35036.183863] [<0>] mdt_create+0xc39/0xe40 [mdt] [35036.188849] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35036.194423] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35036.199593] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35036.205340] [<0>] mdt_reint+0x67/0x150 [mdt] [35036.210157] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35036.216329] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35036.223361] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35036.228816] [<0>] kthread+0xd1/0xe0 [35036.232717] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35036.238465] [<0>] 0xfffffffffffffffe [35051.427856] Lustre: MGS: haven't heard from client 0eb36178-2f47-447e-9910-a57e14c3ddf8 (at 192.168.1.111@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff99e80f15d000, cur 1651670181 expire 1651670031 last 1651669951 [35076.306738] Lustre: mdt00_006: service thread pid 3776 was inactive for 200.350 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35076.323061] Lustre: Skipped 2 previous similar messages [35175.961125] LustreError: 3776:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651670006, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99e7a2263840/0x322afadd0daeda29 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1:0x0].0x0 bits 0x13/0x0 rrc: 17 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 3776 timeout: 0 lvb_type: 0 [35176.005968] LustreError: dumping log to /tmp/lustre-log.1651670306.3776 [35371.690405] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.111@o2ib, keep former export from same NID [35429.090235] ptlrpc_watchdog_fire: 4 callbacks suppressed [35429.090244] Lustre: mdt01_004: service thread pid 3769 was inactive for 593.566 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [35429.090246] Lustre: Skipped 4 previous similar messages [35429.118327] Lustre: mdt01_027: service thread pid 35261 was inactive for 593.572 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [35429.139312] Pid: 35261, comm: mdt01_027 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [35429.150403] Call Trace: [35429.153266] [<0>] ptlrpc_set_wait+0x4b2/0x830 [ptlrpc] [35429.159064] [<0>] ptlrpc_queue_wait+0x86/0x250 [ptlrpc] [35429.164961] [<0>] ldlm_cli_enqueue+0x424/0xa70 [ptlrpc] [35429.170820] [<0>] osp_md_object_lock+0x160/0x300 [osp] [35429.176611] [<0>] lod_object_lock+0x7be/0x7d0 [lod] [35429.182093] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [35429.187385] [<0>] mdt_reint_striped_lock+0x3af/0x620 [mdt] [35429.193553] [<0>] mdt_create+0xc39/0xe40 [mdt] [35429.198537] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [35429.204112] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [35429.209304] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [35429.215063] [<0>] mdt_reint+0x67/0x150 [mdt] [35429.219928] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [35429.226113] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [35429.233153] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [35429.238629] [<0>] kthread+0xd1/0xe0 [35429.242532] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [35429.248309] [<0>] 0xfffffffffffffffe [35430.338332] Lustre: 22105:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99e769916300 x1731763694372608/t42961211840(0) o36->24816f25-f323-4fbc-a2f1-9bdf8a245040@192.168.1.128@o2ib:100/0 lens 520/448 e 22 to 0 dl 1651670565 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [35430.370759] Lustre: 22105:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages [35433.496816] Lustre: soaked-MDT0003-osp-MDT0000: Connection restored to 192.168.1.111@o2ib (at 192.168.1.111@o2ib) [35740.376380] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35740.399111] LustreError: 3572:0:(out_handler.c:910:out_tx_end()) Skipped 2479 previous similar messages [35742.408194] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35742.430969] LustreError: 4805:0:(out_handler.c:910:out_tx_end()) Skipped 27 previous similar messages [35756.046954] LustreError: 3768:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35756.069694] LustreError: 3768:0:(out_handler.c:910:out_tx_end()) Skipped 23 previous similar messages [35764.307386] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35764.330128] LustreError: 4874:0:(out_handler.c:910:out_tx_end()) Skipped 19 previous similar messages [35781.120573] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35781.143317] LustreError: 3773:0:(out_handler.c:910:out_tx_end()) Skipped 47 previous similar messages [35813.697516] LustreError: 34307:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [35813.720359] LustreError: 34307:0:(out_handler.c:910:out_tx_end()) Skipped 119 previous similar messages [36944.534477] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [36944.548693] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 366 previous similar messages [36944.560078] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [36944.574771] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 366 previous similar messages [36963.471657] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [36963.486072] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 156 previous similar messages [36963.497464] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [36963.512332] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 156 previous similar messages [37002.806012] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [37002.820440] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 424 previous similar messages [37002.831826] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [37002.846703] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 424 previous similar messages [37049.763738] LustreError: 3767:0:(llog_cat.c:604:llog_cat_add_rec()) llog_write_rec -116: lh=ffff99ec5d353000 [37049.774746] LustreError: 3767:0:(update_trans.c:1062:top_trans_stop()) soaked-MDT0001-osp-MDT0000: write updates failed: rc = -116 [37054.156407] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37054.179143] LustreError: 3571:0:(out_handler.c:910:out_tx_end()) Skipped 75 previous similar messages [37065.402854] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37065.425591] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 7 previous similar messages [37087.198523] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37087.221263] LustreError: 3857:0:(out_handler.c:910:out_tx_end()) Skipped 215 previous similar messages [37120.009273] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37120.032003] LustreError: 3570:0:(out_handler.c:910:out_tx_end()) Skipped 391 previous similar messages [37289.526468] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [37289.540857] LustreError: 3641:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1717 previous similar messages [37289.552371] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) soaked-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [37289.567229] LustreError: 3641:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1717 previous similar messages [37293.106056] LustreError: 22106:0:(llog_osd.c:626:llog_osd_write_rec()) soaked-MDT0001-osp-MDT0000: index 45193 already set in llog bitmap [0x2400032e9:0xee22:0x0] [37293.122294] LustreError: 22106:0:(llog_osd.c:628:llog_osd_write_rec()) LBUG [37293.130105] Pid: 22106, comm: mdt01_020 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37293.141187] Call Trace: [37293.143960] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [37293.149715] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [37293.155149] [<0>] llog_osd_write_rec+0x1a3a/0x1a70 [obdclass] [37293.161588] [<0>] llog_write_rec+0x293/0x590 [obdclass] [37293.167480] [<0>] llog_cat_add_rec+0x1d9/0x980 [obdclass] [37293.173543] [<0>] llog_add+0x182/0x1f0 [obdclass] [37293.178911] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [37293.184929] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [37293.190583] [<0>] lod_trans_stop+0x25c/0x340 [lod] [37293.195961] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [37293.201230] [<0>] mdd_create+0x154a/0x1cd0 [mdd] [37293.206418] [<0>] mdo_create+0x46/0x48 [mdt] [37293.211215] [<0>] mdt_create+0xab1/0xe40 [mdt] [37293.216199] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37293.221766] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37293.226942] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37293.232702] [<0>] mdt_reint+0x67/0x150 [mdt] [37293.237562] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37293.243740] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37293.250788] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37293.256246] [<0>] kthread+0xd1/0xe0 [37293.260154] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37293.265921] [<0>] 0xfffffffffffffffe [37293.270065] LustreError: dumping log to /tmp/lustre-log.1651672423.22106 [37493.565331] Lustre: mdt01_002: service thread pid 3565 was inactive for 200.486 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [37493.565341] Lustre: mdt01_020: service thread pid 22106 was inactive for 200.486 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37493.565345] Lustre: mdt00_001: service thread pid 3561 was inactive for 200.486 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37493.565350] Pid: 22105, comm: mdt00_018 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.565353] Lustre: Skipped 7 previous similar messages [37493.565356] Lustre: Skipped 9 previous similar messages [37493.565358] Call Trace: [37493.565467] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.565527] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.565585] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.565603] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.565628] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.565648] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.565693] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.565714] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.565748] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.565774] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.565799] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.565824] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.565845] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.565867] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.565951] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.566022] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.566089] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.566095] [<0>] kthread+0xd1/0xe0 [37493.566101] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37493.566133] [<0>] 0xfffffffffffffffe [37493.566136] Pid: 8311, comm: mdt00_014 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.566137] Call Trace: [37493.566217] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.566276] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.566334] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.566363] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.566383] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.566400] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.566424] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.566446] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.566467] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.566493] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.566518] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.566543] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.566563] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.566586] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.566664] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.566734] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.566801] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.566805] [<0>] kthread+0xd1/0xe0 [37493.566810] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37493.566821] [<0>] 0xfffffffffffffffe [37493.880610] Lustre: Skipped 2 previous similar messages [37493.886446] Pid: 3565, comm: mdt01_002 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37493.897439] Call Trace: [37493.900259] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [37493.906425] [<0>] ldlm_cli_enqueue_fini+0xa00/0xea0 [ptlrpc] [37493.912775] [<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc] [37493.918630] [<0>] osp_md_object_lock+0x160/0x300 [osp] [37493.924392] [<0>] lod_object_lock+0xe2/0x7d0 [lod] [37493.929777] [<0>] mdd_object_lock+0x30/0xd0 [mdd] [37493.935073] [<0>] mdt_remote_object_lock_try+0x1db/0x520 [mdt] [37493.941599] [<0>] mdt_object_lock_internal+0x19c/0x390 [mdt] [37493.947968] [<0>] mdt_object_lock+0x20/0x30 [mdt] [37493.953234] [<0>] mdt_create+0x6de/0xe40 [mdt] [37493.958232] [<0>] mdt_reint_create+0x3a0/0x460 [mdt] [37493.963801] [<0>] mdt_reint_rec+0x8a/0x240 [mdt] [37493.968971] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [37493.974726] [<0>] mdt_reint+0x67/0x150 [mdt] [37493.979543] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37493.985710] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37493.992754] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37493.998206] [<0>] kthread+0xd1/0xe0 [37494.002141] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37494.007897] [<0>] 0xfffffffffffffffe [37570.368756] Lustre: mdt01_006: service thread pid 3774 was inactive for 200.042 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [37593.083732] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [37593.102425] LustreError: 8311:0:(ldlm_request.c:142:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651672423, 300s ago), entering recovery for soaked-MDT0001_UUID@192.168.1.109@o2ib ns: soaked-MDT0001-osp-MDT0000 lock: ffff99ec3fc2ba80/0x322afadd0e9cc1b7 lrc: 4/0,1 mode: --/EX res: [0x2400032e9:0x182bf:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT gid 0 flags: 0x1000001000000 nid: local remote: 0xe4a46eacd6424a03 expref: -99 pid: 8311 timeout: 0 lvb_type: 0 [37593.115183] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [37593.160264] LustreError: 8311:0:(ldlm_request.c:142:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [37642.387933] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [37642.410665] LustreError: 4806:0:(out_handler.c:910:out_tx_end()) Skipped 359 previous similar messages [37888.094802] Lustre: 22102:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99e815f05e80 x1731763812062080/t0(0) o36->ab8832f6-98e1-4d13-84e2-130e4afa61aa@192.168.1.138@o2ib:293/0 lens 512/448 e 23 to 0 dl 1651673023 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [37888.126252] Lustre: 22102:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [37894.149486] Lustre: soaked-MDT0000: Client a6431ad6-ad1a-403b-b2a6-2dbd7e58ec7b (at 192.168.1.120@o2ib) reconnecting [37894.161268] Lustre: Skipped 9 previous similar messages [37924.293443] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [37924.305715] Lustre: Skipped 1 previous similar message [37955.409744] ptlrpc_watchdog_fire: 6 callbacks suppressed [37955.415720] Lustre: mdt_rdpg01_008: service thread pid 3851 was inactive for 200.490 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [37955.437078] Pid: 3851, comm: mdt_rdpg01_008 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [37955.448544] Call Trace: [37955.451310] [<0>] call_rwsem_down_write_failed+0x17/0x30 [37955.457295] [<0>] llog_cat_add_rec+0x11d/0x980 [obdclass] [37955.463382] [<0>] llog_add+0x182/0x1f0 [obdclass] [37955.468737] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [37955.474786] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [37955.480443] [<0>] lod_trans_stop+0x25c/0x340 [lod] [37955.485844] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [37955.491112] [<0>] mdd_attr_set+0x81c/0x1070 [mdd] [37955.496419] [<0>] mdt_mfd_close+0x771/0xbb0 [mdt] [37955.501694] [<0>] mdt_close_internal+0x141/0x240 [mdt] [37955.507452] [<0>] mdt_close+0x291/0x900 [mdt] [37955.512378] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [37955.518556] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [37955.525598] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [37955.531054] [<0>] kthread+0xd1/0xe0 [37955.534961] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [37955.540744] [<0>] 0xfffffffffffffffe [37965.394235] Lustre: 3787:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb64244380 x1731770926374272/t42961857331(0) o36->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:370/0 lens 552/448 e 5 to 0 dl 1651673100 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [37965.426477] Lustre: 3787:0:(service.c:1436:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages [37971.398203] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [38191.891205] LustreError: 35178:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651673021, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99eb63453a80/0x322afadd0ea7c0ba lrc: 3/1,0 mode: --/PR res: [0x200000408:0x1:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 35178 timeout: 0 lvb_type: 0 [38349.667201] Lustre: 3849:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99eb4792b180 x1731770935641280/t42961884363(0) o35->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:754/0 lens 392/456 e 15 to 0 dl 1651673484 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [38495.242067] Lustre: soaked-MDT0000: Client 7ad30715-59f4-4c17-88bd-2c39bf00253d (at 192.168.1.126@o2ib) reconnecting [38495.253871] Lustre: Skipped 7 previous similar messages [38552.292169] Lustre: 3511:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673675/real 0] req@ffff99ec40036c00 x1731866522217216/t0(0) o13->soaked-OST0004-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673682 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38552.323860] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38553.268167] Lustre: 3513:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673676/real 0] req@ffff99eb4a68ec00 x1731866522217664/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673683 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38553.299848] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38555.396294] Lustre: 3516:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651673678/real 0] req@ffff99eb6bd44380 x1731866522218048/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651673685 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [38555.427951] Lustre: 3516:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [38555.438697] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [38555.457355] Lustre: Skipped 1 previous similar message [38564.775693] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 16 seconds [38564.787945] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (21): c: 0, oc: 0, rc: 8 [38572.449423] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [38572.461218] Lustre: Skipped 1 previous similar message [38753.127007] Lustre: MGS: haven't heard from client 303621b9-bdff-4c16-9341-be1b54272978 (at 192.168.1.104@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff99e79980c800, cur 1651673883 expire 1651673733 last 1651673655 [38765.461979] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0000_UUID (at 192.168.1.104@o2ib) in 240 seconds. I think it's dead, and I am evicting it. exp ffff99e799808400, cur 1651673895 expire 1651673745 last 1651673655 [38816.786825] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 4 seconds [38816.798192] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 57 previous similar messages [38832.787551] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 2 seconds [38832.798917] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 3 previous similar messages [38864.788965] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 29 seconds [38864.800437] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 18 previous similar messages [38928.791790] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 93 seconds [38928.803261] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 31 previous similar messages [39095.172111] Lustre: mdt01_022: service thread pid 35178 was inactive for 1203.240 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [39095.193195] Pid: 35178, comm: mdt01_022 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [39095.204302] Call Trace: [39095.207186] [<0>] ldlm_completion_ast+0x62d/0xa40 [ptlrpc] [39095.213380] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [39095.219878] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [39095.225954] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [39095.232224] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [39095.238382] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [39095.244166] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [39095.249562] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [39095.255282] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [39095.261285] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [39095.267688] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [39095.273003] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [39095.279207] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [39095.286251] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [39095.291712] [<0>] kthread+0xd1/0xe0 [39095.295618] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [39095.301404] [<0>] 0xfffffffffffffffe [39096.334372] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [39096.346160] Lustre: Skipped 4 previous similar messages [39164.561491] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39173.548182] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [39173.559968] Lustre: Skipped 3 previous similar messages [39263.278068] Lustre: soaked-OST0008-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39358.528400] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39404.342605] Lustre: soaked-OST000c-osc-MDT0000: Connection restored to 192.168.1.105@o2ib (at 192.168.1.105@o2ib) [39418.546991] LustreError: 11-0: soaked-OST0008-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39418.559603] Lustre: soaked-OST0008-osc-MDT0000: Connection to soaked-OST0008 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39424.995285] LustreError: 11-0: soaked-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39425.007865] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39427.320069] Lustre: soaked-OST0004-osc-MDT0000: Connection to soaked-OST0004 (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39427.661464] Lustre: soaked-OST0000-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [39429.395456] LustreError: 11-0: soaked-OST000c-osc-MDT0000: operation ost_statfs to node 192.168.1.105@o2ib failed: rc = -107 [39429.408070] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.105@o2ib) was lost; in progress operations using this service will wait for recovery to complete [39588.537777] Lustre: soaked-OST0004-osc-MDT0000: Connection restored to 192.168.1.104@o2ib (at 192.168.1.104@o2ib) [39588.549284] Lustre: Skipped 2 previous similar messages [39697.429122] Lustre: soaked-MDT0000: Client 0f628bfa-3bc1-48b7-a4a4-37175642f976 (at 192.168.1.135@o2ib) reconnecting [39697.440929] Lustre: Skipped 3 previous similar messages [39774.605233] Lustre: soaked-MDT0000: Client ec6e9766-b582-4c68-bc4c-f7d5702b85d9 (at 192.168.1.117@o2ib) reconnecting [39774.617016] Lustre: Skipped 3 previous similar messages [40298.524117] Lustre: soaked-MDT0000: Client d85fd948-cdf6-4a2e-8806-76ec090bacdc (at 192.168.1.136@o2ib) reconnecting [40298.535890] Lustre: Skipped 4 previous similar messages [40899.622432] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [40899.634229] Lustre: Skipped 8 previous similar messages [41273.316599] Lustre: mdt01_017: service thread pid 8314 was inactive for 200.574 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [41273.316841] Pid: 5287, comm: mdt00_010 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41273.316842] Call Trace: [41273.316959] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [41273.317018] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [41273.317049] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [41273.317072] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [41273.317095] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [41273.317117] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [41273.317138] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [41273.317160] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [41273.317214] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [41273.317275] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [41273.317357] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [41273.317435] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41273.317505] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41273.317572] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41273.317593] [<0>] kthread+0xd1/0xe0 [41273.317600] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41273.317631] [<0>] 0xfffffffffffffffe [41273.448674] Lustre: Skipped 1 previous similar message [41273.454424] Pid: 8314, comm: mdt01_017 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41273.465422] Call Trace: [41273.468254] [<0>] ldlm_completion_ast+0x7e7/0xa40 [ptlrpc] [41273.474418] [<0>] ldlm_cli_enqueue_local+0x25c/0x880 [ptlrpc] [41273.480892] [<0>] mdt_object_local_lock+0x52f/0xba0 [mdt] [41273.486939] [<0>] mdt_object_lock_internal+0x70/0x390 [mdt] [41273.493196] [<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt] [41273.499354] [<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt] [41273.505122] [<0>] mdt_intent_opc+0x1e0/0xc10 [mdt] [41273.510502] [<0>] mdt_intent_policy+0x1a1/0x360 [mdt] [41273.516210] [<0>] ldlm_lock_enqueue+0x3c5/0xb50 [ptlrpc] [41273.522188] [<0>] ldlm_handle_enqueue0+0x8d6/0x1770 [ptlrpc] [41273.528580] [<0>] tgt_enqueue+0x64/0x240 [ptlrpc] [41273.533880] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41273.540043] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41273.547079] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41273.552559] [<0>] kthread+0xd1/0xe0 [41273.556493] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41273.562278] [<0>] 0xfffffffffffffffe [41372.746038] LustreError: 8314:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1651676202, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-soaked-MDT0000_UUID lock: ffff99ebf59469c0/0x322afadd0ea8d7d3 lrc: 3/1,0 mode: --/PR res: [0x200000408:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 8314 timeout: 0 lvb_type: 0 [41372.790593] LustreError: 8314:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) Skipped 1 previous similar message [41500.719617] Lustre: soaked-MDT0000: Client 2b5be8a8-9023-43d6-8098-538f23791711 (at 192.168.1.122@o2ib) reconnecting [41500.731410] Lustre: Skipped 8 previous similar messages [41597.939063] Lustre: mdt_rdpg01_006: service thread pid 3849 was inactive for 200.366 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [41597.960454] Pid: 3849, comm: mdt_rdpg01_006 3.10.0-1160.49.1.el7_lustre.x86_64 #1 SMP Sun Apr 3 16:20:30 UTC 2022 [41597.971939] Call Trace: [41597.974732] [<0>] call_rwsem_down_write_failed+0x17/0x30 [41597.980748] [<0>] llog_cat_add_rec+0x11d/0x980 [obdclass] [41597.986838] [<0>] llog_add+0x182/0x1f0 [obdclass] [41597.992225] [<0>] sub_updates_write+0x302/0xe3b [ptlrpc] [41597.998278] [<0>] top_trans_stop+0x4a2/0xfa0 [ptlrpc] [41598.003984] [<0>] lod_trans_stop+0x25c/0x340 [lod] [41598.009394] [<0>] mdd_trans_stop+0x2e/0x174 [mdd] [41598.014708] [<0>] mdd_attr_set+0x81c/0x1070 [mdd] [41598.020060] [<0>] mdt_mfd_close+0x771/0xbb0 [mdt] [41598.025330] [<0>] mdt_close_internal+0x141/0x240 [mdt] [41598.031102] [<0>] mdt_close+0x291/0x900 [mdt] [41598.036027] [<0>] tgt_request_handle+0x92f/0x19c0 [ptlrpc] [41598.042224] [<0>] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [41598.049276] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [41598.054740] [<0>] kthread+0xd1/0xe0 [41598.058655] [<0>] ret_from_fork_nospec_begin+0x21/0x21 [41598.064439] [<0>] 0xfffffffffffffffe [41667.830205] Lustre: 3770:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec3df5e300 x1731767650932736/t0(0) o101->db61db00-0999-4840-ab32-048bf6b05215@192.168.1.119@o2ib:297/0 lens 576/4112 e 24 to 0 dl 1651676802 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [41954.306953] Lustre: 3780:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/-282), not sending early reply req@ffff99e815a2d100 x1731764235741888/t0(0) o101->7ad30715-59f4-4c17-88bd-2c39bf00253d@192.168.1.126@o2ib:583/0 lens 576/4112 e 18 to 0 dl 1651677088 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [41992.420637] Lustre: 3568:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff99ec41faad00 x1731770952040128/t42961884399(0) o35->ec6e9766-b582-4c68-bc4c-f7d5702b85d9@192.168.1.117@o2ib:622/0 lens 392/456 e 17 to 0 dl 1651677127 ref 2 fl Interpret:/0/0 rc 0/0 job:'' [42101.813123] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [42101.824904] Lustre: Skipped 8 previous similar messages [42403.006938] Lustre: 3492:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651677525/real 0] req@ffff99e813d79200 x1731866524284544/t0(0) o41->soaked-MDT0001-osp-MDT0000@192.168.1.109@o2ib:24/4 lens 224/368 e 0 to 1 dl 1651677532 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [42403.038697] Lustre: soaked-MDT0001-osp-MDT0000: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib) was lost; in progress operations using this service will wait for recovery to complete [42410.166601] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.110@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42410.186431] LustreError: Skipped 269 previous similar messages [42413.946393] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [42413.958647] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.109@o2ib (23): c: 6, oc: 0, rc: 8 [42442.240940] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.122@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42442.260777] LustreError: Skipped 146 previous similar messages [42506.476958] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.135@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42506.496789] LustreError: Skipped 464 previous similar messages [42576.382340] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42576.907687] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42576.918200] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42577.429515] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42577.440036] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42577.951763] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42577.962354] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42578.474456] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42578.484981] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42578.998129] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT returned with sense 05/91/36 [42579.008658] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42579.441223] sd 0:0:1:3: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [42579.449919] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, queueing MODE_SELECT command [42579.868529] sd 0:0:1:1: rdac: array soak-netapp2624-1, ctlr 1, MODE_SELECT completed [42580.395524] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.405858] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42580.413451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42580.420176] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42580.430094] blk_update_request: critical target error, dev sdh, sector 6986547072 [42580.438488] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42580.523943] sd 0:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.534259] sd 0:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current] [42580.541941] sd 0:0:0:0: [sdc] tag#1 <>ASC=0x94 ASCQ=0x1 [42580.548653] sd 0:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 b1 23 78 00 00 00 08 00 00 [42580.558660] blk_update_request: I/O error, dev sdc, sector 11608952 [42580.565750] device-mapper: multipath: Failing path 8:32. [42580.968604] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42580.978940] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42580.986532] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42580.993272] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42581.003186] blk_update_request: critical target error, dev sdh, sector 6986547072 [42581.011581] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42581.020062] Buffer I/O error on dev dm-1, logical block 873318384, async page read [42581.535221] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42581.545550] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42581.553160] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42581.559916] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [42581.569847] blk_update_request: critical target error, dev sdh, sector 0 [42581.577359] blk_update_request: critical target error, dev dm-1, sector 0 [42582.085450] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42582.095779] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42582.103373] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42582.110085] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42582.120003] blk_update_request: critical target error, dev sdh, sector 0 [42582.127525] blk_update_request: critical target error, dev dm-1, sector 0 [42582.135130] Buffer I/O error on dev dm-1, logical block 0, async page read [42582.651985] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42582.662310] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42582.669894] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42582.676612] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42582.686527] blk_update_request: critical target error, dev sdh, sector 6986547192 [42583.202021] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42583.212342] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42583.219935] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42583.226657] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42583.236908] Buffer I/O error on dev dm-1, logical block 873318399, async page read [42583.752073] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42583.762397] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42583.769994] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42583.776702] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42583.786643] Buffer I/O error on dev dm-1, logical block 0, async page read [42584.302096] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42584.312418] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42584.320012] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42584.326729] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42584.336684] Buffer I/O error on dev dm-1, logical block 0, async page read [42584.852140] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42584.862462] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42584.870055] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42584.876766] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [42584.886713] Buffer I/O error on dev dm-1, logical block 3, async page read [42585.418846] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42585.429170] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42585.436755] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42585.443473] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42585.453342] device-mapper: multipath: Reinstating path 8:32. [42585.459715] blk_update_request: 9 callbacks suppressed [42585.465457] blk_update_request: critical target error, dev sdh, sector 6986547072 [42585.473858] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42585.985526] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42585.995835] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42586.003451] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42586.010178] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f 80 00 00 00 08 00 00 [42586.020126] blk_update_request: critical target error, dev sdh, sector 6986547072 [42586.028566] blk_update_request: critical target error, dev dm-1, sector 6986547072 [42586.037053] Buffer I/O error on dev dm-1, logical block 873318384, async page read [42586.552218] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42586.562532] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42586.570123] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42586.576835] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 [42586.586751] blk_update_request: critical target error, dev sdh, sector 0 [42586.594275] blk_update_request: critical target error, dev dm-1, sector 0 [42587.118925] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42587.129242] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42587.136827] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42587.143546] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42587.153463] blk_update_request: critical target error, dev sdh, sector 0 [42587.160993] blk_update_request: critical target error, dev dm-1, sector 0 [42587.168604] Buffer I/O error on dev dm-1, logical block 0, async page read [42587.685619] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42587.695928] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42587.703538] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42587.710259] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42587.720180] blk_update_request: critical target error, dev sdh, sector 6986547192 [42587.728565] blk_update_request: critical target error, dev dm-1, sector 6986547192 [42588.252357] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42588.262668] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42588.270253] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42588.276963] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 01 a0 6e 3f f8 00 00 00 08 00 00 [42588.286912] Buffer I/O error on dev dm-1, logical block 873318399, async page read [42588.802375] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42588.812684] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42588.820268] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42588.826980] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42588.836923] Buffer I/O error on dev dm-1, logical block 0, async page read [42589.352640] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42589.362948] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42589.370553] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42589.377279] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [42589.387242] Buffer I/O error on dev dm-1, logical block 0, async page read [42589.902552] sd 0:0:1:1: [sdh] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [42589.912871] sd 0:0:1:1: [sdh] tag#0 Sense Key : Hardware Error [current] [42589.920465] sd 0:0:1:1: [sdh] tag#0 <>ASC=0x84 ASCQ=0x0 [42589.927198] sd 0:0:1:1: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 18 00 00 00 08 00 00 [42589.937150] Buffer I/O error on dev dm-1, logical block 3, async page read [42606.640932] Lustre: MGS: haven't heard from client b760fa03-2988-4278-b5a0-bfb777a9f6dc (at 192.168.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99e82ee57400, cur 1651677736 expire 1651677586 last 1651677509 [42606.664657] Lustre: Skipped 3 previous similar messages [42634.828784] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.104@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. [42634.848625] LustreError: Skipped 931 previous similar messages [42702.883462] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [42702.895249] Lustre: Skipped 5 previous similar messages [42745.545235] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [42763.811166] Lustre: soaked-MDT0001-osp-MDT0000: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib) [43063.849963] Lustre: soaked-MDT0000: Received new MDS connection from 192.168.1.109@o2ib, keep former export from same NID [43063.862232] Lustre: Skipped 1 previous similar message [43303.992197] Lustre: soaked-MDT0000: Client ab8832f6-98e1-4d13-84e2-130e4afa61aa (at 192.168.1.138@o2ib) reconnecting [43304.003988] Lustre: Skipped 11 previous similar messages [43905.099474] Lustre: soaked-MDT0000: Client a6431ad6-ad1a-403b-b2a6-2dbd7e58ec7b (at 192.168.1.120@o2ib) reconnecting [43905.111260] Lustre: Skipped 14 previous similar messages [44506.196195] Lustre: soaked-MDT0000: Client d85fd948-cdf6-4a2e-8806-76ec090bacdc (at 192.168.1.136@o2ib) reconnecting [44506.207977] Lustre: Skipped 10 previous similar messages [45107.289525] Lustre: soaked-MDT0000: Client 0f628bfa-3bc1-48b7-a4a4-37175642f976 (at 192.168.1.135@o2ib) reconnecting [45107.301324] Lustre: Skipped 9 previous similar messages [45149.065039] Lustre: 3512:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651680271/real 0] req@ffff99eb4a76a400 x1731866525474560/t0(0) o13->soaked-OST0000-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651680278 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [45149.096720] Lustre: soaked-OST0000-osc-MDT0000: Connection to soaked-OST0000 (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [45150.089092] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1651680272/real 0] req@ffff99eb6461cc80 x1731866525474880/t0(0) o13->soaked-OST000c-osc-MDT0000@192.168.1.104@o2ib:7/4 lens 224/368 e 0 to 1 dl 1651680279 ref 2 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'' [45150.120749] Lustre: 3514:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [45150.131478] Lustre: soaked-OST000c-osc-MDT0000: Connection to soaked-OST000c (at 192.168.1.104@o2ib) was lost; in progress operations using this service will wait for recovery to complete [45150.150154] Lustre: Skipped 1 previous similar message [45161.068590] LNetError: 3477:0:(o2iblnd_cb.c:3358:kiblnd_check_txs_locked()) Timed out tx: active_txs(WSQ:010), 17 seconds [45161.080835] LNetError: 3477:0:(o2iblnd_cb.c:3428:kiblnd_check_conns()) Timed out RDMA with 192.168.1.104@o2ib (23): c: 0, oc: 0, rc: 8 [45347.733093] Lustre: MGS: haven't heard from client d8239bef-30bf-4106-a92b-3c8abed3501d (at 192.168.1.104@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff99ec40743800, cur 1651680477 expire 1651680327 last 1651680250 [45349.885118] Lustre: soaked-MDT0000: haven't heard from client soaked-MDT0000-lwp-OST0004_UUID (at 192.168.1.104@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff99e81071a800, cur 1651680479 expire 1651680329 last 1651680250 [45417.079958] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 1 seconds [45417.091325] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 63 previous similar messages [45433.080658] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 6 seconds [45433.092023] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 9 previous similar messages [45465.082086] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 16 seconds [45465.093557] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 12 previous similar messages [45529.084933] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Timed out tx for 192.168.1.104@o2ib: 3 seconds [45529.096300] LNet: 3477:0:(o2iblnd_cb.c:3401:kiblnd_check_conns()) Skipped 22 previous similar messages [45708.386663] Lustre: soaked-MDT0000: Client 0bf2e512-d26c-4658-8a4b-f691f63c7ee2 (at 192.168.1.127@o2ib) reconnecting [45708.398454] Lustre: Skipped 8 previous similar messages