[ 0.000000] microcode: microcode updated early to revision 0xb00003e, date = 2021-02-06 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1160.53.1.1chaos.ch6.x86_64 (mockbuild@builder1-x86.buildfarm.internal) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 0.000000] Command line: initrd=initramfs ip=dhcp root=/dev/mapper/rootdev netroot=iscsi:192.168.64.64::811:0:iqn.2006-04.gov.llnl:lc.copper:lustre1-3.7-17-root-32261-gcdbb4dd.x86-64 netroot=iscsi:192.168.64.65::811:0:iqn.2006-04.gov.llnl:lc.copper:lustre1-3.7-17-root-32261-gcdbb4dd.x86-64 intel_pstate=disable processor.ignore_ppc=1 console=ttyS0,115200n8 crashkernel=512M audit=1 intel_idle.max_cstate=0 rd.plymouth=0 plymouth.enable=0 BOOT_IMAGE=vmlinuz BOOTIF=01-a4-bf-01-55-e3-0b [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009cfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000079fc6fff] usable [ 0.000000] BIOS-e820: [mem 0x0000000079fc7000-0x000000007ac50fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007ac51000-0x000000007b680fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000007b681000-0x000000007b7cbfff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007b7cc000-0x000000007b7fffff] usable [ 0.000000] BIOS-e820: [mem 0x000000007b800000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff400000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000807fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x8080000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000080000000 mask 3FFF80000000 uncachable [ 0.000000] 1 base 380000000000 mask 3F8000000000 uncachable [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0x7b800 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd850-0x000fd85f] mapped at [ffffffffff200850] [ 0.000000] Base memory trampoline at [ffff881380096000] 96000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x1206873000, 0x1206873fff] PGTABLE [ 0.000000] BRK [0x1206874000, 0x1206874fff] PGTABLE [ 0.000000] BRK [0x1206875000, 0x1206875fff] PGTABLE [ 0.000000] BRK [0x1206876000, 0x1206876fff] PGTABLE [ 0.000000] BRK [0x1206877000, 0x1206877fff] PGTABLE [ 0.000000] BRK [0x1206878000, 0x1206878fff] PGTABLE [ 0.000000] BRK [0x1206879000, 0x1206879fff] PGTABLE [ 0.000000] BRK [0x120687a000, 0x120687afff] PGTABLE [ 0.000000] RAMDISK: [mem 0x71ced000-0x79fa5fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f0460 00024 (v02 INTEL ) [ 0.000000] ACPI: XSDT 000000007b7ca0e8 000C4 (v01 INTEL S2600WT 00000000 INTL 01000013) [ 0.000000] ACPI: FACP 000000007b7c9000 000F4 (v04 INTEL S2600WT 00000000 INTL 20091013) [ 0.000000] ACPI: DSDT 000000007b78e000 32DA6 (v02 INTEL S2600WT 00000003 INTL 20091013) [ 0.000000] ACPI: FACS 000000007b652000 00040 [ 0.000000] ACPI: HPET 000000007b7c8000 00038 (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: APIC 000000007b7c7000 00AFC (v03 INTEL S2600WT 00000000 INTL 20091013) [ 0.000000] ACPI: MCFG 000000007b7c6000 0003C (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: MSCT 000000007b7c5000 00090 (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: SLIT 000000007b7c4000 0006C (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: SRAT 000000007b7c3000 00730 (v03 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: SPMI 000000007b7c2000 00041 (v05 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: WDDT 000000007b7c1000 00040 (v01 INTEL S2600WT 00000000 INTL 20091013) [ 0.000000] ACPI: NITR 000000007b6a3000 00071 (v02 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] ACPI: PRAD 000000007b69e000 00102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328) [ 0.000000] ACPI: UEFI 000000007b675000 00042 (v01 INTEL S2600WT 00000002 INTL 01000013) [ 0.000000] ACPI: SSDT 000000007b6a4000 E96B0 (v02 INTEL S2600WT 00004000 INTL 20130328) [ 0.000000] ACPI: SSDT 000000007b6a0000 02679 (v02 INTEL S2600WT 00000002 INTL 20130328) [ 0.000000] ACPI: SSDT 000000007b69f000 00064 (v02 INTEL S2600WT 00000002 INTL 20130328) [ 0.000000] ACPI: HEST 000000007b69d000 000A8 (v01 INTEL S2600WT 00000001 INTL 00000001) [ 0.000000] ACPI: BERT 000000007b69c000 00030 (v01 INTEL S2600WT 00000001 INTL 00000001) [ 0.000000] ACPI: ERST 000000007b69b000 00230 (v01 INTEL S2600WT 00000001 INTL 00000001) [ 0.000000] ACPI: EINJ 000000007b69a000 00130 (v01 INTEL S2600WT 00000001 INTL 00000001) [ 0.000000] ACPI: SPCR 000000007b699000 00050 (v01 INTEL S2600WT 00000000 INTL 00000000) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x10 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x14 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x16 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x18 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x24 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x26 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x30 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x34 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x36 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x38 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x11 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x15 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x17 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x19 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x25 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x27 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x31 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x35 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x37 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x39 -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x407fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x4080000000-0x807fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x407fffffff] -> [mem 0x00000000-0x407fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x407ffd9000-0x407fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x807ffd8000-0x807fffefff] [ 0.000000] Reserving 512MB of memory at 384MB for crashkernel (System RAM: 524191MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x807fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x79fc6fff] [ 0.000000] node 0: [mem 0x7b7cc000-0x7b7fffff] [ 0.000000] node 0: [mem 0x100000000-0x407fffffff] [ 0.000000] node 1: [mem 0x4080000000-0x807fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x407fffffff] [ 0.000000] On node 0 totalpages: 67084183 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 22 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 7744 pages used for memmap [ 0.000000] DMA32 zone: 495611 pages, LIFO batch:31 [ 0.000000] Normal zone: 1040384 pages used for memmap [ 0.000000] Normal zone: 66584576 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x4080000000-0x807fffffff] [ 0.000000] On node 1 totalpages: 67108864 [ 0.000000] Normal zone: 1048576 pages used for memmap [ 0.000000] Normal zone: 67108864 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x18] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x26] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x30] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x34] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x36] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x38] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x19] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x27] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x31] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x35] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x37] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x39] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x90] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x91] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x92] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x93] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x94] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x95] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x96] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x97] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x98] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x99] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaa] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xab] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xac] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xad] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xae] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaf] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xba] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbb] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbc] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbd] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbe] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbf] high level lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 32 CPUs, 0 hotplug CPUs [ 0.000000] smpboot: Ignoring 160 unusable CPUs in ACPI table [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x79fc7000-0x7ac50fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7ac51000-0x7b680fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7b681000-0x7b7cbfff] [ 0.000000] PM: Registered nosave memory: [mem 0x7b800000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xff3fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff400000-0xffffffff] [ 0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:2 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 16 17 18 19 20 21 22 23 [ 0.000000] pcpu-alloc: [1] 08 09 10 11 12 13 14 15 [1] 24 25 26 27 28 29 30 31 [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 132096257 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: initrd=initramfs ip=dhcp root=/dev/mapper/rootdev netroot=iscsi:192.168.64.64::811:0:iqn.2006-04.gov.llnl:lc.copper:lustre1-3.7-17-root-32261-gcdbb4dd.x86-64 netroot=iscsi:192.168.64.65::811:0:iqn.2006-04.gov.llnl:lc.copper:lustre1-3.7-17-root-32261-gcdbb4dd.x86-64 intel_pstate=disable processor.ignore_ppc=1 console=ttyS0,115200n8 crashkernel=512M audit=1 intel_idle.max_cstate=0 rd.plymouth=0 plymouth.enable=0 BOOT_IMAGE=vmlinuz BOOTIF=01-a4-bf-01-55-e3-0b [ 0.000000] audit: enabled (after initialization) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 5468928k/538968064k available (7984k kernel code, 2195876k absent, 9137660k reserved, 5759k data, 1976k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=32, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=32. [ 0.000000] NR_IRQS:327936 nr_irqs:1496 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 4294967296 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3192.752 MHz processor [ 0.000033] Calibrating delay loop (skipped), value calculated using timer frequency.. 6385.50 BogoMIPS (lpj=3192752) [ 0.012371] pid_max: default: 32768 minimum: 301 [ 0.018087] Security Framework initialized [ 0.023152] SELinux: Initializing. [ 0.027552] SELinux: Starting in permissive mode [ 0.027552] Yama: becoming mindful. [ 0.065043] Dentry cache hash table entries: 67108864 (order: 17, 536870912 bytes) [ 0.173127] Inode-cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.224880] Mount-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.233924] Mountpoint-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.245887] Initializing cgroup subsys memory [ 0.251248] Initializing cgroup subsys devices [ 0.256695] Initializing cgroup subsys freezer [ 0.262143] Initializing cgroup subsys net_cls [ 0.267590] Initializing cgroup subsys blkio [ 0.272841] Initializing cgroup subsys perf_event [ 0.278580] Initializing cgroup subsys hugetlb [ 0.284026] Initializing cgroup subsys pids [ 0.289182] Initializing cgroup subsys net_prio [ 0.295544] CPU0: Thermal monitoring enabled (TM1) [ 0.301417] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.307832] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 [ 0.314345] tlb_flushall_shift: 6 [ 0.318554] FEATURE SPEC_CTRL Present [ 0.323128] FEATURE IBPB_SUPPORT Present [ 0.327992] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.339257] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.346920] Spectre V2 : Mitigation: Full retpoline [ 0.352859] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.364104] TAA: Mitigation: Clear CPU buffers [ 0.369549] MDS: Mitigation: Clear CPU buffers [ 0.375388] Freeing SMP alternatives: 28k freed [ 0.382342] ACPI: Core revision 20130517 [ 0.424325] ACPI: All ACPI Tables successfully acquired [ 0.438102] ftrace: allocating 29710 entries in 117 pages [ 0.446467] random: fast init done [ 0.469675] IRQ remapping doesn't support X2APIC mode, disable x2apic. [ 0.477520] Switched APIC routing to physical flat. [ 0.484029] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.501217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (fam: 06, model: 4f, stepping: 01) [ 0.512221] TSC deadline timer enabled [ 0.512272] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 0.524270] ... version: 3 [ 0.529226] ... bit width: 48 [ 0.534280] ... generic registers: 4 [ 0.539235] ... value mask: 0000ffffffffffff [ 0.545646] ... max period: 00007fffffffffff [ 0.552057] ... fixed-purpose events: 3 [ 0.557012] ... event mask: 000000070000000f [ 0.575139] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.564771] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.622861] smpboot: CPU 8 Converting physical 0 to logical die 1 [ 0.609648] smpboot: Booting Node 1, Processors #8 #9 #10 #11 #12 #13 #14 #15 OK [ 0.742500] smpboot: Booting Node 0, Processors #16 [ 0.750635] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 0.766650] TAA CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html for more details. [ 0.783715] #17 #18 #19 #20 #21 #22 #23 OK [ 0.801270] smpboot: Booting Node 1, Processors #24 #25 #26 #27 #28 #29 #30 #31 OK [ 0.829588] Brought up 32 CPUs [ 0.833480] smpboot: Max logical packages: 2 [ 0.838728] smpboot: Total of 32 processors activated (204453.82 BogoMIPS) [ 1.815579] node 0 initialised, 64875146 pages in 904ms [ 1.816240] node 1 initialised, 65404110 pages in 904ms [ 1.828618] devtmpfs: initialized [ 1.832867] x86/mm: Memory block size: 2048MB [ 1.841455] EVM: security.selinux [ 1.845636] EVM: security.ima [ 1.849429] EVM: security.capability [ 1.853988] PM: Registering ACPI NVS region [mem 0x7ac51000-0x7b680fff] (10682368 bytes) [ 1.864714] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.872872] pinctrl core: initialized pinctrl subsystem [ 1.879241] RTC time: 18:33:36, date: 02/08/22 [ 1.884776] NET: Registered protocol family 16 [ 1.890489] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.899421] ACPI: bus type PCI registered [ 1.904377] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.912121] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.922993] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.931059] PCI: Using configuration type 1 for base access [ 1.944437] ACPI: Added _OSI(Module Device) [ 1.949592] ACPI: Added _OSI(Processor Device) [ 1.955025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.960758] ACPI: Added _OSI(Processor Aggregator Device) [ 1.967265] ACPI: Added _OSI(Linux-Dell-Video) [ 1.982079] ACPI: EC: Look up EC in DSDT [ 1.991677] ACPI: Executed 1 blocks of module-level executable AML code [ 2.109158] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 2.198400] ACPI: Dynamic OEM Table Load: [ 2.203677] ACPI: PRAD (null) 00102 (v02 INTEL SpsPrAgg 00000002 INTL 20130328) [ 2.232455] ACPI: Interpreter enabled [ 2.237033] ACPI: (supports S0 S5) [ 2.241310] ACPI: Using IOAPIC for interrupt routing [ 2.247368] HEST: Table parsing has been initialized. [ 2.253489] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 2.264197] ACPI: GPE 0x24 active on init [ 2.269176] ACPI: Enabled 6 GPEs in block 00 to 3F [ 2.301842] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff]) [ 2.308935] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.319962] acpi PNP0A03:02: _OSC: platform does not support [SHPCHotplug AER] [ 2.329078] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.338109] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.347755] PCI host bridge to bus 0000:ff [ 2.352809] pci_bus 0000:ff: root bus resource [bus ff] [ 2.359133] pci 0000:ff:08.0: [8086:6f80] type 00 class 0x088000 [ 2.359185] pci 0000:ff:08.2: [8086:6f32] type 00 class 0x110100 [ 2.359230] pci 0000:ff:08.3: [8086:6f83] type 00 class 0x088000 [ 2.359287] pci 0000:ff:09.0: [8086:6f90] type 00 class 0x088000 [ 2.359330] pci 0000:ff:09.2: [8086:6f33] type 00 class 0x110100 [ 2.359372] pci 0000:ff:09.3: [8086:6f93] type 00 class 0x088000 [ 2.359428] pci 0000:ff:0b.0: [8086:6f81] type 00 class 0x088000 [ 2.359468] pci 0000:ff:0b.1: [8086:6f36] type 00 class 0x110100 [ 2.359508] pci 0000:ff:0b.2: [8086:6f37] type 00 class 0x110100 [ 2.359547] pci 0000:ff:0b.3: [8086:6f76] type 00 class 0x088000 [ 2.359589] pci 0000:ff:0c.0: [8086:6fe0] type 00 class 0x088000 [ 2.359631] pci 0000:ff:0c.1: [8086:6fe1] type 00 class 0x088000 [ 2.359670] pci 0000:ff:0c.2: [8086:6fe2] type 00 class 0x088000 [ 2.359709] pci 0000:ff:0c.3: [8086:6fe3] type 00 class 0x088000 [ 2.359748] pci 0000:ff:0c.4: [8086:6fe4] type 00 class 0x088000 [ 2.359788] pci 0000:ff:0c.5: [8086:6fe5] type 00 class 0x088000 [ 2.359827] pci 0000:ff:0c.6: [8086:6fe6] type 00 class 0x088000 [ 2.359867] pci 0000:ff:0c.7: [8086:6fe7] type 00 class 0x088000 [ 2.359907] pci 0000:ff:0d.0: [8086:6fe8] type 00 class 0x088000 [ 2.359947] pci 0000:ff:0d.1: [8086:6fe9] type 00 class 0x088000 [ 2.359990] pci 0000:ff:0f.0: [8086:6ff8] type 00 class 0x088000 [ 2.360030] pci 0000:ff:0f.1: [8086:6ff9] type 00 class 0x088000 [ 2.360071] pci 0000:ff:0f.4: [8086:6ffc] type 00 class 0x088000 [ 2.360111] pci 0000:ff:0f.5: [8086:6ffd] type 00 class 0x088000 [ 2.360152] pci 0000:ff:0f.6: [8086:6ffe] type 00 class 0x088000 [ 2.360193] pci 0000:ff:10.0: [8086:6f1d] type 00 class 0x088000 [ 2.360234] pci 0000:ff:10.1: [8086:6f34] type 00 class 0x110100 [ 2.360276] pci 0000:ff:10.5: [8086:6f1e] type 00 class 0x088000 [ 2.360315] pci 0000:ff:10.6: [8086:6f7d] type 00 class 0x110100 [ 2.360353] pci 0000:ff:10.7: [8086:6f1f] type 00 class 0x088000 [ 2.360393] pci 0000:ff:12.0: [8086:6fa0] type 00 class 0x088000 [ 2.360420] pci 0000:ff:12.1: [8086:6f30] type 00 class 0x110100 [ 2.360470] pci 0000:ff:13.0: [8086:6fa8] type 00 class 0x088000 [ 2.360545] pci 0000:ff:13.1: [8086:6f71] type 00 class 0x088000 [ 2.360619] pci 0000:ff:13.2: [8086:6faa] type 00 class 0x088000 [ 2.360675] pci 0000:ff:13.3: [8086:6fab] type 00 class 0x088000 [ 2.360732] pci 0000:ff:13.4: [8086:6fac] type 00 class 0x088000 [ 2.360787] pci 0000:ff:13.5: [8086:6fad] type 00 class 0x088000 [ 2.360840] pci 0000:ff:13.6: [8086:6fae] type 00 class 0x088000 [ 2.360883] pci 0000:ff:13.7: [8086:6faf] type 00 class 0x088000 [ 2.360928] pci 0000:ff:14.0: [8086:6fb0] type 00 class 0x088000 [ 2.360984] pci 0000:ff:14.1: [8086:6fb1] type 00 class 0x088000 [ 2.361040] pci 0000:ff:14.2: [8086:6fb2] type 00 class 0x088000 [ 2.361099] pci 0000:ff:14.3: [8086:6fb3] type 00 class 0x088000 [ 2.361155] pci 0000:ff:14.4: [8086:6fbc] type 00 class 0x088000 [ 2.361198] pci 0000:ff:14.5: [8086:6fbd] type 00 class 0x088000 [ 2.361242] pci 0000:ff:14.6: [8086:6fbe] type 00 class 0x088000 [ 2.361284] pci 0000:ff:14.7: [8086:6fbf] type 00 class 0x088000 [ 2.361329] pci 0000:ff:15.0: [8086:6fb4] type 00 class 0x088000 [ 2.361386] pci 0000:ff:15.1: [8086:6fb5] type 00 class 0x088000 [ 2.361442] pci 0000:ff:15.2: [8086:6fb6] type 00 class 0x088000 [ 2.361501] pci 0000:ff:15.3: [8086:6fb7] type 00 class 0x088000 [ 2.361560] pci 0000:ff:16.0: [8086:6f68] type 00 class 0x088000 [ 2.361610] pci 0000:ff:16.6: [8086:6f6e] type 00 class 0x088000 [ 2.361653] pci 0000:ff:16.7: [8086:6f6f] type 00 class 0x088000 [ 2.361697] pci 0000:ff:17.0: [8086:6fd0] type 00 class 0x088000 [ 2.361746] pci 0000:ff:17.4: [8086:6fb8] type 00 class 0x088000 [ 2.361789] pci 0000:ff:17.5: [8086:6fb9] type 00 class 0x088000 [ 2.361832] pci 0000:ff:17.6: [8086:6fba] type 00 class 0x088000 [ 2.361880] pci 0000:ff:17.7: [8086:6fbb] type 00 class 0x088000 [ 2.361928] pci 0000:ff:1e.0: [8086:6f98] type 00 class 0x088000 [ 2.361969] pci 0000:ff:1e.1: [8086:6f99] type 00 class 0x088000 [ 2.362010] pci 0000:ff:1e.2: [8086:6f9a] type 00 class 0x088000 [ 2.362053] pci 0000:ff:1e.3: [8086:6fc0] type 00 class 0x088000 [ 2.362082] pci 0000:ff:1e.4: [8086:6f9c] type 00 class 0x088000 [ 2.362126] pci 0000:ff:1f.0: [8086:6f88] type 00 class 0x088000 [ 2.362169] pci 0000:ff:1f.2: [8086:6f8a] type 00 class 0x088000 [ 2.362252] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f]) [ 2.369344] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.379298] acpi PNP0A03:03: _OSC: platform does not support [SHPCHotplug AER] [ 2.388420] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.397449] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.407090] PCI host bridge to bus 0000:7f [ 2.412145] pci_bus 0000:7f: root bus resource [bus 7f] [ 2.418465] pci 0000:7f:08.0: [8086:6f80] type 00 class 0x088000 [ 2.418505] pci 0000:7f:08.2: [8086:6f32] type 00 class 0x110100 [ 2.418547] pci 0000:7f:08.3: [8086:6f83] type 00 class 0x088000 [ 2.418595] pci 0000:7f:09.0: [8086:6f90] type 00 class 0x088000 [ 2.418632] pci 0000:7f:09.2: [8086:6f33] type 00 class 0x110100 [ 2.418669] pci 0000:7f:09.3: [8086:6f93] type 00 class 0x088000 [ 2.418716] pci 0000:7f:0b.0: [8086:6f81] type 00 class 0x088000 [ 2.418751] pci 0000:7f:0b.1: [8086:6f36] type 00 class 0x110100 [ 2.418787] pci 0000:7f:0b.2: [8086:6f37] type 00 class 0x110100 [ 2.418821] pci 0000:7f:0b.3: [8086:6f76] type 00 class 0x088000 [ 2.418857] pci 0000:7f:0c.0: [8086:6fe0] type 00 class 0x088000 [ 2.418891] pci 0000:7f:0c.1: [8086:6fe1] type 00 class 0x088000 [ 2.418925] pci 0000:7f:0c.2: [8086:6fe2] type 00 class 0x088000 [ 2.418959] pci 0000:7f:0c.3: [8086:6fe3] type 00 class 0x088000 [ 2.418993] pci 0000:7f:0c.4: [8086:6fe4] type 00 class 0x088000 [ 2.419027] pci 0000:7f:0c.5: [8086:6fe5] type 00 class 0x088000 [ 2.419063] pci 0000:7f:0c.6: [8086:6fe6] type 00 class 0x088000 [ 2.419097] pci 0000:7f:0c.7: [8086:6fe7] type 00 class 0x088000 [ 2.419131] pci 0000:7f:0d.0: [8086:6fe8] type 00 class 0x088000 [ 2.419166] pci 0000:7f:0d.1: [8086:6fe9] type 00 class 0x088000 [ 2.419203] pci 0000:7f:0f.0: [8086:6ff8] type 00 class 0x088000 [ 2.419237] pci 0000:7f:0f.1: [8086:6ff9] type 00 class 0x088000 [ 2.419272] pci 0000:7f:0f.4: [8086:6ffc] type 00 class 0x088000 [ 2.419307] pci 0000:7f:0f.5: [8086:6ffd] type 00 class 0x088000 [ 2.419342] pci 0000:7f:0f.6: [8086:6ffe] type 00 class 0x088000 [ 2.419378] pci 0000:7f:10.0: [8086:6f1d] type 00 class 0x088000 [ 2.419412] pci 0000:7f:10.1: [8086:6f34] type 00 class 0x110100 [ 2.419448] pci 0000:7f:10.5: [8086:6f1e] type 00 class 0x088000 [ 2.419484] pci 0000:7f:10.6: [8086:6f7d] type 00 class 0x110100 [ 2.419518] pci 0000:7f:10.7: [8086:6f1f] type 00 class 0x088000 [ 2.419554] pci 0000:7f:12.0: [8086:6fa0] type 00 class 0x088000 [ 2.419579] pci 0000:7f:12.1: [8086:6f30] type 00 class 0x110100 [ 2.419622] pci 0000:7f:13.0: [8086:6fa8] type 00 class 0x088000 [ 2.419688] pci 0000:7f:13.1: [8086:6f71] type 00 class 0x088000 [ 2.419754] pci 0000:7f:13.2: [8086:6faa] type 00 class 0x088000 [ 2.419800] pci 0000:7f:13.3: [8086:6fab] type 00 class 0x088000 [ 2.419848] pci 0000:7f:13.4: [8086:6fac] type 00 class 0x088000 [ 2.419894] pci 0000:7f:13.5: [8086:6fad] type 00 class 0x088000 [ 2.419941] pci 0000:7f:13.6: [8086:6fae] type 00 class 0x088000 [ 2.419978] pci 0000:7f:13.7: [8086:6faf] type 00 class 0x088000 [ 2.420017] pci 0000:7f:14.0: [8086:6fb0] type 00 class 0x088000 [ 2.420064] pci 0000:7f:14.1: [8086:6fb1] type 00 class 0x088000 [ 2.420110] pci 0000:7f:14.2: [8086:6fb2] type 00 class 0x088000 [ 2.420157] pci 0000:7f:14.3: [8086:6fb3] type 00 class 0x088000 [ 2.420202] pci 0000:7f:14.4: [8086:6fbc] type 00 class 0x088000 [ 2.420239] pci 0000:7f:14.5: [8086:6fbd] type 00 class 0x088000 [ 2.420276] pci 0000:7f:14.6: [8086:6fbe] type 00 class 0x088000 [ 2.420314] pci 0000:7f:14.7: [8086:6fbf] type 00 class 0x088000 [ 2.420353] pci 0000:7f:15.0: [8086:6fb4] type 00 class 0x088000 [ 2.420400] pci 0000:7f:15.1: [8086:6fb5] type 00 class 0x088000 [ 2.420447] pci 0000:7f:15.2: [8086:6fb6] type 00 class 0x088000 [ 2.420495] pci 0000:7f:15.3: [8086:6fb7] type 00 class 0x088000 [ 2.420544] pci 0000:7f:16.0: [8086:6f68] type 00 class 0x088000 [ 2.420588] pci 0000:7f:16.6: [8086:6f6e] type 00 class 0x088000 [ 2.420625] pci 0000:7f:16.7: [8086:6f6f] type 00 class 0x088000 [ 2.420663] pci 0000:7f:17.0: [8086:6fd0] type 00 class 0x088000 [ 2.420706] pci 0000:7f:17.4: [8086:6fb8] type 00 class 0x088000 [ 2.420743] pci 0000:7f:17.5: [8086:6fb9] type 00 class 0x088000 [ 2.420781] pci 0000:7f:17.6: [8086:6fba] type 00 class 0x088000 [ 2.420821] pci 0000:7f:17.7: [8086:6fbb] type 00 class 0x088000 [ 2.420863] pci 0000:7f:1e.0: [8086:6f98] type 00 class 0x088000 [ 2.420899] pci 0000:7f:1e.1: [8086:6f99] type 00 class 0x088000 [ 2.420935] pci 0000:7f:1e.2: [8086:6f9a] type 00 class 0x088000 [ 2.420971] pci 0000:7f:1e.3: [8086:6fc0] type 00 class 0x088000 [ 2.420999] pci 0000:7f:1e.4: [8086:6f9c] type 00 class 0x088000 [ 2.421038] pci 0000:7f:1f.0: [8086:6f88] type 00 class 0x088000 [ 2.421075] pci 0000:7f:1f.2: [8086:6f8a] type 00 class 0x088000 [ 2.428244] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 2.435627] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.445576] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug AER] [ 2.454694] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.463724] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.473529] PCI host bridge to bus 0000:00 [ 2.478585] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 2.486646] pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window] [ 2.494706] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.503541] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window] [ 2.512377] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x383fffffffff window] [ 2.521987] pci_bus 0000:00: root bus resource [bus 00-7e] [ 2.528596] pci 0000:00:00.0: [8086:6f00] type 00 class 0x060000 [ 2.528702] pci 0000:00:01.0: [8086:6f02] type 01 class 0x060400 [ 2.528746] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 2.528794] pci 0000:00:01.0: System wakeup disabled by ACPI [ 2.535621] pci 0000:00:02.0: [8086:6f04] type 01 class 0x060400 [ 2.535664] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 2.535711] pci 0000:00:02.0: System wakeup disabled by ACPI [ 2.542540] pci 0000:00:02.2: [8086:6f06] type 01 class 0x060400 [ 2.542584] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold [ 2.542630] pci 0000:00:02.2: System wakeup disabled by ACPI [ 2.549461] pci 0000:00:03.0: [8086:6f08] type 01 class 0x060400 [ 2.549504] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 2.549552] pci 0000:00:03.0: System wakeup disabled by ACPI [ 2.556382] pci 0000:00:03.2: [8086:6f0a] type 01 class 0x060400 [ 2.556425] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold [ 2.556472] pci 0000:00:03.2: System wakeup disabled by ACPI [ 2.563301] pci 0000:00:04.0: [8086:6f20] type 00 class 0x088000 [ 2.563310] pci 0000:00:04.0: reg 0x10: [mem 0x383ffff2c000-0x383ffff2ffff 64bit] [ 2.563403] pci 0000:00:04.1: [8086:6f21] type 00 class 0x088000 [ 2.563412] pci 0000:00:04.1: reg 0x10: [mem 0x383ffff28000-0x383ffff2bfff 64bit] [ 2.563502] pci 0000:00:04.2: [8086:6f22] type 00 class 0x088000 [ 2.563511] pci 0000:00:04.2: reg 0x10: [mem 0x383ffff24000-0x383ffff27fff 64bit] [ 2.563602] pci 0000:00:04.3: [8086:6f23] type 00 class 0x088000 [ 2.563611] pci 0000:00:04.3: reg 0x10: [mem 0x383ffff20000-0x383ffff23fff 64bit] [ 2.563701] pci 0000:00:04.4: [8086:6f24] type 00 class 0x088000 [ 2.563709] pci 0000:00:04.4: reg 0x10: [mem 0x383ffff1c000-0x383ffff1ffff 64bit] [ 2.563804] pci 0000:00:04.5: [8086:6f25] type 00 class 0x088000 [ 2.563812] pci 0000:00:04.5: reg 0x10: [mem 0x383ffff18000-0x383ffff1bfff 64bit] [ 2.563902] pci 0000:00:04.6: [8086:6f26] type 00 class 0x088000 [ 2.563910] pci 0000:00:04.6: reg 0x10: [mem 0x383ffff14000-0x383ffff17fff 64bit] [ 2.564002] pci 0000:00:04.7: [8086:6f27] type 00 class 0x088000 [ 2.564011] pci 0000:00:04.7: reg 0x10: [mem 0x383ffff10000-0x383ffff13fff 64bit] [ 2.564102] pci 0000:00:05.0: [8086:6f28] type 00 class 0x088000 [ 2.564189] pci 0000:00:05.1: [8086:6f29] type 00 class 0x088000 [ 2.564288] pci 0000:00:05.2: [8086:6f2a] type 00 class 0x088000 [ 2.564372] pci 0000:00:05.4: [8086:6f2c] type 00 class 0x080020 [ 2.564378] pci 0000:00:05.4: reg 0x10: [mem 0x91f06000-0x91f06fff] [ 2.564478] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 [ 2.564636] pci 0000:00:11.1: [8086:8d7d] type 00 class 0x0c0500 [ 2.564651] pci 0000:00:11.1: reg 0x10: [mem 0x91f05000-0x91f05fff] [ 2.564681] pci 0000:00:11.1: reg 0x20: [io 0x5060-0x507f] [ 2.564809] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601 [ 2.564823] pci 0000:00:11.4: reg 0x10: [io 0x5098-0x509f] [ 2.564831] pci 0000:00:11.4: reg 0x14: [io 0x50cc-0x50cf] [ 2.564839] pci 0000:00:11.4: reg 0x18: [io 0x5090-0x5097] [ 2.564846] pci 0000:00:11.4: reg 0x1c: [io 0x50c8-0x50cb] [ 2.564854] pci 0000:00:11.4: reg 0x20: [io 0x5020-0x503f] [ 2.564862] pci 0000:00:11.4: reg 0x24: [mem 0x91f00000-0x91f007ff] [ 2.564899] pci 0000:00:11.4: PME# supported from D3hot [ 2.564977] pci 0000:00:14.0: [8086:8d31] type 00 class 0x0c0330 [ 2.564992] pci 0000:00:14.0: reg 0x10: [mem 0x383ffff00000-0x383ffff0ffff 64bit] [ 2.565046] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 2.565097] pci 0000:00:14.0: System wakeup disabled by ACPI [ 2.571929] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 [ 2.571944] pci 0000:00:16.0: reg 0x10: [mem 0x383ffff33000-0x383ffff3300f 64bit] [ 2.571999] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 2.572069] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 [ 2.572085] pci 0000:00:16.1: reg 0x10: [mem 0x383ffff32000-0x383ffff3200f 64bit] [ 2.572139] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 2.572220] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 [ 2.572235] pci 0000:00:1a.0: reg 0x10: [mem 0x91f02000-0x91f023ff] [ 2.572309] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 2.572364] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 2.579198] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 [ 2.579261] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 2.579314] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 2.586144] pci 0000:00:1c.3: [8086:8d16] type 01 class 0x060400 [ 2.586208] pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold [ 2.586256] pci 0000:00:1c.3: System wakeup disabled by ACPI [ 2.593094] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 [ 2.593110] pci 0000:00:1d.0: reg 0x10: [mem 0x91f01000-0x91f013ff] [ 2.593184] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 2.593239] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 2.600074] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 [ 2.600232] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601 [ 2.600244] pci 0000:00:1f.2: reg 0x10: [io 0x50c0-0x50c7] [ 2.600250] pci 0000:00:1f.2: reg 0x14: [io 0x50dc-0x50df] [ 2.600257] pci 0000:00:1f.2: reg 0x18: [io 0x50b8-0x50bf] [ 2.600263] pci 0000:00:1f.2: reg 0x1c: [io 0x50d8-0x50db] [ 2.600270] pci 0000:00:1f.2: reg 0x20: [io 0x5040-0x505f] [ 2.600276] pci 0000:00:1f.2: reg 0x24: [mem 0x91f04000-0x91f047ff] [ 2.600308] pci 0000:00:1f.2: PME# supported from D3hot [ 2.600375] pci 0000:00:1f.3: [8086:8d22] type 00 class 0x0c0500 [ 2.600388] pci 0000:00:1f.3: reg 0x10: [mem 0x383ffff31000-0x383ffff310ff 64bit] [ 2.600405] pci 0000:00:1f.3: reg 0x20: [io 0x5000-0x501f] [ 2.600584] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.606710] pci 0000:00:02.0: PCI bridge to [bus 02] [ 2.612849] pci 0000:03:00.0: [8086:1528] type 00 class 0x020000 [ 2.612865] pci 0000:03:00.0: reg 0x10: [mem 0x383fffc00000-0x383fffdfffff 64bit pref] [ 2.612871] pci 0000:03:00.0: reg 0x18: [io 0x4020-0x403f] [ 2.612885] pci 0000:03:00.0: reg 0x20: [mem 0x383fffe04000-0x383fffe07fff 64bit pref] [ 2.612932] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold [ 2.612953] pci 0000:03:00.0: reg 0x184: [mem 0x91b00000-0x91b03fff 64bit] [ 2.612955] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0x91b00000-0x91bfffff 64bit] (contains BAR0 for 64 VFs) [ 2.624420] pci 0000:03:00.0: reg 0x190: [mem 0x91c00000-0x91c03fff 64bit] [ 2.624422] pci 0000:03:00.0: VF(n) BAR3 space: [mem 0x91c00000-0x91cfffff 64bit] (contains BAR3 for 64 VFs) [ 2.635995] pci 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:02.2 (capable of 63.008 Gb/s with 8 GT/s x8 link) [ 2.651955] pci 0000:03:00.1: [8086:1528] type 00 class 0x020000 [ 2.651970] pci 0000:03:00.1: reg 0x10: [mem 0x383fffa00000-0x383fffbfffff 64bit pref] [ 2.651976] pci 0000:03:00.1: reg 0x18: [io 0x4000-0x401f] [ 2.651990] pci 0000:03:00.1: reg 0x20: [mem 0x383fffe00000-0x383fffe03fff 64bit pref] [ 2.652037] pci 0000:03:00.1: PME# supported from D0 D3hot D3cold [ 2.652055] pci 0000:03:00.1: reg 0x184: [mem 0x91d00000-0x91d03fff 64bit] [ 2.652056] pci 0000:03:00.1: VF(n) BAR0 space: [mem 0x91d00000-0x91dfffff 64bit] (contains BAR0 for 64 VFs) [ 2.663521] pci 0000:03:00.1: reg 0x190: [mem 0x91e00000-0x91e03fff 64bit] [ 2.663523] pci 0000:03:00.1: VF(n) BAR3 space: [mem 0x91e00000-0x91efffff 64bit] (contains BAR3 for 64 VFs) [ 2.675575] pci 0000:00:02.2: PCI bridge to [bus 03-04] [ 2.681893] pci 0000:00:02.2: bridge window [io 0x4000-0x4fff] [ 2.681895] pci 0000:00:02.2: bridge window [mem 0x91b00000-0x91efffff] [ 2.681898] pci 0000:00:02.2: bridge window [mem 0x383fffa00000-0x383fffefffff 64bit pref] [ 2.682006] pci 0000:05:00.0: [1000:0097] type 00 class 0x010700 [ 2.682015] pci 0000:05:00.0: reg 0x10: [io 0x3000-0x30ff] [ 2.682022] pci 0000:05:00.0: reg 0x14: [mem 0x91a40000-0x91a4ffff 64bit] [ 2.682029] pci 0000:05:00.0: reg 0x1c: [mem 0x91a00000-0x91a3ffff 64bit] [ 2.682038] pci 0000:05:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.682111] pci 0000:05:00.0: supports D1 D2 [ 2.682163] pci 0000:00:03.0: PCI bridge to [bus 05] [ 2.688188] pci 0000:00:03.0: bridge window [io 0x3000-0x3fff] [ 2.688190] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff] [ 2.688297] pci 0000:06:00.0: [1000:0097] type 00 class 0x010700 [ 2.688307] pci 0000:06:00.0: reg 0x10: [io 0x2000-0x20ff] [ 2.688314] pci 0000:06:00.0: reg 0x14: [mem 0x91940000-0x9194ffff 64bit] [ 2.688320] pci 0000:06:00.0: reg 0x1c: [mem 0x91900000-0x9193ffff 64bit] [ 2.688329] pci 0000:06:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.688401] pci 0000:06:00.0: supports D1 D2 [ 2.688453] pci 0000:00:03.2: PCI bridge to [bus 06] [ 2.694477] pci 0000:00:03.2: bridge window [io 0x2000-0x2fff] [ 2.694479] pci 0000:00:03.2: bridge window [mem 0x91900000-0x919fffff] [ 2.694515] pci 0000:00:1c.0: PCI bridge to [bus 07] [ 2.700599] pci 0000:08:00.0: [102b:0522] type 00 class 0x030000 [ 2.700624] pci 0000:08:00.0: reg 0x10: [mem 0x90000000-0x90ffffff pref] [ 2.700638] pci 0000:08:00.0: reg 0x14: [mem 0x91800000-0x91803fff] [ 2.700653] pci 0000:08:00.0: reg 0x18: [mem 0x91000000-0x917fffff] [ 2.700705] pci 0000:08:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref] [ 2.700810] pci 0000:08:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:1c.3 (capable of 7.876 Gb/s with 8 GT/s x1 link) [ 2.718741] pci 0000:00:1c.3: PCI bridge to [bus 08] [ 2.724771] pci 0000:00:1c.3: bridge window [mem 0x91000000-0x918fffff] [ 2.724776] pci 0000:00:1c.3: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.724802] pci_bus 0000:00: on NUMA node 0 [ 2.725022] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 *10 11 12 14 15) [ 2.733716] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *9 10 11 12 14 15) [ 2.742409] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 9 10 11 12 14 15) [ 2.751104] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) [ 2.759801] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) [ 2.768497] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 2.778474] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 2.788452] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 2.798598] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 2.805981] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.815926] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug AER] [ 2.825037] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.834066] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.843806] PCI host bridge to bus 0000:80 [ 2.848862] pci_bus 0000:80: root bus resource [io 0x8000-0xffff window] [ 2.856922] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window] [ 2.865756] pci_bus 0000:80: root bus resource [mem 0x384000000000-0x387fffffffff window] [ 2.875368] pci_bus 0000:80: root bus resource [bus 80-fe] [ 2.881982] pci 0000:80:02.0: [8086:6f04] type 01 class 0x060400 [ 2.882038] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold [ 2.882075] pci 0000:80:02.0: System wakeup disabled by ACPI [ 2.888908] pci 0000:80:04.0: [8086:6f20] type 00 class 0x088000 [ 2.888919] pci 0000:80:04.0: reg 0x10: [mem 0x387ffe01c000-0x387ffe01ffff 64bit] [ 2.889007] pci 0000:80:04.1: [8086:6f21] type 00 class 0x088000 [ 2.889017] pci 0000:80:04.1: reg 0x10: [mem 0x387ffe018000-0x387ffe01bfff 64bit] [ 2.889103] pci 0000:80:04.2: [8086:6f22] type 00 class 0x088000 [ 2.889113] pci 0000:80:04.2: reg 0x10: [mem 0x387ffe014000-0x387ffe017fff 64bit] [ 2.889197] pci 0000:80:04.3: [8086:6f23] type 00 class 0x088000 [ 2.889206] pci 0000:80:04.3: reg 0x10: [mem 0x387ffe010000-0x387ffe013fff 64bit] [ 2.889291] pci 0000:80:04.4: [8086:6f24] type 00 class 0x088000 [ 2.889301] pci 0000:80:04.4: reg 0x10: [mem 0x387ffe00c000-0x387ffe00ffff 64bit] [ 2.889384] pci 0000:80:04.5: [8086:6f25] type 00 class 0x088000 [ 2.889394] pci 0000:80:04.5: reg 0x10: [mem 0x387ffe008000-0x387ffe00bfff 64bit] [ 2.889478] pci 0000:80:04.6: [8086:6f26] type 00 class 0x088000 [ 2.889488] pci 0000:80:04.6: reg 0x10: [mem 0x387ffe004000-0x387ffe007fff 64bit] [ 2.889571] pci 0000:80:04.7: [8086:6f27] type 00 class 0x088000 [ 2.889581] pci 0000:80:04.7: reg 0x10: [mem 0x387ffe000000-0x387ffe003fff 64bit] [ 2.889665] pci 0000:80:05.0: [8086:6f28] type 00 class 0x088000 [ 2.889742] pci 0000:80:05.1: [8086:6f29] type 00 class 0x088000 [ 2.889836] pci 0000:80:05.2: [8086:6f2a] type 00 class 0x088000 [ 2.889914] pci 0000:80:05.4: [8086:6f2c] type 00 class 0x080020 [ 2.889922] pci 0000:80:05.4: reg 0x10: [mem 0xc8000000-0xc8000fff] [ 2.890228] pci 0000:81:00.0: [15b3:1013] type 00 class 0x020700 [ 2.890370] pci 0000:81:00.0: reg 0x10: [mem 0x387ffc000000-0x387ffdffffff 64bit pref] [ 2.890627] pci 0000:81:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.891080] pci 0000:81:00.0: PME# supported from D3cold [ 2.891374] pci 0000:81:00.1: [15b3:1013] type 00 class 0x020700 [ 2.891506] pci 0000:81:00.1: reg 0x10: [mem 0x387ffa000000-0x387ffbffffff 64bit pref] [ 2.891764] pci 0000:81:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.892193] pci 0000:81:00.1: PME# supported from D3cold [ 2.892398] pci 0000:80:02.0: PCI bridge to [bus 81] [ 2.898426] pci 0000:80:02.0: bridge window [mem 0x387ffa000000-0x387ffdffffff 64bit pref] [ 2.898433] pci_bus 0000:80: on NUMA node 1 [ 2.898794] vgaarb: device added: PCI:0000:08:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.908311] vgaarb: loaded [ 2.911813] vgaarb: bridge control possible 0000:08:00.0 [ 2.918308] SCSI subsystem initialized [ 2.923000] ACPI: bus type USB registered [ 2.927971] usbcore: registered new interface driver usbfs [ 2.934586] usbcore: registered new interface driver hub [ 2.941072] usbcore: registered new device driver usb [ 2.947350] EDAC MC: Ver: 3.0.0 [ 2.951624] PCI: Using ACPI for IRQ routing [ 2.960604] PCI: pci_cache_line_size set to 64 bytes [ 2.960792] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff] [ 2.960793] e820: reserve RAM buffer [mem 0x79fc7000-0x7bffffff] [ 2.960795] e820: reserve RAM buffer [mem 0x7b800000-0x7bffffff] [ 2.960910] NetLabel: Initializing [ 2.965188] NetLabel: domain hash size = 128 [ 2.970533] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.976569] NetLabel: unlabeled traffic allowed by default [ 2.983387] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.990855] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.999880] amd_nb: Cannot enumerate AMD northbridges [ 3.006057] Switched to clocksource hpet [ 3.015544] pnp: PnP ACPI init [ 3.019447] ACPI: bus type PNP registered [ 3.024864] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 3.024964] system 00:01: [io 0x0500-0x053f] has been reserved [ 3.032058] system 00:01: [io 0x0400-0x047f] could not be reserved [ 3.039531] system 00:01: [io 0x0540-0x057f] has been reserved [ 3.046622] system 00:01: [io 0x0600-0x061f] has been reserved [ 3.053714] system 00:01: [io 0x0ca0-0x0ca5] could not be reserved [ 3.061194] system 00:01: [io 0x0880-0x0883] has been reserved [ 3.068285] system 00:01: [io 0x0800-0x081f] has been reserved [ 3.075377] system 00:01: [mem 0xfed1c000-0xfed3ffff] could not be reserved [ 3.083633] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved [ 3.091500] system 00:01: [mem 0xff000000-0xffffffff] could not be reserved [ 3.099756] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved [ 3.107625] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved [ 3.115492] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved [ 3.123361] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved [ 3.131229] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 3.131371] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) [ 3.131504] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 3.131679] pnp: PnP ACPI: found 4 devices [ 3.136738] ACPI: bus type PNP unregistered [ 3.148646] pci 0000:05:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 3.160198] pci 0000:06:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 3.171751] pci 0000:08:00.0: can't claim BAR 6 [mem 0xffff0000-0xffffffff pref]: no compatible bridge window [ 3.183302] pci 0000:81:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 3.194853] pci 0000:81:00.1: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 3.206424] pci 0000:00:01.0: PCI bridge to [bus 01] [ 3.212453] pci 0000:00:02.0: PCI bridge to [bus 02] [ 3.218482] pci 0000:00:02.2: PCI bridge to [bus 03-04] [ 3.224799] pci 0000:00:02.2: bridge window [io 0x4000-0x4fff] [ 3.232086] pci 0000:00:02.2: bridge window [mem 0x91b00000-0x91efffff] [ 3.240142] pci 0000:00:02.2: bridge window [mem 0x383fffa00000-0x383fffefffff 64bit pref] [ 3.250049] pci 0000:05:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 3.258402] pci 0000:05:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 3.267142] pci 0000:00:03.0: PCI bridge to [bus 05] [ 3.273168] pci 0000:00:03.0: bridge window [io 0x3000-0x3fff] [ 3.280454] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff] [ 3.288518] pci 0000:06:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 3.296872] pci 0000:06:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 3.305612] pci 0000:00:03.2: PCI bridge to [bus 06] [ 3.311637] pci 0000:00:03.2: bridge window [io 0x2000-0x2fff] [ 3.318925] pci 0000:00:03.2: bridge window [mem 0x91900000-0x919fffff] [ 3.326989] pci 0000:00:1c.0: PCI bridge to [bus 07] [ 3.333022] pci 0000:08:00.0: BAR 6: assigned [mem 0x91810000-0x9181ffff pref] [ 3.341568] pci 0000:00:1c.3: PCI bridge to [bus 08] [ 3.347595] pci 0000:00:1c.3: bridge window [mem 0x91000000-0x918fffff] [ 3.355651] pci 0000:00:1c.3: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 3.364782] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 3.364784] pci_bus 0000:00: resource 5 [io 0x1000-0x7fff window] [ 3.364786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 3.364787] pci_bus 0000:00: resource 7 [mem 0x90000000-0xc7ffbfff window] [ 3.364789] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x383fffffffff window] [ 3.364791] pci_bus 0000:03: resource 0 [io 0x4000-0x4fff] [ 3.364792] pci_bus 0000:03: resource 1 [mem 0x91b00000-0x91efffff] [ 3.364794] pci_bus 0000:03: resource 2 [mem 0x383fffa00000-0x383fffefffff 64bit pref] [ 3.364795] pci_bus 0000:05: resource 0 [io 0x3000-0x3fff] [ 3.364797] pci_bus 0000:05: resource 1 [mem 0x91a00000-0x91afffff] [ 3.364798] pci_bus 0000:06: resource 0 [io 0x2000-0x2fff] [ 3.364800] pci_bus 0000:06: resource 1 [mem 0x91900000-0x919fffff] [ 3.364802] pci_bus 0000:08: resource 1 [mem 0x91000000-0x918fffff] [ 3.364803] pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 3.364808] pci 0000:80:02.0: BAR 14: assigned [mem 0xc8100000-0xc82fffff] [ 3.372966] pci 0000:81:00.0: BAR 6: assigned [mem 0xc8100000-0xc81fffff pref] [ 3.381513] pci 0000:81:00.1: BAR 6: assigned [mem 0xc8200000-0xc82fffff pref] [ 3.390059] pci 0000:80:02.0: PCI bridge to [bus 81] [ 3.396085] pci 0000:80:02.0: bridge window [mem 0xc8100000-0xc82fffff] [ 3.404146] pci 0000:80:02.0: bridge window [mem 0x387ffa000000-0x387ffdffffff 64bit pref] [ 3.414052] pci_bus 0000:80: resource 4 [io 0x8000-0xffff window] [ 3.414054] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window] [ 3.414055] pci_bus 0000:80: resource 6 [mem 0x384000000000-0x387fffffffff window] [ 3.414057] pci_bus 0000:81: resource 1 [mem 0xc8100000-0xc82fffff] [ 3.414059] pci_bus 0000:81: resource 2 [mem 0x387ffa000000-0x387ffdffffff 64bit pref] [ 3.414103] NET: Registered protocol family 2 [ 3.420000] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 3.429572] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 3.437713] TCP: Hash tables configured (established 524288 bind 65536) [ 3.445618] TCP: reno registered [ 3.449820] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.457727] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.466168] NET: Registered protocol family 1 [ 3.505093] pci 0000:08:00.0: Boot video device [ 3.505347] PCI: CLS 32 bytes, default 64 [ 3.505380] Unpacking initramfs... [ 4.448457] Freeing initrd memory: 133860k freed [ 4.486799] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.494478] software IO TLB [mem 0x6dced000-0x71ced000] (64MB) mapped at [ffff8813edced000-ffff8813f1cecfff] [ 4.506102] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer [ 4.515427] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules [ 4.522324] RAPL PMU: hw unit of domain package 2^-14 Joules [ 4.529128] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 4.539246] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 4.546660] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 4.554924] futex hash table entries: 8192 (order: 7, 524288 bytes) [ 4.562501] Initialise system trusted keyring [ 4.567868] audit: initializing netlink socket (enabled) [ 4.574296] type=2000 audit(1644345213.048:1): initialized [ 4.600909] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.608486] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.617199] zpool: loaded [ 4.620609] zbud: loaded [ 4.624203] VFS: Disk quotas dquot_6.5.2 [ 4.629109] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.637033] Key type big_key registered [ 4.641802] SELinux: Registering netfilter hooks [ 4.642917] NET: Registered protocol family 38 [ 4.648370] Key type asymmetric registered [ 4.653427] Asymmetric key parser 'x509' registered [ 4.659390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 4.668211] io scheduler noop registered [ 4.673074] io scheduler deadline registered (default) [ 4.679316] io scheduler cfq registered [ 4.684083] io scheduler mq-deadline registered [ 4.689626] io scheduler kyber registered [ 4.694952] pcieport 0000:00:01.0: irq 26 for MSI/MSI-X [ 4.695063] pcieport 0000:00:02.0: irq 27 for MSI/MSI-X [ 4.695164] pcieport 0000:00:02.2: irq 28 for MSI/MSI-X [ 4.695253] pcieport 0000:00:03.0: irq 29 for MSI/MSI-X [ 4.695341] pcieport 0000:00:03.2: irq 30 for MSI/MSI-X [ 4.695442] pcieport 0000:00:1c.0: irq 31 for MSI/MSI-X [ 4.695549] pcieport 0000:00:1c.3: irq 32 for MSI/MSI-X [ 4.695725] pcieport 0000:80:02.0: irq 33 for MSI/MSI-X [ 4.695887] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 4.704147] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded [ 4.704161] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 4.712420] pcie_pme 0000:00:02.0:pcie001: service driver pcie_pme loaded [ 4.712432] pcieport 0000:00:02.2: Signaling PME through PCIe PME interrupt [ 4.720689] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 4.728460] pci 0000:03:00.1: Signaling PME through PCIe PME interrupt [ 4.736231] pcie_pme 0000:00:02.2:pcie001: service driver pcie_pme loaded [ 4.736242] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 4.744497] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt [ 4.752268] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded [ 4.752279] pcieport 0000:00:03.2: Signaling PME through PCIe PME interrupt [ 4.760533] pci 0000:06:00.0: Signaling PME through PCIe PME interrupt [ 4.768306] pcie_pme 0000:00:03.2:pcie001: service driver pcie_pme loaded [ 4.768319] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 4.776577] pcie_pme 0000:00:1c.0:pcie001: service driver pcie_pme loaded [ 4.776592] pcieport 0000:00:1c.3: Signaling PME through PCIe PME interrupt [ 4.784850] pci 0000:08:00.0: Signaling PME through PCIe PME interrupt [ 4.792626] pcie_pme 0000:00:1c.3:pcie001: service driver pcie_pme loaded [ 4.792642] pcieport 0000:80:02.0: Signaling PME through PCIe PME interrupt [ 4.800942] pci 0000:81:00.0: Signaling PME through PCIe PME interrupt [ 4.808718] pci 0000:81:00.1: Signaling PME through PCIe PME interrupt [ 4.816496] pcie_pme 0000:80:02.0:pcie001: service driver pcie_pme loaded [ 4.816594] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.823316] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.831212] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.839251] intel_idle: disabled [ 4.839498] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.848246] ACPI: Power Button [PWRF] [ 4.852869] ACPI: Requesting acpi_cpufreq [ 4.858980] Monitor-Mwait will be used to enter C-1 state [ 4.858992] Monitor-Mwait will be used to enter C-2 state [ 4.859002] ACPI: acpi_idle registered with cpuidle [ 4.893602] ERST: Error Record Serialization Table (ERST) support is initialized. [ 4.902456] pstore: Registered erst as persistent store backend [ 4.909864] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 4.918722] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.946883] 00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.974291] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.981462] Non-volatile memory driver v1.3 [ 4.986644] Linux agpgart interface v0.103 [ 4.991993] crash memory driver: version 1.1 [ 4.997522] rdac: device handler registered [ 5.002715] hp_sw: device handler registered [ 5.007970] emc: device handler registered [ 5.013132] alua: device handler registered [ 5.018367] libphy: Fixed MDIO Bus: probed [ 5.023467] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.031245] ehci-pci: EHCI PCI platform driver [ 5.037097] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 5.043500] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 5.052259] ehci-pci 0000:00:1a.0: debug port 2 [ 5.061702] ehci-pci 0000:00:1a.0: cache line size of 32 is not supported [ 5.061720] ehci-pci 0000:00:1a.0: irq 18, io mem 0x91f02000 [ 5.073974] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 5.080927] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.090637] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.099185] usb usb1: Product: EHCI Host Controller [ 5.105117] usb usb1: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 ehci_hcd [ 5.114346] usb usb1: SerialNumber: 0000:00:1a.0 [ 5.120090] hub 1-0:1.0: USB hub found [ 5.124759] hub 1-0:1.0: 2 ports detected [ 5.129917] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 5.136288] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 5.145047] ehci-pci 0000:00:1d.0: debug port 2 [ 5.154473] ehci-pci 0000:00:1d.0: cache line size of 32 is not supported [ 5.154479] ehci-pci 0000:00:1d.0: irq 18, io mem 0x91f01000 [ 5.166979] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 5.173915] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.183626] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.192175] usb usb2: Product: EHCI Host Controller [ 5.198106] usb usb2: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 ehci_hcd [ 5.207333] usb usb2: SerialNumber: 0000:00:1d.0 [ 5.213063] hub 2-0:1.0: USB hub found [ 5.217739] hub 2-0:1.0: 2 ports detected [ 5.222809] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.230201] ohci-pci: OHCI PCI platform driver [ 5.235680] uhci_hcd: USB Universal Host Controller Interface driver [ 5.243495] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 5.249868] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 [ 5.259706] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810 [ 5.270393] xhci_hcd 0000:00:14.0: cache line size of 32 is not supported [ 5.270411] xhci_hcd 0000:00:14.0: irq 34 for MSI/MSI-X [ 5.270548] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.280259] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.288804] usb usb3: Product: xHCI Host Controller [ 5.294727] usb usb3: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 xhci-hcd [ 5.303955] usb usb3: SerialNumber: 0000:00:14.0 [ 5.309695] hub 3-0:1.0: USB hub found [ 5.314378] hub 3-0:1.0: 15 ports detected [ 5.320916] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 5.327315] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 [ 5.336066] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed [ 5.343683] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 3.10 [ 5.353393] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.361943] usb usb4: Product: xHCI Host Controller [ 5.367873] usb usb4: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 xhci-hcd [ 5.377101] usb usb4: SerialNumber: 0000:00:14.0 [ 5.382840] hub 4-0:1.0: USB hub found [ 5.387528] hub 4-0:1.0: 6 ports detected [ 5.393256] usbcore: registered new interface driver usbserial_generic [ 5.401042] usbserial: USB Serial support registered for generic [ 5.408271] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 5.493948] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 5.537958] tsc: Refined TSC clocksource calibration: 3192.607 MHz [ 5.538945] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 5.929367] i8042: Can't read CTR while initializing i8042 [ 5.935991] i8042: probe of i8042 failed with error -5 [ 5.936005] Switched to clocksource tsc [ 5.947026] mousedev: PS/2 mouse device common for all mice [ 5.953833] rtc_cmos 00:00: RTC can wake from S4 [ 5.959587] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 5.966905] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 5.976304] cpuidle: using governor menu [ 5.981531] hidraw: raw HID events driver (C) Jiri Kosina [ 5.986946] usb 3-9: new full-speed USB device number 2 using xhci_hcd [ 5.995906] usbcore: registered new interface driver usbhid [ 5.999297] usb 1-1: New USB device found, idVendor=8087, idProduct=800a, bcdDevice= 0.05 [ 5.999298] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.999307] usb 2-1: New USB device found, idVendor=8087, idProduct=8002, bcdDevice= 0.05 [ 5.999308] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.999413] hub 1-1:1.0: USB hub found [ 5.999518] hub 1-1:1.0: 6 ports detected [ 5.999538] hub 2-1:1.0: USB hub found [ 5.999644] hub 2-1:1.0: 8 ports detected [ 6.057985] usbhid: USB HID core driver [ 6.062895] drop_monitor: Initializing network drop monitor service [ 6.070432] Netfilter messages via NETLINK v0.30. [ 6.076216] TCP: cubic registered [ 6.080400] Initializing XFRM netlink socket [ 6.085778] NET: Registered protocol family 10 [ 6.091530] NET: Registered protocol family 17 [ 6.096988] mpls_gso: MPLS GSO support [ 6.103651] intel_rdt: Intel RDT L3 allocation detected [ 6.109969] intel_rdt: Intel RDT L3DATA allocation detected [ 6.116673] intel_rdt: Intel RDT L3CODE allocation detected [ 6.123381] intel_rdt: Intel RDT L3 monitoring detected [ 6.129706] mce: Using 22 MCE banks [ 6.130993] usb 3-9: New USB device found, idVendor=046b, idProduct=ff10, bcdDevice= 1.00 [ 6.130996] usb 3-9: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 6.130998] usb 3-9: Product: Virtual Keyboard and Mouse [ 6.131001] usb 3-9: Manufacturer: American Megatrends Inc. [ 6.131003] usb 3-9: SerialNumber: serial [ 6.132311] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-9/3-9:1.0/input/input1 [ 6.183167] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:14.0-9/input0 [ 6.184114] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-9/3-9:1.1/input/input2 [ 6.184246] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:14.0-9/input1 [ 6.232271] microcode: sig=0x406f1, pf=0x1, revision=0xb00003e [ 6.239424] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 6.249828] PM: Hibernation image not present or could not be loaded. [ 6.249831] Loading compiled-in X.509 certificates [ 6.256024] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' [ 6.269760] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' [ 6.282419] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: a1775f904d8c469b10aaacf6a86d40e66f3447a1' [ 6.294774] registered taskstats version 1 [ 6.299854] page_owner is disabled [ 6.306082] Key type trusted registered [ 6.312564] Key type encrypted registered [ 6.317546] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 6.325530] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 6.337682] Magic number: 10:514:594 [ 6.342482] acpi device:168: hash matches [ 6.347461] acpi device:13b: hash matches [ 6.352582] rtc_cmos 00:00: setting system clock to 2022-02-08 18:33:40 UTC (1644345220) [ 6.362941] Freeing unused kernel memory: 1976k freed [ 6.369810] Write protecting the kernel read-only data: 12288k [ 6.377321] Freeing unused kernel memory: 196k freed [ 6.384324] Freeing unused kernel memory: 524k freed [ 6.395834] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 6.416555] systemd[1]: Detected architecture x86-64. [ 6.422683] systemd[1]: Running in initial RAM disk. [ 6.440028] systemd[1]: No hostname configured. [ 6.445593] systemd[1]: Set hostname to . [ 6.451653] systemd[1]: Initializing machine ID from random generator. [ 6.495386] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ 6.507468] systemd[1]: Created slice Root Slice. [ 6.518025] systemd[1]: Reached target Timers. [ 6.528180] systemd[1]: Created slice System Slice. [ 6.539027] systemd[1]: Reached target Local File Systems. [ 6.552158] systemd[1]: Listening on Journal Socket. [ 6.565613] systemd[1]: Starting Journal Service... [ 6.576781] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 6.597820] systemd[1]: Starting dracut cmdline hook... [ 6.610551] systemd[1]: Starting Load Kernel Modules... [ 6.618988] fuse init (API version 7.23) [ 6.626940] systemd[1]: Reached target Local Encrypted Volumes. [ 6.641351] systemd[1]: Starting Setup Virtual Console... [ 6.652951] systemd[1]: Listening on udev Kernel Socket. [ 6.665934] systemd[1]: Reached target Slices. [ 6.676926] systemd[1]: Reached target Swap. [ 6.687953] systemd[1]: Listening on udev Control Socket. [ 6.701176] systemd[1]: Started Journal Service. [ 6.708384] Loading iSCSI transport class v2.0-870. [ 6.718137] type=1130 audit(1644345220.865:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.741566] iscsi: registered transport (tcp) [ 6.755925] type=1130 audit(1644345220.903:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.783991] type=1130 audit(1644345220.931:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.812963] type=1130 audit(1644345220.960:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.840949] type=1130 audit(1644345220.988:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.882013] device-mapper: uevent: version 1.0.3 [ 6.889057] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6.903971] type=1130 audit(1644345221.051:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.933918] type=1130 audit(1644345221.081:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6.957298] RPC: Registered named UNIX socket transport module. [ 6.964395] RPC: Registered udp transport module. [ 6.970132] RPC: Registered tcp transport module. [ 6.975865] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 7.068959] type=1130 audit(1644345221.216:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.105041] type=1130 audit(1644345221.252:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.271967] pps_core: LinuxPPS API ver. 1 registered [ 7.278841] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 7.290071] cryptd: max_cpu_qlen set to 1000 [ 7.297442] PTP clock support registered [ 7.302890] libata version 3.00 loaded. [ 7.304532] dca service started, version 1.12.1 [ 7.311643] ixgbe: loading out-of-tree module taints kernel. [ 7.311710] ixgbe: loading out-of-tree module taints kernel. [ 7.326584] ixgbe: module verification failed: signature and/or required key missing - tainting kernel [ 7.343200] AVX2 version of gcm_enc/dec engaged. [ 7.349543] AES CTR mode by8 optimization enabled [ 7.357731] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 7.366312] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 7.377310] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.6.1 [ 7.386447] Copyright(c) 1999 - 2019 Intel Corporation. [ 7.397882] ixgbe: 0000:03:00.0: ixgbe_check_options: FCoE Offload feature enabled [ 7.407401] ahci 0000:00:11.4: version 3.0 [ 7.407584] mlx5_core 0000:81:00.0: firmware version: 12.28.2006 [ 7.407844] ahci 0000:00:11.4: irq 37 for MSI/MSI-X [ 7.407916] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 7.407919] ahci 0000:00:11.4: flags: 64bit ncq led clo pio slum part ems apst [ 7.408895] mpt3sas version 31.100.01.00 loaded [ 7.409109] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (527771112 kB) [ 7.414294] scsi host1: ahci [ 7.414406] scsi host2: ahci [ 7.414501] scsi host3: ahci [ 7.414594] scsi host4: ahci [ 7.414641] ata1: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00100 irq 37 [ 7.414642] ata2: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00180 irq 37 [ 7.414645] ata3: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00200 irq 37 [ 7.414647] ata4: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00280 irq 37 [ 7.414792] ahci 0000:00:1f.2: irq 39 for MSI/MSI-X [ 7.414867] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 7.414870] ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part ems apst [ 7.425468] scsi host5: ahci [ 7.425565] scsi host6: ahci [ 7.425655] scsi host7: ahci [ 7.425748] scsi host8: ahci [ 7.425837] scsi host9: ahci [ 7.425934] scsi host10: ahci [ 7.425984] ata5: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04100 irq 39 [ 7.425986] ata6: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04180 irq 39 [ 7.425989] ata7: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04200 irq 39 [ 7.425991] ata8: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04280 irq 39 [ 7.425994] ata9: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04300 irq 39 [ 7.425997] ata10: SATA max UDMA/133 abar m2048@0x91f04000 port 0x91f04380 irq 39 [ 7.465880] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 7.465896] mpt3sas_cm0: MSI-X vectors supported: 96 [ 7.465897] no of cores: 32, max_msix_vectors: -1 [ 7.465898] mpt3sas_cm0: 0 32 [ 7.465984] mpt3sas 0000:05:00.0: irq 40 for MSI/MSI-X [ 7.465991] mpt3sas 0000:05:00.0: irq 41 for MSI/MSI-X [ 7.465996] mpt3sas 0000:05:00.0: irq 42 for MSI/MSI-X [ 7.466002] mpt3sas 0000:05:00.0: irq 43 for MSI/MSI-X [ 7.466008] mpt3sas 0000:05:00.0: irq 44 for MSI/MSI-X [ 7.466017] mpt3sas 0000:05:00.0: irq 45 for MSI/MSI-X [ 7.466022] mpt3sas 0000:05:00.0: irq 46 for MSI/MSI-X [ 7.466028] mpt3sas 0000:05:00.0: irq 47 for MSI/MSI-X [ 7.466037] mpt3sas 0000:05:00.0: irq 48 for MSI/MSI-X [ 7.466042] mpt3sas 0000:05:00.0: irq 49 for MSI/MSI-X [ 7.466048] mpt3sas 0000:05:00.0: irq 50 for MSI/MSI-X [ 7.466053] mpt3sas 0000:05:00.0: irq 51 for MSI/MSI-X [ 7.466059] mpt3sas 0000:05:00.0: irq 52 for MSI/MSI-X [ 7.466064] mpt3sas 0000:05:00.0: irq 53 for MSI/MSI-X [ 7.466070] mpt3sas 0000:05:00.0: irq 54 for MSI/MSI-X [ 7.466075] mpt3sas 0000:05:00.0: irq 55 for MSI/MSI-X [ 7.466082] mpt3sas 0000:05:00.0: irq 56 for MSI/MSI-X [ 7.466088] mpt3sas 0000:05:00.0: irq 57 for MSI/MSI-X [ 7.466095] mpt3sas 0000:05:00.0: irq 58 for MSI/MSI-X [ 7.466101] mpt3sas 0000:05:00.0: irq 59 for MSI/MSI-X [ 7.466106] mpt3sas 0000:05:00.0: irq 60 for MSI/MSI-X [ 7.466115] mpt3sas 0000:05:00.0: irq 61 for MSI/MSI-X [ 7.466121] mpt3sas 0000:05:00.0: irq 62 for MSI/MSI-X [ 7.466126] mpt3sas 0000:05:00.0: irq 63 for MSI/MSI-X [ 7.466137] mpt3sas 0000:05:00.0: irq 64 for MSI/MSI-X [ 7.466143] mpt3sas 0000:05:00.0: irq 65 for MSI/MSI-X [ 7.466148] mpt3sas 0000:05:00.0: irq 66 for MSI/MSI-X [ 7.466153] mpt3sas 0000:05:00.0: irq 67 for MSI/MSI-X [ 7.466159] mpt3sas 0000:05:00.0: irq 68 for MSI/MSI-X [ 7.466164] mpt3sas 0000:05:00.0: irq 69 for MSI/MSI-X [ 7.466169] mpt3sas 0000:05:00.0: irq 70 for MSI/MSI-X [ 7.466175] mpt3sas 0000:05:00.0: irq 71 for MSI/MSI-X [ 7.466407] mpt3sas_cm0: High IOPs queues : disabled [ 7.466408] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 40 [ 7.466408] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 41 [ 7.466409] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 42 [ 7.466409] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 43 [ 7.466410] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 44 [ 7.466410] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 45 [ 7.466411] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 46 [ 7.466411] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 47 [ 7.466412] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 48 [ 7.466412] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 49 [ 7.466413] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 50 [ 7.466413] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 51 [ 7.466414] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 52 [ 7.466415] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 53 [ 7.466415] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 54 [ 7.466416] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 55 [ 7.466416] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 56 [ 7.466417] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 57 [ 7.466417] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 58 [ 7.466418] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 59 [ 7.466418] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 60 [ 7.466419] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 61 [ 7.466419] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 62 [ 7.466420] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 63 [ 7.466420] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 64 [ 7.466421] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 65 [ 7.466421] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 66 [ 7.466422] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 67 [ 7.466422] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 68 [ 7.466423] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 69 [ 7.466423] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 70 [ 7.466423] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 71 [ 7.466425] mpt3sas_cm0: iomem(0x0000000091a40000), mapped(0xffffae5b71e00000), size(65536) [ 7.466425] mpt3sas_cm0: ioport(0x0000000000003000), size(256) [ 7.484861] mlx5_core 0000:81:00.0: 126.016 Gb/s available PCIe bandwidth (8 GT/s x16 link) [ 7.523405] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 7.640985] mpt3sas_cm0: Allocated physical memory: size(9474 kB) [ 7.640986] mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664) [ 7.640987] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 7.720887] ata1: SATA link down (SStatus 0 SControl 300) [ 7.720917] ata4: SATA link down (SStatus 0 SControl 300) [ 7.720945] ata3: SATA link down (SStatus 0 SControl 300) [ 7.720975] ata2: SATA link down (SStatus 0 SControl 300) [ 7.732883] ata5: SATA link down (SStatus 0 SControl 300) [ 7.733889] ata6: SATA link down (SStatus 0 SControl 300) [ 7.749884] ata7: SATA link down (SStatus 0 SControl 300) [ 7.750885] ata8: SATA link down (SStatus 0 SControl 300) [ 7.751886] ata9: SATA link down (SStatus 0 SControl 300) [ 7.757884] ata10: SATA link down (SStatus 0 SControl 300) [ 7.807870] mpt3sas_cm0: _base_display_fwpkg_version: complete [ 7.808116] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.01.00), ChipRevision(0x02), BiosVersion(08.37.00.00) [ 7.808120] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 7.808191] scsi host0: Fusion MPT SAS Host [ 7.808428] mpt3sas_cm0: sending port enable !! [ 7.890759] [TTM] Zone kernel: Available graphics memory: 263885556 kiB [ 7.890760] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 7.890760] [TTM] Initializing pool allocator [ 7.890763] [TTM] Initializing DMA pool allocator [ 7.930408] fbcon: mgadrmfb (fb0) is primary device [ 7.931264] mpt3sas_cm1: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (527771112 kB) [ 7.977933] ixgbe 0000:03:00.0: irq 73 for MSI/MSI-X [ 7.977944] ixgbe 0000:03:00.0: irq 74 for MSI/MSI-X [ 7.977953] ixgbe 0000:03:00.0: irq 75 for MSI/MSI-X [ 7.977964] ixgbe 0000:03:00.0: irq 76 for MSI/MSI-X [ 7.977977] ixgbe 0000:03:00.0: irq 77 for MSI/MSI-X [ 7.977986] ixgbe 0000:03:00.0: irq 78 for MSI/MSI-X [ 7.977996] ixgbe 0000:03:00.0: irq 79 for MSI/MSI-X [ 7.978006] ixgbe 0000:03:00.0: irq 80 for MSI/MSI-X [ 7.978019] ixgbe 0000:03:00.0: irq 81 for MSI/MSI-X [ 7.978029] ixgbe 0000:03:00.0: irq 82 for MSI/MSI-X [ 7.978039] ixgbe 0000:03:00.0: irq 83 for MSI/MSI-X [ 7.978049] ixgbe 0000:03:00.0: irq 84 for MSI/MSI-X [ 7.978060] ixgbe 0000:03:00.0: irq 85 for MSI/MSI-X [ 7.978069] ixgbe 0000:03:00.0: irq 86 for MSI/MSI-X [ 7.978079] ixgbe 0000:03:00.0: irq 87 for MSI/MSI-X [ 7.978089] ixgbe 0000:03:00.0: irq 88 for MSI/MSI-X [ 7.978099] ixgbe 0000:03:00.0: irq 89 for MSI/MSI-X [ 7.978109] ixgbe 0000:03:00.0: irq 90 for MSI/MSI-X [ 7.978118] ixgbe 0000:03:00.0: irq 91 for MSI/MSI-X [ 7.978128] ixgbe 0000:03:00.0: irq 92 for MSI/MSI-X [ 7.978139] ixgbe 0000:03:00.0: irq 93 for MSI/MSI-X [ 7.978148] ixgbe 0000:03:00.0: irq 94 for MSI/MSI-X [ 7.978157] ixgbe 0000:03:00.0: irq 95 for MSI/MSI-X [ 7.978166] ixgbe 0000:03:00.0: irq 96 for MSI/MSI-X [ 7.978179] ixgbe 0000:03:00.0: irq 97 for MSI/MSI-X [ 7.978189] ixgbe 0000:03:00.0: irq 98 for MSI/MSI-X [ 7.978209] ixgbe 0000:03:00.0: irq 99 for MSI/MSI-X [ 7.978213] ixgbe 0000:03:00.0: irq 100 for MSI/MSI-X [ 7.978218] ixgbe 0000:03:00.0: irq 101 for MSI/MSI-X [ 7.978223] ixgbe 0000:03:00.0: irq 102 for MSI/MSI-X [ 7.978228] ixgbe 0000:03:00.0: irq 103 for MSI/MSI-X [ 7.978232] ixgbe 0000:03:00.0: irq 104 for MSI/MSI-X [ 7.978236] ixgbe 0000:03:00.0: irq 105 for MSI/MSI-X [ 7.978312] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 [ 7.992424] mpt3sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k [ 7.992443] mpt3sas_cm1: MSI-X vectors supported: 96 [ 7.992444] no of cores: 32, max_msix_vectors: -1 [ 7.992446] mpt3sas_cm1: 0 32 [ 7.992550] mpt3sas 0000:06:00.0: irq 106 for MSI/MSI-X [ 7.992561] mpt3sas 0000:06:00.0: irq 107 for MSI/MSI-X [ 7.992571] mpt3sas 0000:06:00.0: irq 108 for MSI/MSI-X [ 7.992581] mpt3sas 0000:06:00.0: irq 109 for MSI/MSI-X [ 7.992589] mpt3sas 0000:06:00.0: irq 110 for MSI/MSI-X [ 7.992598] mpt3sas 0000:06:00.0: irq 111 for MSI/MSI-X [ 7.992607] mpt3sas 0000:06:00.0: irq 112 for MSI/MSI-X [ 7.992630] mpt3sas 0000:06:00.0: irq 113 for MSI/MSI-X [ 7.992633] mpt3sas 0000:06:00.0: irq 114 for MSI/MSI-X [ 7.992637] mpt3sas 0000:06:00.0: irq 115 for MSI/MSI-X [ 7.992641] mpt3sas 0000:06:00.0: irq 116 for MSI/MSI-X [ 7.992646] mpt3sas 0000:06:00.0: irq 117 for MSI/MSI-X [ 7.992651] mpt3sas 0000:06:00.0: irq 118 for MSI/MSI-X [ 7.992656] mpt3sas 0000:06:00.0: irq 119 for MSI/MSI-X [ 7.992660] mpt3sas 0000:06:00.0: irq 120 for MSI/MSI-X [ 7.992665] mpt3sas 0000:06:00.0: irq 121 for MSI/MSI-X [ 7.992675] mpt3sas 0000:06:00.0: irq 122 for MSI/MSI-X [ 7.992681] mpt3sas 0000:06:00.0: irq 123 for MSI/MSI-X [ 7.992687] mpt3sas 0000:06:00.0: irq 124 for MSI/MSI-X [ 7.992693] mpt3sas 0000:06:00.0: irq 125 for MSI/MSI-X [ 7.992699] mpt3sas 0000:06:00.0: irq 126 for MSI/MSI-X [ 7.992704] mpt3sas 0000:06:00.0: irq 127 for MSI/MSI-X [ 7.992711] mpt3sas 0000:06:00.0: irq 128 for MSI/MSI-X [ 7.992721] mpt3sas 0000:06:00.0: irq 129 for MSI/MSI-X [ 7.992727] mpt3sas 0000:06:00.0: irq 130 for MSI/MSI-X [ 7.992733] mpt3sas 0000:06:00.0: irq 131 for MSI/MSI-X [ 7.992739] mpt3sas 0000:06:00.0: irq 132 for MSI/MSI-X [ 7.992745] mpt3sas 0000:06:00.0: irq 133 for MSI/MSI-X [ 7.992750] mpt3sas 0000:06:00.0: irq 134 for MSI/MSI-X [ 7.992755] mpt3sas 0000:06:00.0: irq 135 for MSI/MSI-X [ 7.992761] mpt3sas 0000:06:00.0: irq 136 for MSI/MSI-X [ 7.992766] mpt3sas 0000:06:00.0: irq 137 for MSI/MSI-X [ 7.993010] mpt3sas_cm1: High IOPs queues : disabled [ 7.993011] mpt3sas1-msix0: PCI-MSI-X enabled: IRQ 106 [ 7.993012] mpt3sas1-msix1: PCI-MSI-X enabled: IRQ 107 [ 7.993012] mpt3sas1-msix2: PCI-MSI-X enabled: IRQ 108 [ 7.993013] mpt3sas1-msix3: PCI-MSI-X enabled: IRQ 109 [ 7.993013] mpt3sas1-msix4: PCI-MSI-X enabled: IRQ 110 [ 7.993014] mpt3sas1-msix5: PCI-MSI-X enabled: IRQ 111 [ 7.993014] mpt3sas1-msix6: PCI-MSI-X enabled: IRQ 112 [ 7.993015] mpt3sas1-msix7: PCI-MSI-X enabled: IRQ 113 [ 7.993015] mpt3sas1-msix8: PCI-MSI-X enabled: IRQ 114 [ 7.993016] mpt3sas1-msix9: PCI-MSI-X enabled: IRQ 115 [ 7.993016] mpt3sas1-msix10: PCI-MSI-X enabled: IRQ 116 [ 7.993017] mpt3sas1-msix11: PCI-MSI-X enabled: IRQ 117 [ 7.993017] mpt3sas1-msix12: PCI-MSI-X enabled: IRQ 118 [ 7.993018] mpt3sas1-msix13: PCI-MSI-X enabled: IRQ 119 [ 7.993018] mpt3sas1-msix14: PCI-MSI-X enabled: IRQ 120 [ 7.993019] mpt3sas1-msix15: PCI-MSI-X enabled: IRQ 121 [ 7.993019] mpt3sas1-msix16: PCI-MSI-X enabled: IRQ 122 [ 7.993019] mpt3sas1-msix17: PCI-MSI-X enabled: IRQ 123 [ 7.993020] mpt3sas1-msix18: PCI-MSI-X enabled: IRQ 124 [ 7.993020] mpt3sas1-msix19: PCI-MSI-X enabled: IRQ 125 [ 7.993021] mpt3sas1-msix20: PCI-MSI-X enabled: IRQ 126 [ 7.993021] mpt3sas1-msix21: PCI-MSI-X enabled: IRQ 127 [ 7.993022] mpt3sas1-msix22: PCI-MSI-X enabled: IRQ 128 [ 7.993022] mpt3sas1-msix23: PCI-MSI-X enabled: IRQ 129 [ 7.993023] mpt3sas1-msix24: PCI-MSI-X enabled: IRQ 130 [ 7.993023] mpt3sas1-msix25: PCI-MSI-X enabled: IRQ 131 [ 7.993023] mpt3sas1-msix26: PCI-MSI-X enabled: IRQ 132 [ 7.993024] mpt3sas1-msix27: PCI-MSI-X enabled: IRQ 133 [ 7.993024] mpt3sas1-msix28: PCI-MSI-X enabled: IRQ 134 [ 7.993025] mpt3sas1-msix29: PCI-MSI-X enabled: IRQ 135 [ 7.993025] mpt3sas1-msix30: PCI-MSI-X enabled: IRQ 136 [ 7.993026] mpt3sas1-msix31: PCI-MSI-X enabled: IRQ 137 [ 7.993027] mpt3sas_cm1: iomem(0x0000000091940000), mapped(0xffffae5b71d40000), size(65536) [ 7.993028] mpt3sas_cm1: ioport(0x0000000000002000), size(256) [ 8.043260] ixgbe 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:02.2 (capable of 63.008 Gb/s with 8 GT/s x8 link) [ 8.049854] mpt3sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k [ 8.050504] 8021q: 802.1Q VLAN Support v1.8 [ 8.050824] ixgbe 0000:03:00.0 eth0: MAC: 3, PHY: 3, PBA No: 000000-000 [ 8.050826] ixgbe 0000:03:00.0: a4:bf:01:55:e3:0b [ 8.050828] ixgbe 0000:03:00.0 eth0: Enabled Features: RxQ: 32 TxQ: 32 FdirHash [ 8.056919] ixgbe 0000:03:00.0 eth0: Intel(R) 10 Gigabit Network Connection [ 8.060897] ixgbe: 0000:03:00.1: ixgbe_check_options: FCoE Offload feature enabled [ 8.162483] Console: switching to colour frame buffer device 128x48 [ 8.175758] mpt3sas_cm1: Allocated physical memory: size(9474 kB) [ 8.175759] mpt3sas_cm1: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664) [ 8.175759] mpt3sas_cm1: Scatter Gather Elements per IO(128) [ 8.211817] mgag200 0000:08:00.0: fb0: mgadrmfb frame buffer device [ 8.219499] ixgbe 0000:03:00.1: irq 139 for MSI/MSI-X [ 8.219511] ixgbe 0000:03:00.1: irq 140 for MSI/MSI-X [ 8.219521] ixgbe 0000:03:00.1: irq 141 for MSI/MSI-X [ 8.219531] ixgbe 0000:03:00.1: irq 142 for MSI/MSI-X [ 8.219542] ixgbe 0000:03:00.1: irq 143 for MSI/MSI-X [ 8.219551] ixgbe 0000:03:00.1: irq 144 for MSI/MSI-X [ 8.219561] ixgbe 0000:03:00.1: irq 145 for MSI/MSI-X [ 8.219575] ixgbe 0000:03:00.1: irq 146 for MSI/MSI-X [ 8.219584] ixgbe 0000:03:00.1: irq 147 for MSI/MSI-X [ 8.219595] ixgbe 0000:03:00.1: irq 148 for MSI/MSI-X [ 8.219605] ixgbe 0000:03:00.1: irq 149 for MSI/MSI-X [ 8.219615] ixgbe 0000:03:00.1: irq 150 for MSI/MSI-X [ 8.219625] ixgbe 0000:03:00.1: irq 151 for MSI/MSI-X [ 8.219634] ixgbe 0000:03:00.1: irq 152 for MSI/MSI-X [ 8.219644] ixgbe 0000:03:00.1: irq 153 for MSI/MSI-X [ 8.219654] ixgbe 0000:03:00.1: irq 154 for MSI/MSI-X [ 8.219664] ixgbe 0000:03:00.1: irq 155 for MSI/MSI-X [ 8.219674] ixgbe 0000:03:00.1: irq 156 for MSI/MSI-X [ 8.219684] ixgbe 0000:03:00.1: irq 157 for MSI/MSI-X [ 8.219693] ixgbe 0000:03:00.1: irq 158 for MSI/MSI-X [ 8.219703] ixgbe 0000:03:00.1: irq 159 for MSI/MSI-X [ 8.219712] ixgbe 0000:03:00.1: irq 160 for MSI/MSI-X [ 8.219721] ixgbe 0000:03:00.1: irq 161 for MSI/MSI-X [ 8.219745] ixgbe 0000:03:00.1: irq 162 for MSI/MSI-X [ 8.219750] ixgbe 0000:03:00.1: irq 163 for MSI/MSI-X [ 8.219755] ixgbe 0000:03:00.1: irq 164 for MSI/MSI-X [ 8.219760] ixgbe 0000:03:00.1: irq 165 for MSI/MSI-X [ 8.219765] ixgbe 0000:03:00.1: irq 166 for MSI/MSI-X [ 8.219770] ixgbe 0000:03:00.1: irq 167 for MSI/MSI-X [ 8.219774] ixgbe 0000:03:00.1: irq 168 for MSI/MSI-X [ 8.219778] ixgbe 0000:03:00.1: irq 169 for MSI/MSI-X [ 8.219783] ixgbe 0000:03:00.1: irq 170 for MSI/MSI-X [ 8.219787] ixgbe 0000:03:00.1: irq 171 for MSI/MSI-X [ 8.219872] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 32, Tx Queue count = 32 [ 8.284932] ixgbe 0000:03:00.1: 32.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x8 link at 0000:00:02.2 (capable of 63.008 Gb/s with 8 GT/s x8 link) [ 8.292868] ixgbe 0000:03:00.1 eth1: MAC: 3, PHY: 3, PBA No: 000000-000 [ 8.292873] ixgbe 0000:03:00.1: a4:bf:01:55:e3:0c [ 8.292877] ixgbe 0000:03:00.1 eth1: Enabled Features: RxQ: 32 TxQ: 32 FdirHash [ 8.298912] ixgbe 0000:03:00.1 eth1: Intel(R) 10 Gigabit Network Connection [ 8.346874] mpt3sas_cm1: _base_display_fwpkg_version: complete [ 8.347170] mpt3sas_cm1: LSISAS3008: FWVersion(16.00.01.00), ChipRevision(0x02), BiosVersion(08.37.00.00) [ 8.347177] mpt3sas_cm1: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 8.347272] scsi host11: Fusion MPT SAS Host [ 8.347482] mpt3sas_cm1: sending port enable !! [ 8.764845] [drm] Initialized mgag200 1.0.0 20110418 for 0000:08:00.0 on minor 0 [ 8.809909] iscsi: registered transport (qla4xxx) [ 8.815709] QLogic iSCSI HBA Driver [ 8.825260] libcxgbi:libcxgbi_init_module: Chelsio iSCSI driver library libcxgbi v0.9.1-ko (Apr. 2015) [ 8.849080] Chelsio T3 iSCSI Driver cxgb3i v2.0.1-ko (Apr. 2015) [ 8.856283] iscsi: registered transport (cxgb3i) [ 8.913062] Chelsio T4-T6 iSCSI Driver cxgb4i v0.9.5-ko (Apr. 2015) [ 8.920556] iscsi: registered transport (cxgb4i) [ 8.931192] cnic: QLogic cnicDriver v2.5.22 (July 20, 2015) [ 8.949900] QLogic NetXtreme II iSCSI Driver bnx2i v2.7.10.1 (Jul 16, 2014) [ 8.958162] iscsi: registered transport (bnx2i) [ 8.971485] iscsi: registered transport (be2iscsi) [ 8.977317] In beiscsi_module_init, tt=ffffffffc053e120 [ 9.147658] pps pps0: new PPS source ptp0 [ 9.152675] ixgbe 0000:03:00.0: registered PHC device on eno1 [ 9.203873] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready [ 9.210868] 8021q: adding VLAN 0 to HW filter on device eno1 [ 9.595663] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00db98450), phys(8) [ 9.606785] mpt3sas_cm0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500093d00212103f), phys(49) [ 9.624493] scsi 0:0:0:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 9.634014] scsi 0:0:0:0: SSP: handle(0x000b), sas_addr(0x5000c500302b8005), phy(0), device_name(0x5000c500302b8004) [ 9.646242] scsi 0:0:0:0: enclosure logical id (0x500093d002121000), slot(0) [ 9.654687] scsi 0:0:0:0: enclosure level(0x0000), connector name( ) [ 9.662653] scsi 0:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.753866] mlx5_core 0000:81:00.0: irq 172 for MSI/MSI-X [ 9.753873] mlx5_core 0000:81:00.0: irq 173 for MSI/MSI-X [ 9.753880] mlx5_core 0000:81:00.0: irq 174 for MSI/MSI-X [ 9.753886] mlx5_core 0000:81:00.0: irq 175 for MSI/MSI-X [ 9.753891] mlx5_core 0000:81:00.0: irq 176 for MSI/MSI-X [ 9.753896] mlx5_core 0000:81:00.0: irq 177 for MSI/MSI-X [ 9.753902] mlx5_core 0000:81:00.0: irq 178 for MSI/MSI-X [ 9.753907] mlx5_core 0000:81:00.0: irq 179 for MSI/MSI-X [ 9.753913] mlx5_core 0000:81:00.0: irq 180 for MSI/MSI-X [ 9.753919] mlx5_core 0000:81:00.0: irq 181 for MSI/MSI-X [ 9.753925] mlx5_core 0000:81:00.0: irq 182 for MSI/MSI-X [ 9.753931] mlx5_core 0000:81:00.0: irq 183 for MSI/MSI-X [ 9.753942] mlx5_core 0000:81:00.0: irq 184 for MSI/MSI-X [ 9.753948] mlx5_core 0000:81:00.0: irq 185 for MSI/MSI-X [ 9.753953] mlx5_core 0000:81:00.0: irq 186 for MSI/MSI-X [ 9.753960] mlx5_core 0000:81:00.0: irq 187 for MSI/MSI-X [ 9.753965] mlx5_core 0000:81:00.0: irq 188 for MSI/MSI-X [ 9.753971] mlx5_core 0000:81:00.0: irq 189 for MSI/MSI-X [ 9.753977] mlx5_core 0000:81:00.0: irq 190 for MSI/MSI-X [ 9.753982] mlx5_core 0000:81:00.0: irq 191 for MSI/MSI-X [ 9.753988] mlx5_core 0000:81:00.0: irq 192 for MSI/MSI-X [ 9.753993] mlx5_core 0000:81:00.0: irq 193 for MSI/MSI-X [ 9.753998] mlx5_core 0000:81:00.0: irq 194 for MSI/MSI-X [ 9.754003] mlx5_core 0000:81:00.0: irq 195 for MSI/MSI-X [ 9.754009] mlx5_core 0000:81:00.0: irq 196 for MSI/MSI-X [ 9.754014] mlx5_core 0000:81:00.0: irq 197 for MSI/MSI-X [ 9.754020] mlx5_core 0000:81:00.0: irq 198 for MSI/MSI-X [ 9.754025] mlx5_core 0000:81:00.0: irq 199 for MSI/MSI-X [ 9.754033] mlx5_core 0000:81:00.0: irq 200 for MSI/MSI-X [ 9.754038] mlx5_core 0000:81:00.0: irq 201 for MSI/MSI-X [ 9.754043] mlx5_core 0000:81:00.0: irq 202 for MSI/MSI-X [ 9.754047] mlx5_core 0000:81:00.0: irq 203 for MSI/MSI-X [ 9.754052] mlx5_core 0000:81:00.0: irq 204 for MSI/MSI-X [ 9.754057] mlx5_core 0000:81:00.0: irq 205 for MSI/MSI-X [ 9.754062] mlx5_core 0000:81:00.0: irq 206 for MSI/MSI-X [ 9.754067] mlx5_core 0000:81:00.0: irq 207 for MSI/MSI-X [ 9.756006] mlx5_port_module_event: 12 callbacks suppressed [ 9.762710] mlx5_core 0000:81:00.0: Port module event: module 0, Cable plugged [ 9.779862] scsi 0:0:1:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 9.789383] scsi 0:0:1:0: SSP: handle(0x000c), sas_addr(0x5000c500302b8071), phy(1), device_name(0x5000c500302b8070) [ 9.801610] scsi 0:0:1:0: enclosure logical id (0x500093d002121000), slot(1) [ 9.810057] scsi 0:0:1:0: enclosure level(0x0000), connector name( ) [ 9.818026] scsi 0:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.845874] scsi 0:0:2:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 9.855401] scsi 0:0:2:0: SSP: handle(0x000d), sas_addr(0x5000c500302e8f71), phy(2), device_name(0x5000c500302e8f70) [ 9.867629] scsi 0:0:2:0: enclosure logical id (0x500093d002121000), slot(2) [ 9.876075] scsi 0:0:2:0: enclosure level(0x0000), connector name( ) [ 9.884044] scsi 0:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.911177] mlx5_core 0000:81:00.1: firmware version: 12.28.2006 [ 9.918399] mlx5_core 0000:81:00.1: 126.016 Gb/s available PCIe bandwidth (8 GT/s x16 link) [ 9.933798] scsi 0:0:3:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 9.943321] scsi 0:0:3:0: SSP: handle(0x000e), sas_addr(0x5000c500302e8f49), phy(6), device_name(0x5000c500302e8f48) [ 9.955548] scsi 0:0:3:0: enclosure logical id (0x500093d002121000), slot(6) [ 9.963994] scsi 0:0:3:0: enclosure level(0x0000), connector name( ) [ 9.967584] mpt3sas_cm1: host_add: handle(0x0001), sas_addr(0x500605b00db97920), phys(8) [ 9.971250] mpt3sas_cm1: expander_add: handle(0x0009), parent(0x0002), sas_addr(0x500093d00211407f), phys(49) [ 9.978055] scsi 11:0:0:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 9.978058] scsi 11:0:0:0: SSP: handle(0x000a), sas_addr(0x5000c500302980d2), phy(0), device_name(0x5000c500302980d0) [ 9.978059] scsi 11:0:0:0: enclosure logical id (0x500093d002114000), slot(0) [ 9.978060] scsi 11:0:0:0: enclosure level(0x0000), connector name( ) [ 9.978062] scsi 11:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.042127] scsi 0:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.692835] scsi 0:0:4:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.692852] scsi 11:0:1:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.692857] scsi 11:0:1:0: SSP: handle(0x000b), sas_addr(0x5000c5003029810e), phy(1), device_name(0x5000c5003029810c) [ 10.692858] scsi 11:0:1:0: enclosure logical id (0x500093d002114000), slot(1) [ 10.692859] scsi 11:0:1:0: enclosure level(0x0000), connector name( ) [ 10.692861] scsi 11:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.693559] mlx5_core 0000:81:00.1: irq 209 for MSI/MSI-X [ 10.693565] mlx5_core 0000:81:00.1: irq 210 for MSI/MSI-X [ 10.693572] mlx5_core 0000:81:00.1: irq 211 for MSI/MSI-X [ 10.693577] mlx5_core 0000:81:00.1: irq 212 for MSI/MSI-X [ 10.693587] mlx5_core 0000:81:00.1: irq 213 for MSI/MSI-X [ 10.693592] mlx5_core 0000:81:00.1: irq 214 for MSI/MSI-X [ 10.693597] mlx5_core 0000:81:00.1: irq 215 for MSI/MSI-X [ 10.693602] mlx5_core 0000:81:00.1: irq 216 for MSI/MSI-X [ 10.693613] mlx5_core 0000:81:00.1: irq 217 for MSI/MSI-X [ 10.693623] mlx5_core 0000:81:00.1: irq 218 for MSI/MSI-X [ 10.693633] mlx5_core 0000:81:00.1: irq 219 for MSI/MSI-X [ 10.693643] mlx5_core 0000:81:00.1: irq 220 for MSI/MSI-X [ 10.693652] mlx5_core 0000:81:00.1: irq 221 for MSI/MSI-X [ 10.693662] mlx5_core 0000:81:00.1: irq 222 for MSI/MSI-X [ 10.693671] mlx5_core 0000:81:00.1: irq 223 for MSI/MSI-X [ 10.693681] mlx5_core 0000:81:00.1: irq 224 for MSI/MSI-X [ 10.693690] mlx5_core 0000:81:00.1: irq 225 for MSI/MSI-X [ 10.693704] mlx5_core 0000:81:00.1: irq 226 for MSI/MSI-X [ 10.693714] mlx5_core 0000:81:00.1: irq 227 for MSI/MSI-X [ 10.693724] mlx5_core 0000:81:00.1: irq 228 for MSI/MSI-X [ 10.693738] mlx5_core 0000:81:00.1: irq 229 for MSI/MSI-X [ 10.693747] mlx5_core 0000:81:00.1: irq 230 for MSI/MSI-X [ 10.693757] mlx5_core 0000:81:00.1: irq 231 for MSI/MSI-X [ 10.693766] mlx5_core 0000:81:00.1: irq 232 for MSI/MSI-X [ 10.693776] mlx5_core 0000:81:00.1: irq 233 for MSI/MSI-X [ 10.693786] mlx5_core 0000:81:00.1: irq 234 for MSI/MSI-X [ 10.693797] mlx5_core 0000:81:00.1: irq 235 for MSI/MSI-X [ 10.693806] mlx5_core 0000:81:00.1: irq 236 for MSI/MSI-X [ 10.693815] mlx5_core 0000:81:00.1: irq 237 for MSI/MSI-X [ 10.693825] mlx5_core 0000:81:00.1: irq 238 for MSI/MSI-X [ 10.693834] mlx5_core 0000:81:00.1: irq 239 for MSI/MSI-X [ 10.693843] mlx5_core 0000:81:00.1: irq 240 for MSI/MSI-X [ 10.693852] mlx5_core 0000:81:00.1: irq 241 for MSI/MSI-X [ 10.693862] mlx5_core 0000:81:00.1: irq 242 for MSI/MSI-X [ 10.693872] mlx5_core 0000:81:00.1: irq 243 for MSI/MSI-X [ 10.693881] mlx5_core 0000:81:00.1: irq 244 for MSI/MSI-X [ 10.695498] mlx5_core 0000:81:00.1: Port module event: module 1, Cable unplugged [ 10.760229] scsi 0:0:4:0: SSP: handle(0x000f), sas_addr(0x5000c500302e8f39), phy(7), device_name(0x5000c500302e8f38) [ 10.772457] scsi 0:0:4:0: enclosure logical id (0x500093d002121000), slot(7) [ 10.780904] scsi 0:0:4:0: enclosure level(0x0000), connector name( ) [ 10.788869] scsi 0:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.811798] scsi 11:0:2:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.821418] scsi 11:0:2:0: SSP: handle(0x000c), sas_addr(0x5000c50030297fee), phy(2), device_name(0x5000c50030297fec) [ 10.822654] scsi 0:0:5:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.822657] scsi 0:0:5:0: SSP: handle(0x0015), sas_addr(0x5000c500302e8f5d), phy(8), device_name(0x5000c500302e8f5c) [ 10.822658] scsi 0:0:5:0: enclosure logical id (0x500093d002121000), slot(8) [ 10.822659] scsi 0:0:5:0: enclosure level(0x0000), connector name( ) [ 10.822661] scsi 0:0:5:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.882365] scsi 11:0:2:0: enclosure logical id (0x500093d002114000), slot(2) [ 10.890907] scsi 11:0:2:0: enclosure level(0x0000), connector name( ) [ 10.898971] scsi 11:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.911989] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0 [ 10.926778] scsi 0:0:6:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.936299] scsi 0:0:6:0: SSP: handle(0x0016), sas_addr(0x5000c500302e8ed5), phy(12), device_name(0x5000c500302e8ed4) [ 10.937712] scsi 11:0:3:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 10.937715] scsi 11:0:3:0: SSP: handle(0x000d), sas_addr(0x5000c50030288a26), phy(6), device_name(0x5000c50030288a24) [ 10.937717] scsi 11:0:3:0: enclosure logical id (0x500093d002114000), slot(6) [ 10.937718] scsi 11:0:3:0: enclosure level(0x0000), connector name( ) [ 10.937720] scsi 11:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.997733] scsi 0:0:6:0: enclosure logical id (0x500093d002121000), slot(12) [ 11.006277] scsi 0:0:6:0: enclosure level(0x0000), connector name( ) [ 11.014242] scsi 0:0:6:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.037788] scsi 11:0:4:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.037853] scsi 0:0:7:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.037856] scsi 0:0:7:0: SSP: handle(0x0017), sas_addr(0x5000c500302e8ec5), phy(13), device_name(0x5000c500302e8ec4) [ 11.037857] scsi 0:0:7:0: enclosure logical id (0x500093d002121000), slot(13) [ 11.037858] scsi 0:0:7:0: enclosure level(0x0000), connector name( ) [ 11.037860] scsi 0:0:7:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.078181] random: crng init done [ 11.100507] scsi 11:0:4:0: SSP: handle(0x000e), sas_addr(0x5000c500302981c6), phy(7), device_name(0x5000c500302981c4) [ 11.112823] scsi 11:0:4:0: enclosure logical id (0x500093d002114000), slot(7) [ 11.121366] scsi 11:0:4:0: enclosure level(0x0000), connector name( ) [ 11.129430] scsi 11:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.150811] scsi 0:0:8:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.160335] scsi 0:0:8:0: SSP: handle(0x0018), sas_addr(0x5000c500302e8f1d), phy(14), device_name(0x5000c500302e8f1c) [ 11.161756] scsi 11:0:5:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.161760] scsi 11:0:5:0: SSP: handle(0x000f), sas_addr(0x5000c5003028910e), phy(8), device_name(0x5000c5003028910c) [ 11.161761] scsi 11:0:5:0: enclosure logical id (0x500093d002114000), slot(8) [ 11.161763] scsi 11:0:5:0: enclosure level(0x0000), connector name( ) [ 11.161764] scsi 11:0:5:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.221770] scsi 0:0:8:0: enclosure logical id (0x500093d002121000), slot(14) [ 11.230314] scsi 0:0:8:0: enclosure level(0x0000), connector name( ) [ 11.238281] scsi 0:0:8:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.266834] scsi 0:0:9:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.266864] scsi 11:0:6:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.266873] scsi 11:0:6:0: SSP: handle(0x0010), sas_addr(0x5000c500302891d6), phy(12), device_name(0x5000c500302891d4) [ 11.266876] scsi 11:0:6:0: enclosure logical id (0x500093d002114000), slot(12) [ 11.266879] scsi 11:0:6:0: enclosure level(0x0000), connector name( ) [ 11.266884] scsi 11:0:6:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.325666] scsi 0:0:9:0: SSP: handle(0x001d), sas_addr(0x5000c500302e8ead), phy(18), device_name(0x5000c500302e8eac) [ 11.337992] scsi 0:0:9:0: enclosure logical id (0x500093d002121000), slot(18) [ 11.346534] scsi 0:0:9:0: enclosure level(0x0000), connector name( ) [ 11.354498] scsi 0:0:9:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.385739] scsi 11:0:7:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.395358] scsi 11:0:7:0: SSP: handle(0x0011), sas_addr(0x5000c5003029819e), phy(13), device_name(0x5000c5003029819c) [ 11.397734] scsi 0:0:10:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.397738] scsi 0:0:10:0: SSP: handle(0x001e), sas_addr(0x5000c50030279ddd), phy(19), device_name(0x5000c50030279ddc) [ 11.397739] scsi 0:0:10:0: enclosure logical id (0x500093d002121000), slot(19) [ 11.397740] scsi 0:0:10:0: enclosure level(0x0000), connector name( ) [ 11.397742] scsi 0:0:10:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.457083] scsi 11:0:7:0: enclosure logical id (0x500093d002114000), slot(13) [ 11.465724] scsi 11:0:7:0: enclosure level(0x0000), connector name( ) [ 11.473787] scsi 11:0:7:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.500789] scsi 11:0:8:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.510410] scsi 11:0:8:0: SSP: handle(0x0012), sas_addr(0x5000c500302891ee), phy(14), device_name(0x5000c500302891ec) [ 11.513734] scsi 0:0:11:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.513737] scsi 0:0:11:0: SSP: handle(0x001f), sas_addr(0x5000c500302b7ea1), phy(20), device_name(0x5000c500302b7ea0) [ 11.513739] scsi 0:0:11:0: enclosure logical id (0x500093d002121000), slot(20) [ 11.513740] scsi 0:0:11:0: enclosure level(0x0000), connector name( ) [ 11.513742] scsi 0:0:11:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.572133] scsi 11:0:8:0: enclosure logical id (0x500093d002114000), slot(14) [ 11.580771] scsi 11:0:8:0: enclosure level(0x0000), connector name( ) [ 11.588835] scsi 11:0:8:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.616184] scsi 0:0:12:0: Enclosure RAIDINC 24BAY EBOD 0105 PQ: 0 ANSI: 6 [ 11.616915] scsi 11:0:9:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.616918] scsi 11:0:9:0: SSP: handle(0x0013), sas_addr(0x5000c50030298016), phy(18), device_name(0x5000c50030298014) [ 11.616919] scsi 11:0:9:0: enclosure logical id (0x500093d002114000), slot(18) [ 11.616921] scsi 11:0:9:0: enclosure level(0x0000), connector name( ) [ 11.616922] scsi 11:0:9:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.675115] scsi 0:0:12:0: set ignore_delay_remove for handle(0x0023) [ 11.682785] scsi 0:0:12:0: SES: handle(0x0023), sas_addr(0x500093d00212103e), phy(48), device_name(0x0000000000000000) [ 11.695207] scsi 0:0:12:0: enclosure logical id (0x500093d002121000), slot(24) [ 11.703846] scsi 0:0:12:0: enclosure level(0x0000), connector name( ) [ 11.711908] scsi 0:0:12:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.736486] mpt3sas_cm0: expander_add: handle(0x000a), parent(0x0002), sas_addr(0x500093d00211403f), phys(49) [ 11.749815] scsi 11:0:10:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.755050] scsi 0:0:13:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.755053] scsi 0:0:13:0: SSP: handle(0x0010), sas_addr(0x5000c500302980d1), phy(0), device_name(0x5000c500302980d0) [ 11.755054] scsi 0:0:13:0: enclosure logical id (0x500093d002114000), slot(0) [ 11.755056] scsi 0:0:13:0: enclosure level(0x0000), connector name( ) [ 11.755057] scsi 0:0:13:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.808652] scsi 11:0:10:0: SSP: handle(0x0014), sas_addr(0x5000c50030289182), phy(19), device_name(0x5000c50030289180) [ 11.821171] scsi 11:0:10:0: enclosure logical id (0x500093d002114000), slot(19) [ 11.829907] scsi 11:0:10:0: enclosure level(0x0000), connector name( ) [ 11.838065] scsi 11:0:10:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.860918] scsi 11:0:11:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.861650] scsi 0:0:14:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.861654] scsi 0:0:14:0: SSP: handle(0x0011), sas_addr(0x5000c5003029810d), phy(1), device_name(0x5000c5003029810c) [ 11.861655] scsi 0:0:14:0: enclosure logical id (0x500093d002114000), slot(1) [ 11.861656] scsi 0:0:14:0: enclosure level(0x0000), connector name( ) [ 11.861658] scsi 0:0:14:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.919751] scsi 11:0:11:0: SSP: handle(0x0015), sas_addr(0x5000c50030297f7a), phy(20), device_name(0x5000c50030297f78) [ 11.932270] scsi 11:0:11:0: enclosure logical id (0x500093d002114000), slot(20) [ 11.941006] scsi 11:0:11:0: enclosure level(0x0000), connector name( ) [ 11.949166] scsi 11:0:11:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 11.971697] scsi 0:0:15:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 11.981316] scsi 0:0:15:0: SSP: handle(0x0012), sas_addr(0x5000c50030297fed), phy(2), device_name(0x5000c50030297fec) [ 11.984551] scsi 11:0:12:0: Enclosure RAIDINC 24BAY EBOD 0105 PQ: 0 ANSI: 6 [ 11.984554] scsi 11:0:12:0: set ignore_delay_remove for handle(0x0016) [ 11.984555] scsi 11:0:12:0: SES: handle(0x0016), sas_addr(0x500093d00211407e), phy(48), device_name(0x0000000000000000) [ 11.984557] scsi 11:0:12:0: enclosure logical id (0x500093d002114000), slot(25) [ 11.984558] scsi 11:0:12:0: enclosure level(0x0000), connector name( ) [ 11.984559] scsi 11:0:12:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.051197] scsi 0:0:15:0: enclosure logical id (0x500093d002114000), slot(2) [ 12.059741] scsi 0:0:15:0: enclosure level(0x0000), connector name( ) [ 12.067803] scsi 0:0:15:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.094768] scsi 0:0:16:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.104389] scsi 0:0:16:0: SSP: handle(0x0013), sas_addr(0x5000c50030288a25), phy(6), device_name(0x5000c50030288a24) [ 12.105840] mpt3sas_cm1: expander_add: handle(0x0017), parent(0x0001), sas_addr(0x500093d00212107f), phys(49) [ 12.113128] scsi 11:0:13:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.113131] scsi 11:0:13:0: SSP: handle(0x0018), sas_addr(0x5000c500302b8006), phy(0), device_name(0x5000c500302b8004) [ 12.113133] scsi 11:0:13:0: enclosure logical id (0x500093d002121000), slot(0) [ 12.113134] scsi 11:0:13:0: enclosure level(0x0000), connector name( ) [ 12.113136] scsi 11:0:13:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.177858] scsi 0:0:16:0: enclosure logical id (0x500093d002114000), slot(6) [ 12.186400] scsi 0:0:16:0: enclosure level(0x0000), connector name( ) [ 12.194462] scsi 0:0:16:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.219732] scsi 11:0:14:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.219749] scsi 0:0:17:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.219752] scsi 0:0:17:0: SSP: handle(0x0014), sas_addr(0x5000c500302981c5), phy(7), device_name(0x5000c500302981c4) [ 12.219754] scsi 0:0:17:0: enclosure logical id (0x500093d002114000), slot(7) [ 12.219755] scsi 0:0:17:0: enclosure level(0x0000), connector name( ) [ 12.219756] scsi 0:0:17:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.278566] scsi 11:0:14:0: SSP: handle(0x0019), sas_addr(0x5000c500302b8072), phy(1), device_name(0x5000c500302b8070) [ 12.290988] scsi 11:0:14:0: enclosure logical id (0x500093d002121000), slot(1) [ 12.299629] scsi 11:0:14:0: enclosure level(0x0000), connector name( ) [ 12.307787] scsi 11:0:14:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.329738] scsi 0:0:18:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.339356] scsi 0:0:18:0: SSP: handle(0x0019), sas_addr(0x5000c5003028910d), phy(8), device_name(0x5000c5003028910c) [ 12.340636] scsi 11:0:15:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.340639] scsi 11:0:15:0: SSP: handle(0x001a), sas_addr(0x5000c500302e8f72), phy(2), device_name(0x5000c500302e8f70) [ 12.340641] scsi 11:0:15:0: enclosure logical id (0x500093d002121000), slot(2) [ 12.340642] scsi 11:0:15:0: enclosure level(0x0000), connector name( ) [ 12.340644] scsi 11:0:15:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.401280] scsi 0:0:18:0: enclosure logical id (0x500093d002114000), slot(8) [ 12.409822] scsi 0:0:18:0: enclosure level(0x0000), connector name( ) [ 12.417885] scsi 0:0:18:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.433867] ixgbe 0000:03:00.0 eno1: NIC Link is Up 1 Gbps, Flow Control: None [ 12.442658] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 12.454634] scsi 11:0:16:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.464349] scsi 11:0:16:0: SSP: handle(0x001b), sas_addr(0x5000c500302e8f4a), phy(6), device_name(0x5000c500302e8f48) [ 12.467648] scsi 0:0:19:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.467651] scsi 0:0:19:0: SSP: handle(0x001a), sas_addr(0x5000c500302891d5), phy(12), device_name(0x5000c500302891d4) [ 12.467653] scsi 0:0:19:0: enclosure logical id (0x500093d002114000), slot(12) [ 12.467654] scsi 0:0:19:0: enclosure level(0x0000), connector name( ) [ 12.467656] scsi 0:0:19:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.526075] scsi 11:0:16:0: enclosure logical id (0x500093d002121000), slot(6) [ 12.534715] scsi 11:0:16:0: enclosure level(0x0000), connector name( ) [ 12.542876] scsi 11:0:16:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.571699] scsi 0:0:20:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.571724] scsi 11:0:17:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.571728] scsi 11:0:17:0: SSP: handle(0x001c), sas_addr(0x5000c500302e8f3a), phy(7), device_name(0x5000c500302e8f38) [ 12.571729] scsi 11:0:17:0: enclosure logical id (0x500093d002121000), slot(7) [ 12.571730] scsi 11:0:17:0: enclosure level(0x0000), connector name( ) [ 12.571732] scsi 11:0:17:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.630918] scsi 0:0:20:0: SSP: handle(0x001b), sas_addr(0x5000c5003029819d), phy(13), device_name(0x5000c5003029819c) [ 12.643339] scsi 0:0:20:0: enclosure logical id (0x500093d002114000), slot(13) [ 12.651979] scsi 0:0:20:0: enclosure level(0x0000), connector name( ) [ 12.660041] scsi 0:0:20:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.684734] scsi 0:0:21:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.684789] scsi 11:0:18:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.684793] scsi 11:0:18:0: SSP: handle(0x001d), sas_addr(0x5000c500302e8f5e), phy(8), device_name(0x5000c500302e8f5c) [ 12.684794] scsi 11:0:18:0: enclosure logical id (0x500093d002121000), slot(8) [ 12.684795] scsi 11:0:18:0: enclosure level(0x0000), connector name( ) [ 12.684797] scsi 11:0:18:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.743967] scsi 0:0:21:0: SSP: handle(0x001c), sas_addr(0x5000c500302891ed), phy(14), device_name(0x5000c500302891ec) [ 12.756380] scsi 0:0:21:0: enclosure logical id (0x500093d002114000), slot(14) [ 12.765020] scsi 0:0:21:0: enclosure level(0x0000), connector name( ) [ 12.773086] scsi 0:0:21:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.792825] scsi 11:0:19:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.802543] scsi 11:0:19:0: SSP: handle(0x001e), sas_addr(0x5000c500302e8ed6), phy(12), device_name(0x5000c500302e8ed4) [ 12.803617] scsi 0:0:22:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.803621] scsi 0:0:22:0: SSP: handle(0x0020), sas_addr(0x5000c50030298015), phy(18), device_name(0x5000c50030298014) [ 12.803622] scsi 0:0:22:0: enclosure logical id (0x500093d002114000), slot(18) [ 12.803624] scsi 0:0:22:0: enclosure level(0x0000), connector name( ) [ 12.803626] scsi 0:0:22:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.864365] scsi 11:0:19:0: enclosure logical id (0x500093d002121000), slot(12) [ 12.873104] scsi 11:0:19:0: enclosure level(0x0000), connector name( ) [ 12.881263] scsi 11:0:19:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.903659] scsi 0:0:23:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.913284] scsi 0:0:23:0: SSP: handle(0x0021), sas_addr(0x5000c50030289181), phy(19), device_name(0x5000c50030289180) [ 12.914624] scsi 11:0:20:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 12.914627] scsi 11:0:20:0: SSP: handle(0x001f), sas_addr(0x5000c500302e8ec6), phy(13), device_name(0x5000c500302e8ec4) [ 12.914629] scsi 11:0:20:0: enclosure logical id (0x500093d002121000), slot(13) [ 12.914630] scsi 11:0:20:0: enclosure level(0x0000), connector name( ) [ 12.914632] scsi 11:0:20:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 12.975492] scsi 0:0:23:0: enclosure logical id (0x500093d002114000), slot(19) [ 12.984134] scsi 0:0:23:0: enclosure level(0x0000), connector name( ) [ 12.992197] scsi 0:0:23:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.020653] scsi 11:0:21:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 13.030371] scsi 11:0:21:0: SSP: handle(0x0020), sas_addr(0x5000c500302e8f1e), phy(14), device_name(0x5000c500302e8f1c) [ 13.033595] scsi 0:0:24:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 13.033599] scsi 0:0:24:0: SSP: handle(0x0022), sas_addr(0x5000c50030297f79), phy(20), device_name(0x5000c50030297f78) [ 13.033600] scsi 0:0:24:0: enclosure logical id (0x500093d002114000), slot(20) [ 13.033601] scsi 0:0:24:0: enclosure level(0x0000), connector name( ) [ 13.033603] scsi 0:0:24:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.092191] scsi 11:0:21:0: enclosure logical id (0x500093d002121000), slot(14) [ 13.100929] scsi 11:0:21:0: enclosure level(0x0000), connector name( ) [ 13.109088] scsi 11:0:21:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.134649] scsi 11:0:22:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 13.144369] scsi 11:0:22:0: SSP: handle(0x0021), sas_addr(0x5000c500302e8eae), phy(18), device_name(0x5000c500302e8eac) [ 13.146527] scsi 0:0:25:0: Enclosure RAIDINC 24BAY EBOD 0105 PQ: 0 ANSI: 6 [ 13.146530] scsi 0:0:25:0: set ignore_delay_remove for handle(0x0024) [ 13.146532] scsi 0:0:25:0: SES: handle(0x0024), sas_addr(0x500093d00211403e), phy(48), device_name(0x0000000000000000) [ 13.146534] scsi 0:0:25:0: enclosure logical id (0x500093d002114000), slot(24) [ 13.146535] scsi 0:0:25:0: enclosure level(0x0000), connector name( ) [ 13.146537] scsi 0:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.213863] scsi 11:0:22:0: enclosure logical id (0x500093d002121000), slot(18) [ 13.222600] scsi 11:0:22:0: enclosure level(0x0000), connector name( ) [ 13.230760] scsi 11:0:22:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.255657] scsi 11:0:23:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 13.265377] scsi 11:0:23:0: SSP: handle(0x0022), sas_addr(0x5000c50030279dde), phy(19), device_name(0x5000c50030279ddc) [ 13.277896] scsi 11:0:23:0: enclosure logical id (0x500093d002121000), slot(19) [ 13.286632] scsi 11:0:23:0: enclosure level(0x0000), connector name( ) [ 13.294792] scsi 11:0:23:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.319663] scsi 11:0:24:0: Direct-Access SEAGATE ST1600FM0003 0007 PQ: 0 ANSI: 6 [ 13.329381] scsi 11:0:24:0: SSP: handle(0x0023), sas_addr(0x5000c500302b7ea2), phy(20), device_name(0x5000c500302b7ea0) [ 13.341899] scsi 11:0:24:0: enclosure logical id (0x500093d002121000), slot(20) [ 13.350637] scsi 11:0:24:0: enclosure level(0x0000), connector name( ) [ 13.358796] scsi 11:0:24:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 13.387655] scsi 11:0:25:0: Enclosure RAIDINC 24BAY EBOD 0105 PQ: 0 ANSI: 6 [ 13.397372] scsi 11:0:25:0: set ignore_delay_remove for handle(0x0024) [ 13.405139] scsi 11:0:25:0: SES: handle(0x0024), sas_addr(0x500093d00212107e), phy(48), device_name(0x0000000000000000) [ 13.417656] scsi 11:0:25:0: enclosure logical id (0x500093d002121000), slot(25) [ 13.426395] scsi 11:0:25:0: enclosure level(0x0000), connector name( ) [ 13.434554] scsi 11:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 14.841574] mpt3sas_cm0: port enable: SUCCESS [ 15.476542] mpt3sas_cm1: port enable: SUCCESS [ 15.492028] sd 0:0:0:0: [sda] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492039] sd 0:0:1:0: [sdb] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492040] sd 0:0:1:0: [sdb] 4096-byte physical blocks [ 15.492069] sd 0:0:2:0: [sdc] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492072] sd 0:0:2:0: [sdc] 4096-byte physical blocks [ 15.492129] sd 0:0:3:0: [sdd] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492130] sd 0:0:3:0: [sdd] 4096-byte physical blocks [ 15.492158] sd 0:0:4:0: [sde] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492159] sd 0:0:4:0: [sde] 4096-byte physical blocks [ 15.492200] sd 0:0:5:0: [sdf] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492202] sd 0:0:5:0: [sdf] 4096-byte physical blocks [ 15.492227] sd 0:0:6:0: [sdg] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492228] sd 0:0:6:0: [sdg] 4096-byte physical blocks [ 15.492284] sd 0:0:7:0: [sdh] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492286] sd 0:0:7:0: [sdh] 4096-byte physical blocks [ 15.492312] sd 0:0:8:0: [sdi] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492313] sd 0:0:8:0: [sdi] 4096-byte physical blocks [ 15.492489] sd 0:0:9:0: [sdj] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492490] sd 0:0:9:0: [sdj] 4096-byte physical blocks [ 15.492591] sd 0:0:10:0: [sdk] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492592] sd 0:0:10:0: [sdk] 4096-byte physical blocks [ 15.492815] sd 0:0:11:0: [sdl] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.492817] sd 0:0:11:0: [sdl] 4096-byte physical blocks [ 15.493416] sd 0:0:13:0: [sdm] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493417] sd 0:0:13:0: [sdm] 4096-byte physical blocks [ 15.493452] sd 0:0:14:0: [sdn] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493453] sd 0:0:14:0: [sdn] 4096-byte physical blocks [ 15.493486] sd 0:0:15:0: [sdo] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493487] sd 0:0:15:0: [sdo] 4096-byte physical blocks [ 15.493508] sd 0:0:16:0: [sdp] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493509] sd 0:0:16:0: [sdp] 4096-byte physical blocks [ 15.493599] sd 0:0:17:0: [sdq] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493600] sd 0:0:17:0: [sdq] 4096-byte physical blocks [ 15.493687] sd 0:0:1:0: [sdb] Write Protect is off [ 15.493689] sd 0:0:1:0: [sdb] Mode Sense: db 00 10 08 [ 15.493732] sd 0:0:2:0: [sdc] Write Protect is off [ 15.493733] sd 0:0:2:0: [sdc] Mode Sense: db 00 10 08 [ 15.493771] sd 0:0:3:0: [sdd] Write Protect is off [ 15.493773] sd 0:0:3:0: [sdd] Mode Sense: db 00 10 08 [ 15.493798] sd 0:0:4:0: [sde] Write Protect is off [ 15.493799] sd 0:0:4:0: [sde] Mode Sense: db 00 10 08 [ 15.493869] sd 0:0:6:0: [sdg] Write Protect is off [ 15.493870] sd 0:0:6:0: [sdg] Mode Sense: db 00 10 08 [ 15.493920] sd 0:0:18:0: [sdr] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.493921] sd 0:0:18:0: [sdr] 4096-byte physical blocks [ 15.493941] sd 0:0:7:0: [sdh] Write Protect is off [ 15.493943] sd 0:0:7:0: [sdh] Mode Sense: db 00 10 08 [ 15.493956] sd 0:0:8:0: [sdi] Write Protect is off [ 15.493957] sd 0:0:5:0: [sdf] Write Protect is off [ 15.493958] sd 0:0:8:0: [sdi] Mode Sense: db 00 10 08 [ 15.493959] sd 0:0:5:0: [sdf] Mode Sense: db 00 10 08 [ 15.494028] sd 0:0:19:0: [sds] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.494029] sd 0:0:19:0: [sds] 4096-byte physical blocks [ 15.494174] sd 0:0:9:0: [sdj] Write Protect is off [ 15.494176] sd 0:0:9:0: [sdj] Mode Sense: db 00 10 08 [ 15.494189] sd 0:0:20:0: [sdt] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.494191] sd 0:0:20:0: [sdt] 4096-byte physical blocks [ 15.494239] sd 0:0:10:0: [sdk] Write Protect is off [ 15.494240] sd 0:0:10:0: [sdk] Mode Sense: db 00 10 08 [ 15.494450] sd 0:0:11:0: [sdl] Write Protect is off [ 15.494451] sd 0:0:11:0: [sdl] Mode Sense: db 00 10 08 [ 15.494521] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494558] sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494601] sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494620] sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494698] sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494755] sd 0:0:21:0: [sdu] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.494757] sd 0:0:21:0: [sdu] 4096-byte physical blocks [ 15.494770] sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494785] sd 0:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.494790] sd 0:0:22:0: [sdv] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.494792] sd 0:0:22:0: [sdv] 4096-byte physical blocks [ 15.494999] sd 0:0:23:0: [sdw] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.495000] sd 0:0:23:0: [sdw] 4096-byte physical blocks [ 15.495007] sd 0:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.495067] sd 0:0:13:0: [sdm] Write Protect is off [ 15.495068] sd 0:0:13:0: [sdm] Mode Sense: db 00 10 08 [ 15.495069] sd 0:0:10:0: [sdk] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 15.495108] sd 0:0:14:0: [sdn] Write Protect is off [ 15.495110] sd 0:0:14:0: [sdn] Mode Sense: db 00 10 08 [ 15.495128] sd 0:0:15:0: [sdo] Write Protect is off [ 15.495130] sd 0:0:15:0: [sdo] Mode Sense: db 00 10 08 [ 15.495162] sd 0:0:16:0: [sdp] Write Protect is off [ 15.495163] sd 0:0:16:0: [sdp] Mode Sense: db 00 10 08 [ 15.495238] sd 0:0:17:0: [sdq] Write Protect is off [ 15.495240] sd 0:0:17:0: [sdq] Mode Sense: db 00 10 08 [ 15.495284] sd 0:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.495349] sd 0:0:24:0: [sdx] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.495351] sd 0:0:24:0: [sdx] 4096-byte physical blocks [ 15.495582] sd 0:0:18:0: [sdr] Write Protect is off [ 15.495584] sd 0:0:18:0: [sdr] Mode Sense: db 00 10 08 [ 15.495742] sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.495763] sd 11:0:0:0: [sdy] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.495765] sd 11:0:0:0: [sdy] 4096-byte physical blocks [ 15.495847] sd 0:0:20:0: [sdt] Write Protect is off [ 15.495849] sd 0:0:20:0: [sdt] Mode Sense: db 00 10 08 [ 15.496000] sd 11:0:1:0: [sdz] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.496002] sd 11:0:1:0: [sdz] 4096-byte physical blocks [ 15.496074] sd 0:0:17:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496099] sd 0:0:19:0: [sds] Write Protect is off [ 15.496101] sd 0:0:19:0: [sds] Mode Sense: db 00 10 08 [ 15.496123] sd 0:0:13:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496231] sd 0:0:15:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496244] sd 0:0:16:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496411] sd 0:0:18:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496425] sd 0:0:14:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496461] sd 0:0:22:0: [sdv] Write Protect is off [ 15.496463] sd 0:0:22:0: [sdv] Mode Sense: db 00 10 08 [ 15.496639] sd 11:0:2:0: [sdaa] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.496640] sd 11:0:2:0: [sdaa] 4096-byte physical blocks [ 15.496667] sd 0:0:21:0: [sdu] Write Protect is off [ 15.496669] sd 0:0:21:0: [sdu] Mode Sense: db 00 10 08 [ 15.496672] sd 0:0:23:0: [sdw] Write Protect is off [ 15.496673] sd 0:0:23:0: [sdw] Mode Sense: db 00 10 08 [ 15.496711] sd 0:0:20:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.496851] sd 11:0:3:0: [sdab] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.496853] sd 11:0:3:0: [sdab] 4096-byte physical blocks [ 15.496926] sd 11:0:4:0: [sdac] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.496928] sd 11:0:4:0: [sdac] 4096-byte physical blocks [ 15.496929] sd 0:0:19:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.497014] sd 0:0:24:0: [sdx] Write Protect is off [ 15.497016] sd 0:0:24:0: [sdx] Mode Sense: db 00 10 08 [ 15.497251] sd 11:0:5:0: [sdad] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.497253] sd 11:0:5:0: [sdad] 4096-byte physical blocks [ 15.497295] sd 0:0:22:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.497501] sd 0:0:23:0: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.497600] sd 11:0:6:0: [sdae] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.497602] sd 11:0:6:0: [sdae] 4096-byte physical blocks [ 15.497618] sd 0:0:21:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.497731] sd 11:0:7:0: [sdaf] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.497733] sd 11:0:7:0: [sdaf] 4096-byte physical blocks [ 15.498013] sd 0:0:24:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.498024] sd 11:0:8:0: [sdag] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.498026] sd 11:0:8:0: [sdag] 4096-byte physical blocks [ 15.498162] sd 11:0:9:0: [sdah] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.498164] sd 11:0:9:0: [sdah] 4096-byte physical blocks [ 15.498326] sd 11:0:0:0: [sdy] Write Protect is off [ 15.498328] sd 11:0:0:0: [sdy] Mode Sense: db 00 10 08 [ 15.498537] sd 11:0:1:0: [sdz] Write Protect is off [ 15.498538] sd 11:0:1:0: [sdz] Mode Sense: db 00 10 08 [ 15.498646] sd 11:0:10:0: [sdai] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.498647] sd 11:0:10:0: [sdai] 4096-byte physical blocks [ 15.498862] sd 11:0:11:0: [sdaj] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.498864] sd 11:0:11:0: [sdaj] 4096-byte physical blocks [ 15.499056] sd 11:0:13:0: [sdak] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.499057] sd 11:0:13:0: [sdak] 4096-byte physical blocks [ 15.499236] sd 11:0:4:0: [sdac] Write Protect is off [ 15.499238] sd 11:0:4:0: [sdac] Mode Sense: db 00 10 08 [ 15.499252] sd 11:0:2:0: [sdaa] Write Protect is off [ 15.499253] sd 11:0:2:0: [sdaa] Mode Sense: db 00 10 08 [ 15.499525] sd 11:0:0:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.499527] sd 11:0:14:0: [sdal] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.499529] sd 11:0:14:0: [sdal] 4096-byte physical blocks [ 15.499549] sd 11:0:5:0: [sdad] Write Protect is off [ 15.499550] sd 11:0:5:0: [sdad] Mode Sense: db 00 10 08 [ 15.499662] sd 11:0:3:0: [sdab] Write Protect is off [ 15.499664] sd 11:0:3:0: [sdab] Mode Sense: db 00 10 08 [ 15.499674] sd 11:0:15:0: [sdam] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.499676] sd 11:0:15:0: [sdam] 4096-byte physical blocks [ 15.499762] sd 11:0:1:0: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.499793] sd 11:0:16:0: [sdan] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.499795] sd 11:0:16:0: [sdan] 4096-byte physical blocks [ 15.499858] sd 11:0:6:0: [sdae] Write Protect is off [ 15.499860] sd 11:0:6:0: [sdae] Mode Sense: db 00 10 08 [ 15.500112] sd 11:0:17:0: [sdao] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.500113] sd 11:0:17:0: [sdao] 4096-byte physical blocks [ 15.500204] sd 11:0:7:0: [sdaf] Write Protect is off [ 15.500206] sd 11:0:7:0: [sdaf] Mode Sense: db 00 10 08 [ 15.500351] sd 11:0:8:0: [sdag] Write Protect is off [ 15.500353] sd 11:0:8:0: [sdag] Mode Sense: db 00 10 08 [ 15.500376] sd 11:0:18:0: [sdap] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.500378] sd 11:0:18:0: [sdap] 4096-byte physical blocks [ 15.500430] sd 11:0:4:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.500450] sd 11:0:2:0: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.500619] sd 11:0:9:0: [sdah] Write Protect is off [ 15.500621] sd 11:0:9:0: [sdah] Mode Sense: db 00 10 08 [ 15.500637] sd 11:0:19:0: [sdaq] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.500639] sd 11:0:19:0: [sdaq] 4096-byte physical blocks [ 15.500675] sd 11:0:20:0: [sdar] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.500676] sd 11:0:20:0: [sdar] 4096-byte physical blocks [ 15.500705] sd 11:0:13:0: [sdak] Write Protect is off [ 15.500706] sd 11:0:13:0: [sdak] Mode Sense: db 00 10 08 [ 15.500752] sd 11:0:5:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.500844] sd 11:0:21:0: [sdas] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.500845] sd 11:0:21:0: [sdas] 4096-byte physical blocks [ 15.500864] sd 11:0:3:0: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501060] sd 11:0:6:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501209] sd 11:0:22:0: [sdat] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.501210] sd 11:0:22:0: [sdat] 4096-byte physical blocks [ 15.501264] sd 11:0:10:0: [sdai] Write Protect is off [ 15.501265] sd 11:0:10:0: [sdai] Mode Sense: db 00 10 08 [ 15.501327] sd 11:0:11:0: [sdaj] Write Protect is off [ 15.501329] sd 11:0:11:0: [sdaj] Mode Sense: db 00 10 08 [ 15.501403] sd 11:0:7:0: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501428] sd 11:0:23:0: [sdau] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.501429] sd 11:0:23:0: [sdau] 4096-byte physical blocks [ 15.501543] sd 11:0:13:0: [sdak] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501567] sd 11:0:8:0: [sdag] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501707] sd 11:0:24:0: [sdav] 3125627568 512-byte logical blocks: (1.60 TB/1.45 TiB) [ 15.501708] sd 11:0:24:0: [sdav] 4096-byte physical blocks [ 15.501842] sd 11:0:9:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.501867] sd 11:0:14:0: [sdal] Write Protect is off [ 15.501869] sd 11:0:14:0: [sdal] Mode Sense: db 00 10 08 [ 15.502000] sd 11:0:16:0: [sdan] Write Protect is off [ 15.502008] sd 11:0:16:0: [sdan] Mode Sense: db 00 10 08 [ 15.502043] sd 11:0:15:0: [sdam] Write Protect is off [ 15.502045] sd 11:0:15:0: [sdam] Mode Sense: db 00 10 08 [ 15.502439] sd 11:0:17:0: [sdao] Write Protect is off [ 15.502441] sd 11:0:17:0: [sdao] Mode Sense: db 00 10 08 [ 15.502527] sd 11:0:11:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.502666] sd 11:0:10:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.502893] sd 11:0:18:0: [sdap] Write Protect is off [ 15.502894] sd 11:0:18:0: [sdap] Mode Sense: db 00 10 08 [ 15.503063] sd 11:0:14:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.503204] sd 11:0:16:0: [sdan] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.503266] sd 11:0:15:0: [sdam] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.503349] sd 11:0:21:0: [sdas] Write Protect is off [ 15.503350] sd 11:0:21:0: [sdas] Mode Sense: db 00 10 08 [ 15.503645] sd 11:0:17:0: [sdao] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.503865] sd 11:0:19:0: [sdaq] Write Protect is off [ 15.503867] sd 11:0:19:0: [sdaq] Mode Sense: db 00 10 08 [ 15.503905] sd 11:0:20:0: [sdar] Write Protect is off [ 15.503906] sd 11:0:20:0: [sdar] Mode Sense: db 00 10 08 [ 15.504097] sd 11:0:18:0: [sdap] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.504455] sd 11:0:22:0: [sdat] Write Protect is off [ 15.504457] sd 11:0:22:0: [sdat] Mode Sense: db 00 10 08 [ 15.504666] sd 11:0:21:0: [sdas] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.504669] sd 11:0:23:0: [sdau] Write Protect is off [ 15.504671] sd 11:0:23:0: [sdau] Mode Sense: db 00 10 08 [ 15.505249] sd 11:0:19:0: [sdaq] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.505273] sd 11:0:20:0: [sdar] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.505398] sd 11:0:24:0: [sdav] Write Protect is off [ 15.505399] sd 11:0:24:0: [sdav] Mode Sense: db 00 10 08 [ 15.505813] sd 0:0:6:0: [sdg] Attached SCSI disk [ 15.505833] sd 11:0:22:0: [sdat] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.505863] sd 0:0:1:0: [sdb] Attached SCSI disk [ 15.505878] sd 0:0:2:0: [sdc] Attached SCSI disk [ 15.505905] sd 0:0:7:0: [sdh] Attached SCSI disk [ 15.506040] sd 11:0:23:0: [sdau] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 15.506193] sd 0:0:11:0: [sdl] Attached SCSI disk [ 15.506206] sd 0:0:3:0: [sdd] Attached SCSI disk [ 15.506395] sd 0:0:9:0: [sdj] Attached SCSI disk [ 15.506587] sd 11:0:24:0: [sdav] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.506605] sd 0:0:10:0: [sdk] Attached SCSI disk [ 15.506929] sd 0:0:8:0: [sdi] Attached SCSI disk [ 15.507047] sd 0:0:4:0: [sde] Attached SCSI disk [ 15.507243] sd 0:0:5:0: [sdf] Attached SCSI disk [ 15.509877] sd 11:0:13:0: [sdak] Attached SCSI disk [ 15.509913] sd 0:0:16:0: [sdp] Attached SCSI disk [ 15.509925] sd 0:0:17:0: [sdq] Attached SCSI disk [ 15.510409] sd 0:0:20:0: [sdt] Attached SCSI disk [ 15.510434] sd 0:0:18:0: [sdr] Attached SCSI disk [ 15.510699] sd 0:0:13:0: [sdm] Attached SCSI disk [ 15.510787] sd 0:0:14:0: [sdn] Attached SCSI disk [ 15.511089] sd 0:0:15:0: [sdo] Attached SCSI disk [ 15.511091] sd 0:0:22:0: [sdv] Attached SCSI disk [ 15.511353] sd 0:0:23:0: [sdw] Attached SCSI disk [ 15.511692] sd 0:0:19:0: [sds] Attached SCSI disk [ 15.512109] sd 0:0:21:0: [sdu] Attached SCSI disk [ 15.512216] sd 0:0:24:0: [sdx] Attached SCSI disk [ 15.513041] sd 11:0:15:0: [sdam] Attached SCSI disk [ 15.513172] sd 11:0:0:0: [sdy] Attached SCSI disk [ 15.513223] sd 11:0:4:0: [sdac] Attached SCSI disk [ 15.513245] sd 11:0:1:0: [sdz] Attached SCSI disk [ 15.513411] sd 11:0:5:0: [sdad] Attached SCSI disk [ 15.513424] sd 11:0:2:0: [sdaa] Attached SCSI disk [ 15.513436] sd 11:0:3:0: [sdab] Attached SCSI disk [ 15.513913] sd 11:0:17:0: [sdao] Attached SCSI disk [ 15.514048] sd 11:0:6:0: [sdae] Attached SCSI disk [ 15.514056] sd 11:0:19:0: [sdaq] Attached SCSI disk [ 15.514102] sd 11:0:21:0: [sdas] Attached SCSI disk [ 15.514168] sd 11:0:7:0: [sdaf] Attached SCSI disk [ 15.514509] sd 11:0:22:0: [sdat] Attached SCSI disk [ 15.514525] sd 11:0:9:0: [sdah] Attached SCSI disk [ 15.514567] sd 11:0:8:0: [sdag] Attached SCSI disk [ 15.514663] sd 11:0:23:0: [sdau] Attached SCSI disk [ 15.514676] sd 11:0:16:0: [sdan] Attached SCSI disk [ 15.514714] sd 11:0:24:0: [sdav] Attached SCSI disk [ 15.514978] sd 11:0:20:0: [sdar] Attached SCSI disk [ 15.514996] sd 11:0:10:0: [sdai] Attached SCSI disk [ 15.515487] sd 11:0:14:0: [sdal] Attached SCSI disk [ 15.516079] sd 11:0:11:0: [sdaj] Attached SCSI disk [ 15.516099] sd 11:0:18:0: [sdap] Attached SCSI disk [ 17.271822] sd 0:0:0:0: [sda] 4096-byte physical blocks [ 17.279819] sd 0:0:0:0: [sda] Write Protect is off [ 17.285658] sd 0:0:0:0: [sda] Mode Sense: db 00 10 08 [ 17.286496] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 17.305190] sd 0:0:0:0: [sda] Attached SCSI disk [ 17.425661] device-mapper: multipath service-time: version 0.3.0 loaded [ 17.465483] type=1131 audit(1644345231.613:15): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.498579] type=1130 audit(1644345231.646:16): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 19.517965] scsi host12: iSCSI Initiator over TCP/IP [ 19.534502] scsi 12:0:0:0: Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5 [ 19.559413] scsi 12:0:0:0: alua: supports implicit and explicit TPGS [ 19.567004] scsi 12:0:0:0: alua: device naa.6001405fc6df94637210ee05ea428967 port group 0 rel port 1 [ 19.577679] scsi 12:0:0:0: alua: Attached [ 19.582839] sd 12:0:0:0: alua: transition timeout set to 60 seconds [ 19.590357] sd 12:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA [ 19.599414] sd 12:0:0:0: [sdaw] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 19.609467] sd 12:0:0:0: [sdaw] Write Protect is on [ 19.615948] sd 12:0:0:0: [sdaw] Mode Sense: 43 00 80 08 [ 19.616420] sd 12:0:0:0: [sdaw] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 19.634406] type=1130 audit(1644345233.782:17): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsistart_192.168.64.64::811:0:iqn.2006\x2d04.gov.llnl:lc.copper:lustre1\x2d3.7\x2d17\x2droot\x2d32261\x2dgcdbb4dd.x86\x2d64 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 19.667344] sd 12:0:0:0: [sdaw] Optimal transfer size 4194304 bytes [ 19.679212] sd 12:0:0:0: [sdaw] Attached SCSI disk [ 21.637610] scsi host13: iSCSI Initiator over TCP/IP [ 21.652556] scsi 13:0:0:0: Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5 [ 21.675311] scsi 13:0:0:0: alua: supports implicit and explicit TPGS [ 21.682900] scsi 13:0:0:0: alua: device naa.6001405fc6df94637210ee05ea428967 port group 0 rel port 1 [ 21.693584] scsi 13:0:0:0: alua: Attached [ 21.698847] sd 13:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA [ 21.707907] sd 13:0:0:0: [sdax] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 21.718429] sd 13:0:0:0: [sdax] Write Protect is on [ 21.724628] sd 13:0:0:0: [sdax] Mode Sense: 43 00 80 08 [ 21.725092] sd 13:0:0:0: [sdax] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 21.743312] type=1130 audit(1644345235.891:18): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsistart_192.168.64.65::811:0:iqn.2006\x2d04.gov.llnl:lc.copper:lustre1\x2d3.7\x2d17\x2droot\x2d32261\x2dgcdbb4dd.x86\x2d64 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 21.776278] sd 13:0:0:0: [sdax] Optimal transfer size 4194304 bytes [ 21.788377] sd 13:0:0:0: [sdax] Attached SCSI disk [ 21.816738] device-mapper: table: 253:24: multipath: error getting device [ 21.824815] device-mapper: ioctl: error adding target to table [ 21.837637] device-mapper: table: 253:24: multipath: error getting device [ 21.845711] device-mapper: ioctl: error adding target to table [ 21.897282] type=1130 audit(1644345236.045:19): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 22.673319] type=1130 audit(1644345236.821:20): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 22.727319] type=1130 audit(1644345236.875:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 22.806819] EXT4-fs (dm-24): mounted filesystem without journal. Opts: (null) [ 22.899264] type=1130 audit(1644345237.047:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 22.921309] type=1131 audit(1644345237.069:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 22.977899] TECH PREVIEW: Overlay filesystem may not be fully supported. Please review provided documentation for limitations. [ 23.009471] FS-Cache: Loaded [ 23.036734] FS-Cache: Netfs 'nfs' registered for caching [ 23.045494] Key type dns_resolver registered [ 23.073018] NFS: Registering the id_resolver key type [ 23.079150] Key type id_resolver registered [ 23.084314] Key type id_legacy registered [ 29.098394] type=1305 audit(1644345243.246:24): audit_backlog_limit=320 old=64 auid=4294967295 ses=4294967295 subj=kernel res=1 [ 29.117908] type=1305 audit(1644345243.266:25): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 29.143909] type=1305 audit(1644345243.292:26): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 29.161934] type=1305 audit(1644345243.310:27): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 29.179901] type=1305 audit(1644345243.328:28): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 29.197932] type=1305 audit(1644345243.346:29): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 29.211668] type=1305 audit(1644345243.360:30): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=0 [ 29.225414] type=1305 audit(1644345243.373:31): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=0 [ 29.243915] type=1305 audit(1644345243.392:32): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="tmpdir" list=4 res=1 [ 29.261934] type=1305 audit(1644345243.410:33): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="tmpdir" list=4 res=1 [ 34.113969] audit_printk_skb: 5991 callbacks suppressed [ 34.120309] type=1300 audit(1644345248.262:473): arch=c000003e syscall=59 success=yes exit=0 a0=1c5c220 a1=1c9c020 a2=1c63960 a3=8 items=2 ppid=1952 pid=2805 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="umount" exe="/usr/bin/umount" subj=kernel key="root_commands" [ 34.152612] type=1309 audit(1644345248.262:473): argc=2 a0="umount" a1="/sysroot/var/cfengine/inputs" [ 34.163439] type=1307 audit(1644345248.262:473): cwd="/" [ 34.169959] type=1302 audit(1644345248.262:473): item=0 name="/usr/bin/umount" inode=11283 dev=00:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 34.193727] type=1302 audit(1644345248.262:473): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=7927 dev=00:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 34.218565] type=1327 audit(1644345248.262:473): proctitle=756D6F756E74002F737973726F6F742F7661722F6366656E67696E652F696E70757473 [ 34.253658] type=1300 audit(1644345248.401:474): arch=c000003e syscall=59 success=yes exit=0 a0=1cb7820 a1=1cb8fb0 a2=1c63960 a3=8 items=2 ppid=1952 pid=2807 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="udevadm" exe="/usr/bin/udevadm" subj=kernel key="root_commands" [ 34.286157] type=1309 audit(1644345248.401:474): argc=3 a0="udevadm" a1="settle" a2="--timeout=30" [ 34.296647] type=1307 audit(1644345248.401:474): cwd="/" [ 34.303159] type=1302 audit(1644345248.401:474): item=0 name="/usr/sbin/udevadm" inode=11280 dev=00:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 35.191026] systemd-journald[286]: Received SIGTERM from PID 1 (systemd). [ 35.260801] SELinux: Disabled at runtime. [ 35.265894] SELinux: Unregistering netfilter hooks [ 35.332977] ip_tables: (C) 2000-2006 Netfilter Core Team [ 35.340475] systemd[1]: Inserted module 'ip_tables' [ 35.898582] SPL: Loaded module v0.7.11-9_g76578aa [ 35.906156] znvpair: module license 'CDDL' taints kernel. [ 35.914106] Disabling lock debugging due to kernel taint [ 38.404636] ZFS: Loaded module v0.7.11-17_gb1cd24f, ZFS pool version 5000, ZFS filesystem version 5 [ 38.509465] systemd-journald[2884]: Received request to flush runtime journal from PID 1 [ 39.121568] audit_printk_skb: 7237 callbacks suppressed [ 39.127904] type=1300 audit(1644345253.255:939): arch=c000003e syscall=175 success=no exit=-17 a0=55c845d42e20 a1=5e41 a2=7f37472e5079 a3=55c845d4b400 items=0 ppid=2908 pid=2914 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-udevd" exe="/usr/lib/systemd/systemd-udevd" key="modules" [ 39.162556] type=1330 audit(1644345253.255:939): name="pcc_cpufreq" [ 39.169874] type=1327 audit(1644345253.255:939): proctitle="/usr/lib/systemd/systemd-udevd" [ 39.183943] power_meter ACPI000D:00: Found ACPI power meter. [ 39.191417] power_meter ACPI000D:00: Ignoring unsafe software power cap! [ 39.201138] type=1300 audit(1644345253.268:940): arch=c000003e syscall=175 success=no exit=-17 a0=55c845d8d9e0 a1=5e41 a2=7f37472e5079 a3=55c845d8d700 items=0 ppid=2908 pid=3041 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-udevd" exe="/usr/lib/systemd/systemd-udevd" key="modules" [ 39.201188] type=1300 audit(1644345253.231:941): arch=c000003e syscall=175 success=yes exit=0 a0=55c845d42e20 a1=6771 a2=7f37472e5079 a3=55c845d4a400 items=0 ppid=2908 pid=3103 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-udevd" exe="/usr/lib/systemd/systemd-udevd" key="modules" [ 39.201191] type=1330 audit(1644345253.231:941): name="acpi_power_meter" [ 39.201196] type=1327 audit(1644345253.231:941): proctitle="/usr/lib/systemd/systemd-udevd" [ 39.289567] type=1330 audit(1644345253.268:940): name="pcc_cpufreq" [ 39.297057] type=1327 audit(1644345253.268:940): proctitle="/usr/lib/systemd/systemd-udevd" [ 39.312024] type=1300 audit(1644345253.270:942): arch=c000003e syscall=175 success=no exit=-17 a0=55c845d824c0 a1=5e41 a2=7f37472e5079 a3=55c845d3eb00 items=0 ppid=2908 pid=3032 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-udevd" exe="/usr/lib/systemd/systemd-udevd" key="modules" [ 39.546365] device-mapper: multipath round-robin: version 1.2.0 loaded [ 39.582928] device-mapper: table: 253:24: multipath: error getting device [ 39.591017] device-mapper: ioctl: error adding target to table [ 40.702866] IPMI message handler: version 39.2 [ 40.710421] ipmi device interface [ 40.722192] ipmi_si: IPMI System Interface driver [ 40.727965] ipmi_si dmi-ipmi-si.0: probing via SMBIOS [ 40.734100] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 40.742686] ipmi_si: Adding SMBIOS-specified kcs state machine [ 40.749878] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 40.828569] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 40.835091] sd 0:0:1:0: Attached scsi generic sg1 type 0 [ 40.841585] sd 0:0:2:0: Attached scsi generic sg2 type 0 [ 40.848131] sd 0:0:3:0: Attached scsi generic sg3 type 0 [ 40.854777] sd 0:0:4:0: Attached scsi generic sg4 type 0 [ 40.861305] sd 0:0:5:0: Attached scsi generic sg5 type 0 [ 40.867807] sd 0:0:6:0: Attached scsi generic sg6 type 0 [ 40.874360] sd 0:0:7:0: Attached scsi generic sg7 type 0 [ 40.881028] ipmi_si dmi-ipmi-si.0: Found new BMC (man_id: 0x000157, prod_id: 0x006f, dev_id: 0x21) [ 40.881211] sd 0:0:8:0: Attached scsi generic sg8 type 0 [ 40.881341] sd 0:0:9:0: Attached scsi generic sg9 type 0 [ 40.881418] sd 0:0:10:0: Attached scsi generic sg10 type 0 [ 40.881490] sd 0:0:11:0: Attached scsi generic sg11 type 0 [ 40.881537] scsi 0:0:12:0: Attached scsi generic sg12 type 13 [ 40.881572] sd 0:0:13:0: Attached scsi generic sg13 type 0 [ 40.881634] sd 0:0:14:0: Attached scsi generic sg14 type 0 [ 40.881703] sd 0:0:15:0: Attached scsi generic sg15 type 0 [ 40.881773] sd 0:0:16:0: Attached scsi generic sg16 type 0 [ 40.881835] sd 0:0:17:0: Attached scsi generic sg17 type 0 [ 40.881909] sd 0:0:18:0: Attached scsi generic sg18 type 0 [ 40.881966] sd 0:0:19:0: Attached scsi generic sg19 type 0 [ 40.882011] sd 0:0:20:0: Attached scsi generic sg20 type 0 [ 40.882049] sd 0:0:21:0: Attached scsi generic sg21 type 0 [ 40.882085] sd 0:0:22:0: Attached scsi generic sg22 type 0 [ 40.882130] sd 0:0:23:0: Attached scsi generic sg23 type 0 [ 40.882174] sd 0:0:24:0: Attached scsi generic sg24 type 0 [ 40.882213] scsi 0:0:25:0: Attached scsi generic sg25 type 13 [ 40.882252] sd 11:0:0:0: Attached scsi generic sg26 type 0 [ 40.882292] sd 11:0:1:0: Attached scsi generic sg27 type 0 [ 40.882336] sd 11:0:2:0: Attached scsi generic sg28 type 0 [ 40.882373] sd 11:0:3:0: Attached scsi generic sg29 type 0 [ 40.882408] sd 11:0:4:0: Attached scsi generic sg30 type 0 [ 40.882446] sd 11:0:5:0: Attached scsi generic sg31 type 0 [ 40.882484] sd 11:0:6:0: Attached scsi generic sg32 type 0 [ 40.882518] sd 11:0:7:0: Attached scsi generic sg33 type 0 [ 40.882558] sd 11:0:8:0: Attached scsi generic sg34 type 0 [ 40.882595] sd 11:0:9:0: Attached scsi generic sg35 type 0 [ 40.882633] sd 11:0:10:0: Attached scsi generic sg36 type 0 [ 40.882687] sd 11:0:11:0: Attached scsi generic sg37 type 0 [ 40.882734] scsi 11:0:12:0: Attached scsi generic sg38 type 13 [ 40.883403] sd 11:0:13:0: Attached scsi generic sg39 type 0 [ 40.883513] sd 11:0:14:0: Attached scsi generic sg40 type 0 [ 40.883654] sd 11:0:15:0: Attached scsi generic sg41 type 0 [ 40.883777] sd 11:0:16:0: Attached scsi generic sg42 type 0 [ 40.883908] sd 11:0:17:0: Attached scsi generic sg43 type 0 [ 40.884046] sd 11:0:18:0: Attached scsi generic sg44 type 0 [ 40.884183] sd 11:0:19:0: Attached scsi generic sg45 type 0 [ 40.884308] sd 11:0:20:0: Attached scsi generic sg46 type 0 [ 40.884442] sd 11:0:21:0: Attached scsi generic sg47 type 0 [ 40.884567] sd 11:0:22:0: Attached scsi generic sg48 type 0 [ 40.884678] sd 11:0:23:0: Attached scsi generic sg49 type 0 [ 40.884810] sd 11:0:24:0: Attached scsi generic sg50 type 0 [ 40.884927] scsi 11:0:25:0: Attached scsi generic sg51 type 13 [ 40.884975] sd 12:0:0:0: Attached scsi generic sg52 type 0 [ 40.885124] sd 13:0:0:0: Attached scsi generic sg53 type 0 [ 41.102255] ses 0:0:12:0: Attached Enclosure device [ 41.102278] ses 0:0:25:0: Attached Enclosure device [ 41.102297] ses 11:0:12:0: Attached Enclosure device [ 41.102321] ses 11:0:25:0: Attached Enclosure device [ 41.222082] ioatdma: Intel(R) QuickData Technology Driver 4.00 [ 41.230844] ioatdma 0000:00:04.0: irq 246 for MSI/MSI-X [ 41.240334] ioatdma 0000:00:04.1: irq 248 for MSI/MSI-X [ 41.247187] ioatdma 0000:00:04.2: irq 249 for MSI/MSI-X [ 41.253788] ioatdma 0000:00:04.3: irq 250 for MSI/MSI-X [ 41.260469] ioatdma 0000:00:04.4: irq 251 for MSI/MSI-X [ 41.267334] ioatdma 0000:00:04.5: irq 252 for MSI/MSI-X [ 41.273961] ioatdma 0000:00:04.6: irq 253 for MSI/MSI-X [ 41.281975] ioatdma 0000:00:04.7: irq 254 for MSI/MSI-X [ 41.290041] i801_smbus 0000:00:11.1: Enabling SMBus device [ 41.290082] ioatdma 0000:80:04.0: irq 256 for MSI/MSI-X [ 41.290464] ioatdma 0000:80:04.1: irq 258 for MSI/MSI-X [ 41.290754] ioatdma 0000:80:04.2: irq 259 for MSI/MSI-X [ 41.291034] ioatdma 0000:80:04.3: irq 260 for MSI/MSI-X [ 41.291302] ioatdma 0000:80:04.4: irq 261 for MSI/MSI-X [ 41.291573] ioatdma 0000:80:04.5: irq 262 for MSI/MSI-X [ 41.291837] ioatdma 0000:80:04.6: irq 263 for MSI/MSI-X [ 41.292103] ioatdma 0000:80:04.7: irq 264 for MSI/MSI-X [ 41.348559] i801_smbus 0000:00:11.1: SMBus using PCI interrupt [ 41.355772] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 41.361429] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt [ 41.395437] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized [ 41.423053] kvm: disabled by bios [ 41.455216] kvm: disabled by bios [ 41.463153] intel_rapl: Found RAPL domain package [ 41.468907] intel_rapl: Found RAPL domain dram [ 41.474358] intel_rapl: DRAM domain energy unit 15300pj [ 41.480718] intel_rapl: Found RAPL domain package [ 41.486466] intel_rapl: Found RAPL domain dram [ 41.491945] intel_rapl: DRAM domain energy unit 15300pj [ 41.501608] kvm: disabled by bios [ 41.512858] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 41.519202] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 41.525538] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 41.531863] EDAC sbridge: Seeking for: PCI ID 8086:6f60 [ 41.538202] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 41.544522] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 41.550852] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 41.557180] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 41.563502] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 41.571377] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 41.579241] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 41.587102] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 41.594974] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 41.602160] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 41.608480] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 41.614804] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 41.621129] EDAC sbridge: Seeking for: PCI ID 8086:6fac [ 41.627447] EDAC sbridge: Seeking for: PCI ID 8086:6fac [ 41.633770] EDAC sbridge: Seeking for: PCI ID 8086:6fac [ 41.640094] EDAC sbridge: Seeking for: PCI ID 8086:6fad [ 41.646417] EDAC sbridge: Seeking for: PCI ID 8086:6fad [ 41.652743] EDAC sbridge: Seeking for: PCI ID 8086:6fad [ 41.659060] EDAC sbridge: Seeking for: PCI ID 8086:6f68 [ 41.665382] EDAC sbridge: Seeking for: PCI ID 8086:6f79 [ 41.671711] EDAC sbridge: Seeking for: PCI ID 8086:6f6a [ 41.678043] EDAC sbridge: Seeking for: PCI ID 8086:6f6b [ 41.684373] EDAC sbridge: Seeking for: PCI ID 8086:6f6c [ 41.690703] EDAC sbridge: Seeking for: PCI ID 8086:6f6d [ 41.697031] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 41.703349] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 41.709675] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 41.715998] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 41.722318] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 41.728640] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 41.734962] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 41.741288] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 41.747613] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 41.754203] EDAC MC0: Giving out device to 'sb_edac.c' 'Broadwell SrcID#1_Ha#0': DEV 0000:ff:12.0 [ 41.764796] EDAC MC1: Giving out device to 'sb_edac.c' 'Broadwell SrcID#0_Ha#0': DEV 0000:7f:12.0 [ 41.776663] EDAC sbridge: Ver: 1.1.2 [ 41.783343] kvm: disabled by bios [ 41.783430] iscsi: registered transport (iser) [ 41.805982] kvm: disabled by bios [ 41.815228] RPC: Registered rdma transport module. [ 41.821079] RPC: Registered rdma backchannel transport module. [ 41.831539] kvm: disabled by bios [ 41.846096] kvm: disabled by bios [ 41.861650] kvm: disabled by bios [ 41.877393] kvm: disabled by bios [ 41.891619] kvm: disabled by bios [ 41.904921] kvm: disabled by bios [ 41.926510] kvm: disabled by bios [ 41.946624] kvm: disabled by bios [ 41.962064] kvm: disabled by bios [ 41.984551] kvm: disabled by bios [ 42.007981] kvm: disabled by bios [ 42.025516] kvm: disabled by bios [ 42.044898] kvm: disabled by bios [ 42.063485] kvm: disabled by bios [ 42.081585] kvm: disabled by bios [ 42.099550] kvm: disabled by bios [ 42.114694] kvm: disabled by bios [ 42.133447] kvm: disabled by bios [ 42.164433] kvm: disabled by bios [ 42.195568] kvm: disabled by bios [ 42.215701] kvm: disabled by bios [ 42.244435] kvm: disabled by bios [ 42.287703] kvm: disabled by bios [ 42.314659] kvm: disabled by bios [ 42.340860] kvm: disabled by bios [ 42.364079] kvm: disabled by bios [ 42.394752] kvm: disabled by bios [ 42.488436] iTCO_vendor_support: vendor-support=0 [ 42.497753] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 42.504583] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 82.632324] IPv6: ADDRCONF(NETDEV_CHANGE): lo: link becomes ready [ 83.352128] ib0: enabling connected mode will cause multicast packet drops [ 83.360383] ib0: mtu > 4092 will cause multicast packet drops. [ 83.393903] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 83.404328] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 130.010503] LNet: HW NUMA nodes: 2, HW CPU cores: 32, npartitions: 2 [ 130.020895] alg: No test for adler32 (adler32-zlib) [ 130.811409] LNet: 30704:0:(config.c:1641:lnet_inet_enumerate()) lnet: Ignoring interface eno2: it's down [ 130.822808] LNet: Using FastReg for registration [ 130.827681] LNetError: 232:0:(o2iblnd_cb.c:2526:kiblnd_passive_connect()) Can't accept conn from 172.19.3.56@o2ib600 on NA (ib0:0:172.19.3.99): bad dst nid 172.19.3.99@o2ib600 [ 130.930043] LNet: Added LNI 172.19.3.99@o2ib600 [8/256/0/180] [ 833.025900] Lustre: Lustre: Build Version: 2.12.8_6.llnl [ 883.443940] LustreError: 137-5: ls1-MDT0001_UUID: not available for connect from 192.168.131.54@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.774626] LNet: 30707:0:(o2iblnd_cb.c:3432:kiblnd_check_conns()) Timed out tx for 172.19.3.98@o2ib600: 0 seconds [ 884.028819] LustreError: 137-5: ls1-MDT0001_UUID: not available for connect from 192.168.132.97@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.049109] LustreError: Skipped 15 previous similar messages [ 885.064361] LustreError: 137-5: ls1-MDT0001_UUID: not available for connect from 192.168.128.228@o2ib36 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.084749] LustreError: Skipped 36 previous similar messages [ 887.065933] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.128.45@o2ib37 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.086217] LustreError: Skipped 195 previous similar messages [ 891.069932] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.132.102@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.090316] LustreError: Skipped 278 previous similar messages [ 899.074334] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.132.83@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.094620] LustreError: Skipped 1041 previous similar messages [ 900.499906] LustreError: 2109:0:(mgc_request.c:249:do_config_log_add()) MGC172.19.3.98@o2ib600: failed processing log, type 1: rc = -5 [ 915.221734] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.128.51@o2ib39 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.242025] LustreError: Skipped 1369 previous similar messages [ 947.220517] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 172.19.3.163@o2ib600 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.240710] LustreError: Skipped 1572 previous similar messages [ 976.510264] LustreError: 2109:0:(mgc_request.c:249:do_config_log_add()) MGC172.19.3.98@o2ib600: failed processing log, type 4: rc = -110 [ 988.553404] Lustre: ls1-MDT0001: Not available for connect from 192.168.128.156@o2ib35 (not set up) [ 988.564012] Lustre: Skipped 2 previous similar messages [ 1011.221714] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.128.227@o2ib39 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.242094] LustreError: Skipped 7973 previous similar messages [ 1032.918564] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [mdt01_007:3041] [ 1032.927595] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1033.007491] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1033.067838] CPU: 15 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OE ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1033.082101] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1033.094437] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1033.103271] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1033.112902] RSP: 0018:ffff88528ad4fae0 EFLAGS: 00000246 [ 1033.119312] RAX: 0000000000000000 RBX: ffff88528f991d30 RCX: ffff88528ad4ffd8 [ 1033.127757] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88528f991ccc [ 1033.136202] RBP: ffff88528ad4faf8 R08: ffff88528ad4fb48 R09: 0000000000000000 [ 1033.144648] R10: 0000000000000001 R11: 0000000000000005 R12: 0000000000000001 [ 1033.153094] R13: 0000000000000005 R14: 0000000000000001 R15: ffff88528ad4fac8 [ 1033.161543] FS: 0000000000000000(0000) GS:ffff8892ff3c0000(0000) knlGS:0000000000000000 [ 1033.171056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1033.177951] CR2: 00007ffff6dab5b0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1033.186401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1033.194848] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1033.203294] Call Trace: [ 1033.206524] [] ? lnet_peer_is_uptodate+0x1b/0xa0 [lnet] [ 1033.214684] [] lnet_discover_peer_locked+0x2b3/0x450 [lnet] [ 1033.223224] [] ? wake_up_atomic_t+0x30/0x30 [ 1033.230220] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1033.237649] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1033.245920] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1033.253903] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1033.262560] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1033.270442] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1033.277753] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1033.285928] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1033.294864] [] ? __getnstimeofday64+0x3f/0xd0 [ 1033.302070] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1033.311115] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1033.319178] [] ? __wake_up_common_lock+0x91/0xc0 [ 1033.326649] [] ? sched_feat_set+0xf0/0xf0 [ 1033.333464] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1033.340945] [] ? __switch_to+0xce/0x5a0 [ 1033.347573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1033.356312] [] kthread+0xd1/0xe0 [ 1033.362238] [] ? insert_kthread_work+0x40/0x40 [ 1033.369525] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1033.377285] [] ? insert_kthread_work+0x40/0x40 [ 1033.384567] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1060.917262] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [mdt01_007:3041] [ 1060.926292] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1061.006157] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1061.066455] CPU: 15 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1061.080718] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1061.093042] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1061.101876] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1061.113440] RSP: 0018:ffff88528ad4fab0 EFLAGS: 00000246 [ 1061.119850] RAX: 0000000000000001 RBX: ffff8892ff3c0000 RCX: 0000000000790000 [ 1061.128296] RDX: ffff8892ff31b8c0 RSI: 0000000000610001 RDI: ffff8852f5420480 [ 1061.136740] RBP: ffff88528ad4fab0 R08: ffff8892ff3db8c0 R09: 0000000000000000 [ 1061.145187] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ad4fa78 [ 1061.153634] R13: ffffffff930ec9b1 R14: ffff88528ad4fa58 R15: ffff88528ad4fa88 [ 1061.162081] FS: 0000000000000000(0000) GS:ffff8892ff3c0000(0000) knlGS:0000000000000000 [ 1061.171594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1061.178490] CR2: 00007ffff6dab5b0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1061.186937] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1061.195383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1061.203830] Call Trace: [ 1061.207046] [] queued_spin_lock_slowpath+0xb/0xf [ 1061.214526] [] _raw_spin_lock+0x30/0x40 [ 1061.221144] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1061.228825] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1061.237275] [] ? wake_up_atomic_t+0x30/0x30 [ 1061.244262] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1061.251688] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1061.259958] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1061.267939] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1061.276596] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1061.284476] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1061.291789] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1061.299966] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1061.308897] [] ? __getnstimeofday64+0x3f/0xd0 [ 1061.316101] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1061.325147] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1061.333206] [] ? __wake_up_common_lock+0x91/0xc0 [ 1061.340683] [] ? sched_feat_set+0xf0/0xf0 [ 1061.347499] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1061.354977] [] ? __switch_to+0xce/0x5a0 [ 1061.361599] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1061.370338] [] kthread+0xd1/0xe0 [ 1061.376262] [] ? insert_kthread_work+0x40/0x40 [ 1061.383547] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1061.391316] [] ? insert_kthread_work+0x40/0x40 [ 1061.398599] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1066.526001] INFO: rcu_sched self-detected stall on CPU [ 1066.528029] INFO: rcu_sched detected stalls on CPUs/tasks: [ 1066.528030] { [ 1066.528031] 15 [ 1066.528031] } [ 1066.528033] (detected by 19, t=60002 jiffies, g=25117, c=25116, q=9161) [ 1066.528034] Task dump for CPU 15: [ 1066.528035] mdt01_007 R [ 1066.528036] running task [ 1066.528037] 0 3041 2 0x00000088 [ 1066.528038] Call Trace: [ 1066.528046] [] ? native_queued_spin_lock_slowpath+0x158/0x200 [ 1066.528052] [] ? queued_spin_lock_slowpath+0xb/0xf [ 1066.528057] [] ? _raw_spin_lock+0x30/0x40 [ 1066.528076] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1066.528097] [] ? lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1066.528100] [] ? wake_up_atomic_t+0x30/0x30 [ 1066.528108] [] ? LNetPrimaryNID+0xd5/0x220 [lnet] [ 1066.528173] [] ? ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1066.528202] [] ? ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1066.528233] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1066.528263] [] ? ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1066.528292] [] ? ptlrpc_error+0x10/0x20 [ptlrpc] [ 1066.528334] [] ? tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1066.528367] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1066.528370] [] ? __getnstimeofday64+0x3f/0xd0 [ 1066.528401] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1066.528431] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1066.528433] [] ? __wake_up_common_lock+0x91/0xc0 [ 1066.528434] [] ? sched_feat_set+0xf0/0xf0 [ 1066.528465] [] ? ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1066.528468] [] ? __switch_to+0xce/0x5a0 [ 1066.528499] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1066.528502] [] ? kthread+0xd1/0xe0 [ 1066.528503] [] ? insert_kthread_work+0x40/0x40 [ 1066.528506] [] ? ret_from_fork_nospec_begin+0x21/0x21 [ 1066.528507] [] ? insert_kthread_work+0x40/0x40 [ 1066.763534] { [ 1066.765881] 15} (t=60009 jiffies g=25117 c=25116 q=9211) [ 1066.770567] Task dump for CPU 15: [ 1066.774748] mdt01_007 R running task 0 3041 2 0x00000088 [ 1066.783118] Call Trace: [ 1066.786329] [] sched_show_task+0xbf/0x120 [ 1066.793822] [] dump_cpu_task+0x39/0x70 [ 1066.800332] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 1066.807424] [] rcu_check_callbacks+0x482/0x770 [ 1066.814710] [] update_process_times+0x46/0x80 [ 1066.821899] [] tick_sched_handle+0x30/0x70 [ 1066.828794] [] tick_sched_timer+0x39/0x80 [ 1066.835592] [] __hrtimer_run_queues+0x13e/0x2f0 [ 1066.842971] [] ? tick_sched_do_timer+0x50/0x50 [ 1066.850255] [] hrtimer_interrupt+0xb9/0x1f0 [ 1066.857249] [] local_apic_timer_interrupt+0x3b/0x60 [ 1066.865020] [] smp_apic_timer_interrupt+0x43/0x60 [ 1066.872595] [] apic_timer_interrupt+0x16a/0x170 [ 1066.879974] [] ? native_queued_spin_lock_slowpath+0x158/0x200 [ 1066.889402] [] queued_spin_lock_slowpath+0xb/0xf [ 1066.896881] [] _raw_spin_lock+0x30/0x40 [ 1066.903495] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1066.911174] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1066.919621] [] ? wake_up_atomic_t+0x30/0x30 [ 1066.926618] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1066.934015] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1066.942287] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1066.950266] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1066.958923] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1066.966804] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1066.974113] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1066.982291] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1066.991223] [] ? __getnstimeofday64+0x3f/0xd0 [ 1066.998428] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1067.007475] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1067.015534] [] ? __wake_up_common_lock+0x91/0xc0 [ 1067.023011] [] ? sched_feat_set+0xf0/0xf0 [ 1067.029828] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1067.037306] [] ? __switch_to+0xce/0x5a0 [ 1067.043928] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1067.052667] [] kthread+0xd1/0xe0 [ 1067.058593] [] ? insert_kthread_work+0x40/0x40 [ 1067.065877] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1067.073644] [] ? insert_kthread_work+0x40/0x40 [ 1080.897332] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1080.902330] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [mdt01_000:2847] [ 1080.902368] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1080.902396] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1080.902399] CPU: 12 PID: 2847 Comm: mdt01_000 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1080.902400] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1080.902401] task: ffff885290ac9080 ti: ffff885296b70000 task.ti: ffff885296b70000 [ 1080.902407] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1080.902408] RSP: 0018:ffff885296b73ab0 EFLAGS: 00000246 [ 1080.902409] RAX: 0000000000000000 RBX: ffff8892ff300000 RCX: 0000000000610000 [ 1080.902409] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1080.902410] RBP: ffff885296b73ab0 R08: ffff8892ff31b8c0 R09: 0000000000000000 [ 1080.902411] R10: 0000000000000001 R11: 7fffffffffffffff R12: ffff885296b73a78 [ 1080.902411] R13: ffffffff930ec9b1 R14: ffff885296b73a58 R15: ffff885296b73a88 [ 1080.902413] FS: 0000000000000000(0000) GS:ffff8892ff300000(0000) knlGS:0000000000000000 [ 1080.902413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1080.902414] CR2: 00007fffeef71224 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1080.902415] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1080.902416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1080.902416] Call Trace: [ 1080.902421] [] queued_spin_lock_slowpath+0xb/0xf [ 1080.902425] [] _raw_spin_lock+0x30/0x40 [ 1080.902439] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1080.902450] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1080.902454] [] ? wake_up_atomic_t+0x30/0x30 [ 1080.902460] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1080.902504] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1080.902523] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1080.902542] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1080.902560] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1080.902577] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1080.902607] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1080.902628] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1080.902630] [] ? __getnstimeofday64+0x3f/0xd0 [ 1080.902649] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1080.902667] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1080.902670] [] ? __wake_up_common_lock+0x91/0xc0 [ 1080.902671] [] ? sched_feat_set+0xf0/0xf0 [ 1080.902689] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1080.902692] [] ? __switch_to+0xce/0x5a0 [ 1080.902710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1080.902712] [] kthread+0xd1/0xe0 [ 1080.902713] [] ? insert_kthread_work+0x40/0x40 [ 1080.902716] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1080.902717] [] ? insert_kthread_work+0x40/0x40 [ 1080.902731] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1081.407420] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1081.487257] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1081.547548] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1081.561812] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1081.574137] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1081.582972] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1081.594535] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1081.600944] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1081.609390] RDX: ffff8892ff3db8c0 RSI: 0000000000790000 RDI: ffff8852f5420480 [ 1081.617836] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1081.626281] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1081.634727] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1081.643173] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1081.652685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1081.659580] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1081.668025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1081.676472] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1081.684919] Call Trace: [ 1081.688132] [] queued_spin_lock_slowpath+0xb/0xf [ 1081.695609] [] _raw_spin_lock+0x30/0x40 [ 1081.702221] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1081.709900] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1081.718347] [] ? wake_up_atomic_t+0x30/0x30 [ 1081.725345] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1081.732754] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1081.741032] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1081.749017] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1081.757682] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1081.765570] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1081.772883] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1081.781066] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1081.789998] [] ? __getnstimeofday64+0x3f/0xd0 [ 1081.797211] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1081.806262] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1081.814322] [] ? __wake_up_common_lock+0x91/0xc0 [ 1081.821799] [] ? sched_feat_set+0xf0/0xf0 [ 1081.828620] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1081.836097] [] ? __switch_to+0xce/0x5a0 [ 1081.842725] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1081.851464] [] kthread+0xd1/0xe0 [ 1081.857390] [] ? insert_kthread_work+0x40/0x40 [ 1081.864673] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1081.872442] [] ? insert_kthread_work+0x40/0x40 [ 1081.879724] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1092.915772] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [mdt01_007:3041] [ 1092.924804] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1093.004713] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1093.065072] CPU: 15 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1093.079337] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1093.091661] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1093.100494] RIP: 0010:[] [] finish_wait+0x19/0x70 [ 1093.109820] RSP: 0018:ffff88528ad4fae8 EFLAGS: 00000202 [ 1093.116230] RAX: ffff8852a6e84200 RBX: 0000000000790000 RCX: 0000000000000001 [ 1093.124675] RDX: ffff88528ad4ffd8 RSI: ffff88528ad4fb30 RDI: ffff88528f991d28 [ 1093.133121] RBP: ffff88528ad4faf8 R08: ffff88528f991d30 R09: ffff8892ff31b8c0 [ 1093.141566] R10: 0000000000000001 R11: 0000000000000005 R12: ffff8892ff3db8c0 [ 1093.150013] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000005 [ 1093.158460] FS: 0000000000000000(0000) GS:ffff8892ff3c0000(0000) knlGS:0000000000000000 [ 1093.167973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1093.174868] CR2: 00007ffff6dab5b0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1093.183313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1093.191758] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1093.200203] Call Trace: [ 1093.203427] [] lnet_discover_peer_locked+0x232/0x450 [lnet] [ 1093.211971] [] ? wake_up_atomic_t+0x30/0x30 [ 1093.218969] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1093.226387] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1093.234666] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1093.242648] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1093.251307] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1093.259188] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1093.266499] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1093.274680] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1093.283611] [] ? __getnstimeofday64+0x3f/0xd0 [ 1093.290816] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1093.299860] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1093.307920] [] ? __wake_up_common_lock+0x91/0xc0 [ 1093.315398] [] ? sched_feat_set+0xf0/0xf0 [ 1093.322213] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1093.329692] [] ? __switch_to+0xce/0x5a0 [ 1093.336320] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1093.345059] [] kthread+0xd1/0xe0 [ 1093.350985] [] ? insert_kthread_work+0x40/0x40 [ 1093.358270] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1093.366039] [] ? insert_kthread_work+0x40/0x40 [ 1093.373322] Code: d5 6e 00 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 65 48 8b 04 25 c0 0e 01 00 48 89 e5 41 56 41 55 49 89 fd <41> 54 4c 8d 66 18 53 48 c7 00 00 00 00 00 48 89 f3 4c 3b 66 18 [ 1097.764619] LNet: 30707:0:(o2iblnd_cb.c:3432:kiblnd_check_conns()) Timed out tx for 172.19.3.98@o2ib600: 8 seconds [ 1108.896030] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1108.901029] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [mdt01_000:2847] [ 1108.901064] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1108.901087] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1108.901089] CPU: 12 PID: 2847 Comm: mdt01_000 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1108.901090] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1108.901092] task: ffff885290ac9080 ti: ffff885296b70000 task.ti: ffff885296b70000 [ 1108.901098] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1108.901099] RSP: 0018:ffff885296b73ab0 EFLAGS: 00000246 [ 1108.901099] RAX: 0000000000000000 RBX: ffff8892ff300000 RCX: 0000000000610000 [ 1108.901100] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1108.901101] RBP: ffff885296b73ab0 R08: ffff8892ff31b8c0 R09: 0000000000000000 [ 1108.901101] R10: 0000000000000001 R11: 7fffffffffffffff R12: ffff885296b73a78 [ 1108.901102] R13: ffffffff930ec9b1 R14: ffff885296b73a58 R15: ffff885296b73a88 [ 1108.901103] FS: 0000000000000000(0000) GS:ffff8892ff300000(0000) knlGS:0000000000000000 [ 1108.901104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1108.901105] CR2: 00007fffeef71224 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1108.901106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1108.901106] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1108.901107] Call Trace: [ 1108.901111] [] queued_spin_lock_slowpath+0xb/0xf [ 1108.901115] [] _raw_spin_lock+0x30/0x40 [ 1108.901129] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1108.901142] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1108.901144] [] ? wake_up_atomic_t+0x30/0x30 [ 1108.901150] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1108.901194] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1108.901219] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1108.901244] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1108.901268] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1108.901292] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1108.901328] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1108.901354] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1108.901356] [] ? __getnstimeofday64+0x3f/0xd0 [ 1108.901381] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1108.901407] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1108.901409] [] ? __wake_up_common_lock+0x91/0xc0 [ 1108.901410] [] ? sched_feat_set+0xf0/0xf0 [ 1108.901436] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1108.901438] [] ? __switch_to+0xce/0x5a0 [ 1108.901464] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1108.901466] [] kthread+0xd1/0xe0 [ 1108.901467] [] ? insert_kthread_work+0x40/0x40 [ 1108.901469] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1108.901471] [] ? insert_kthread_work+0x40/0x40 [ 1108.901485] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1109.406104] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1109.485933] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1109.546219] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1109.560481] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1109.572806] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1109.581640] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1109.593200] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1109.599612] RAX: 0000000000000001 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1109.608059] RDX: ffff8852ffd9b8c0 RSI: 0000000000b10001 RDI: ffff8852f5420480 [ 1109.616505] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1109.624952] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1109.633398] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1109.641844] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1109.651356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1109.658251] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1109.666698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1109.675144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1109.683589] Call Trace: [ 1109.686802] [] queued_spin_lock_slowpath+0xb/0xf [ 1109.694280] [] _raw_spin_lock+0x30/0x40 [ 1109.700892] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1109.708572] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1109.717018] [] ? wake_up_atomic_t+0x30/0x30 [ 1109.724018] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1109.731424] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1109.739701] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1109.747689] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1109.756352] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1109.764241] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1109.771555] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1109.779737] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1109.788671] [] ? __getnstimeofday64+0x3f/0xd0 [ 1109.795880] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1109.804934] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1109.812994] [] ? __wake_up_common_lock+0x91/0xc0 [ 1109.820470] [] ? sched_feat_set+0xf0/0xf0 [ 1109.827293] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1109.834771] [] ? __switch_to+0xce/0x5a0 [ 1109.841399] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1109.850137] [] kthread+0xd1/0xe0 [ 1109.856063] [] ? insert_kthread_work+0x40/0x40 [ 1109.863346] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1109.871113] [] ? insert_kthread_work+0x40/0x40 [ 1109.878397] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1120.914472] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [mdt01_007:3041] [ 1120.923503] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1121.003400] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1121.063751] CPU: 15 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1121.078016] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1121.090339] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1121.099172] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x15/0x30 [ 1121.110054] RSP: 0018:ffff88528ad4fac0 EFLAGS: 00000246 [ 1121.116465] RAX: 0000000000000246 RBX: 0000000000000001 RCX: ffff8852a6e84200 [ 1121.124910] RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246 [ 1121.133357] RBP: ffff88528ad4fac0 R08: ffff88528d0ffb48 R09: ffff8892ff2db8c0 [ 1121.141802] R10: 0000000000000001 R11: 0000000000000005 R12: ffff8892ff3c0000 [ 1121.150247] R13: ffff88528ad4fab0 R14: ffff88528ad4fa78 R15: ffffffff930ec9b1 [ 1121.158692] FS: 0000000000000000(0000) GS:ffff8892ff3c0000(0000) knlGS:0000000000000000 [ 1121.168206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1121.175099] CR2: 00007ffff6dab5b0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1121.183547] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1121.191994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1121.200443] Call Trace: [ 1121.203660] [] prepare_to_wait+0x56/0x90 [ 1121.210372] [] lnet_discover_peer_locked+0x209/0x450 [lnet] [ 1121.218917] [] ? wake_up_atomic_t+0x30/0x30 [ 1121.225913] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1121.233325] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1121.241595] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1121.249575] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1121.258234] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1121.266116] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1121.273429] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1121.281602] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1121.290535] [] ? __getnstimeofday64+0x3f/0xd0 [ 1121.297745] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1121.306794] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1121.314853] [] ? __wake_up_common_lock+0x91/0xc0 [ 1121.322333] [] ? sched_feat_set+0xf0/0xf0 [ 1121.329150] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1121.336629] [] ? __switch_to+0xce/0x5a0 [ 1121.343251] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1121.351990] [] kthread+0xd1/0xe0 [ 1121.357916] [] ? insert_kthread_work+0x40/0x40 [ 1121.365201] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1121.372968] [] ? insert_kthread_work+0x40/0x40 [ 1121.380250] Code: a8 44 c0 ff ff 01 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 a8 44 c0 ff ff 01 [ 1136.894731] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1136.899729] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [mdt01_000:2847] [ 1136.899769] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1136.899796] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1136.899799] CPU: 12 PID: 2847 Comm: mdt01_000 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1136.899799] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1136.899801] task: ffff885290ac9080 ti: ffff885296b70000 task.ti: ffff885296b70000 [ 1136.899808] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1136.899808] RSP: 0018:ffff885296b73ad8 EFLAGS: 00000246 [ 1136.899809] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff885296b73fd8 [ 1136.899810] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8892fdfb3340 [ 1136.899810] RBP: ffff885296b73af8 R08: ffff885296b73b48 R09: 0000000000000000 [ 1136.899811] R10: 0000000000000001 R11: 7fffffffffffffff R12: 0000000000000001 [ 1136.899811] R13: ffff885296b73ad0 R14: ffff8892ff300000 R15: ffff885296b73ab0 [ 1136.899813] FS: 0000000000000000(0000) GS:ffff8892ff300000(0000) knlGS:0000000000000000 [ 1136.899813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1136.899814] CR2: 00007fffeef71224 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1136.899815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1136.899815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1136.899816] Call Trace: [ 1136.899832] [] ? cfs_percpt_lock+0x58/0x110 [libcfs] [ 1136.899836] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1136.899850] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1136.899853] [] ? wake_up_atomic_t+0x30/0x30 [ 1136.899859] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1136.899907] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1136.899925] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1136.899945] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1136.899962] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1136.899979] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1136.900012] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1136.900033] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1136.900035] [] ? __getnstimeofday64+0x3f/0xd0 [ 1136.900060] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1136.900085] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1136.900088] [] ? __wake_up_common_lock+0x91/0xc0 [ 1136.900089] [] ? sched_feat_set+0xf0/0xf0 [ 1136.900115] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1136.900118] [] ? __switch_to+0xce/0x5a0 [ 1136.900143] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1136.900147] [] kthread+0xd1/0xe0 [ 1136.900149] [] ? insert_kthread_work+0x40/0x40 [ 1136.900151] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1136.900152] [] ? insert_kthread_work+0x40/0x40 [ 1136.900166] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1137.397037] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1137.476896] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1137.537220] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1137.551483] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1137.563809] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1137.572642] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1137.582264] RSP: 0018:ffff88528ac83ad8 EFLAGS: 00000297 [ 1137.588674] RAX: 0000000000610001 RBX: ffff8892ff2db8c0 RCX: 0000000000000000 [ 1137.597123] RDX: 0000000000000001 RSI: 00000000ffffffff RDI: ffff8852f5420480 [ 1137.605569] RBP: ffff88528ac83af8 R08: ffff88528ac83b48 R09: ffff8892ff3db8c0 [ 1137.614017] R10: 0000000000000001 R11: 0000000000000005 R12: 0000000000000001 [ 1137.622465] R13: 0000000000000005 R14: ffff8892ff2c0000 R15: ffff88528ac83ab0 [ 1137.630913] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1137.640427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1137.647323] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1137.655772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1137.664217] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1137.672662] Call Trace: [ 1137.675882] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1137.683755] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1137.692206] [] ? wake_up_atomic_t+0x30/0x30 [ 1137.699194] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1137.706597] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1137.714869] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1137.722850] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1137.731508] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1137.739391] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1137.746706] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1137.754885] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1137.763817] [] ? __getnstimeofday64+0x3f/0xd0 [ 1137.771023] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1137.780070] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1137.788131] [] ? __wake_up_common_lock+0x91/0xc0 [ 1137.795608] [] ? sched_feat_set+0xf0/0xf0 [ 1137.802424] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1137.809902] [] ? __switch_to+0xce/0x5a0 [ 1137.816526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1137.825266] [] kthread+0xd1/0xe0 [ 1137.831195] [] ? insert_kthread_work+0x40/0x40 [ 1137.838477] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1137.846246] [] ? insert_kthread_work+0x40/0x40 [ 1137.853527] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1139.299725] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.136.49@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.320009] LustreError: Skipped 6809 previous similar messages [ 1144.245425] LNet: Service thread pid 3041 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1144.264747] Pid: 3041, comm: mdt01_007 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 1144.276305] Call Trace: [ 1144.279522] [] 0xffffffffffffffff [ 1144.285585] LustreError: dumping log to /tmp/lustre-log.1644346358.3041 [ 1148.913174] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [mdt01_007:3041] [ 1148.922206] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1149.002132] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1149.062484] CPU: 15 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1149.076748] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1149.089073] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1149.097907] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1149.107533] RSP: 0018:ffff88528ad4fae0 EFLAGS: 00000246 [ 1149.113943] RAX: 0000000000000000 RBX: ffff88528d0ffb48 RCX: ffff88528ad4ffd8 [ 1149.122388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88528f991ccc [ 1149.130833] RBP: ffff88528ad4faf8 R08: ffff88528ad4fb48 R09: ffff8892ff2db8c0 [ 1149.139280] R10: 0000000000000001 R11: 0000000000000005 R12: 0000000000000001 [ 1149.147726] R13: 0000000000000005 R14: 0000000000000001 R15: ffff88528ad4fac0 [ 1149.156171] FS: 0000000000000000(0000) GS:ffff8892ff3c0000(0000) knlGS:0000000000000000 [ 1149.165685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1149.172578] CR2: 00007ffff6dab5b0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1149.181026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1149.189473] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1149.197918] Call Trace: [ 1149.201143] [] ? lnet_peer_is_uptodate+0x1b/0xa0 [lnet] [ 1149.209305] [] lnet_discover_peer_locked+0x2b3/0x450 [lnet] [ 1149.217853] [] ? wake_up_atomic_t+0x30/0x30 [ 1149.224840] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1149.232266] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1149.240536] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1149.248518] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1149.257176] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1149.265057] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1149.272370] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1149.280538] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1149.289472] [] ? __getnstimeofday64+0x3f/0xd0 [ 1149.296681] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1149.305734] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1149.313795] [] ? __wake_up_common_lock+0x91/0xc0 [ 1149.321272] [] ? sched_feat_set+0xf0/0xf0 [ 1149.328095] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1149.335573] [] ? __switch_to+0xce/0x5a0 [ 1149.342202] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1149.350940] [] kthread+0xd1/0xe0 [ 1149.356866] [] ? insert_kthread_work+0x40/0x40 [ 1149.364155] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1149.371923] [] ? insert_kthread_work+0x40/0x40 [ 1149.379205] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1155.898097] Lustre: 2847:0:(service.c:2169:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:2s); client may timeout. req@ffff8892fab90480 x1723872075763456/t0(0) o38->@:0/0 lens 520/192 e 0 to 0 dl 1644346368 ref 1 fl Complete:/0/0 rc -11/-11 [ 1155.898376] LNet: Service thread pid 3041 completed after 212.30s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [ 1155.944292] Lustre: 2847:0:(service.c:2169:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 1164.893434] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1164.902464] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1164.982325] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1165.042639] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1165.056903] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1165.069228] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1165.078063] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1d0/0x200 [ 1165.089619] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000202 [ 1165.096028] RAX: 0000000000000101 RBX: ffff8892ff2c0000 RCX: 0000000000000001 [ 1165.104474] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8852f5420480 [ 1165.112920] RBP: ffff88528ac83ab0 R08: 0000000000000101 R09: 0000000000000000 [ 1165.121366] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1165.129813] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1165.138260] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1165.147773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1165.154667] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1165.163113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1165.171560] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1165.180004] Call Trace: [ 1165.183219] [] queued_spin_lock_slowpath+0xb/0xf [ 1165.190697] [] _raw_spin_lock+0x30/0x40 [ 1165.197314] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1165.204988] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1165.213439] [] ? wake_up_atomic_t+0x30/0x30 [ 1165.220436] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1165.227858] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1165.236134] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1165.244116] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1165.252775] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1165.260658] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1165.267971] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1165.276140] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1165.285073] [] ? __getnstimeofday64+0x3f/0xd0 [ 1165.292280] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1165.301327] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1165.309387] [] ? __wake_up_common_lock+0x91/0xc0 [ 1165.316866] [] ? sched_feat_set+0xf0/0xf0 [ 1165.323683] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1165.331163] [] ? __switch_to+0xce/0x5a0 [ 1165.337787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1165.346526] [] kthread+0xd1/0xe0 [ 1165.352452] [] ? insert_kthread_work+0x40/0x40 [ 1165.359738] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1165.367505] [] ? insert_kthread_work+0x40/0x40 [ 1165.374788] Code: fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 8b 07 <0f> b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 0f 1f [ 1180.948688] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1180.957719] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1181.037583] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1181.097879] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1181.112144] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1181.124468] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1181.133303] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1c8/0x200 [ 1181.144866] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000293 [ 1181.151276] RAX: 0000000000000001 RBX: ffff8892ff540000 RCX: 0000000000000001 [ 1181.159721] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8852f5420480 [ 1181.168167] RBP: ffff8892a669bab0 R08: 0000000000000101 R09: 0000000000000000 [ 1181.176612] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1181.185058] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1181.193504] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1181.203016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1181.209910] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1181.218356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1181.226803] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1181.235250] Call Trace: [ 1181.238466] [] queued_spin_lock_slowpath+0xb/0xf [ 1181.245936] [] _raw_spin_lock+0x30/0x40 [ 1181.252554] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1181.260235] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1181.268683] [] ? wake_up_atomic_t+0x30/0x30 [ 1181.275680] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1181.283100] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1181.291371] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1181.299353] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1181.308010] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1181.315892] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1181.323206] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1181.331373] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1181.340307] [] ? __getnstimeofday64+0x3f/0xd0 [ 1181.347513] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1181.356558] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1181.364618] [] ? __wake_up_common_lock+0x91/0xc0 [ 1181.372096] [] ? sched_feat_set+0xf0/0xf0 [ 1181.378913] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1181.386394] [] ? __switch_to+0xce/0x5a0 [ 1181.393016] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1181.401756] [] kthread+0xd1/0xe0 [ 1181.407683] [] ? insert_kthread_work+0x40/0x40 [ 1181.414969] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1181.422736] [] ? insert_kthread_work+0x40/0x40 [ 1181.430019] Code: c3 66 90 f3 90 8b 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 <85> c0 74 0c f3 90 8b 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 [ 1192.892137] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1192.901186] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1192.981036] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1193.041321] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1193.055585] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1193.067910] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1193.076745] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1193.086368] RSP: 0018:ffff88528ac83ad8 EFLAGS: 00000246 [ 1193.092777] RAX: 0000000000000000 RBX: 0000000000000101 RCX: ffff88528ac83fd8 [ 1193.101223] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8892fdfb3340 [ 1193.109668] RBP: ffff88528ac83af8 R08: ffff88528ac83b48 R09: 0000000000000000 [ 1193.118114] R10: 0000000000000001 R11: 0000000000000005 R12: 0000000000000001 [ 1193.126560] R13: 0000000000000005 R14: ffff8892ff2c0000 R15: ffff88528ac83ab0 [ 1193.135006] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1193.144517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1193.151412] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1193.159858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1193.168304] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1193.176749] Call Trace: [ 1193.179970] [] ? cfs_percpt_lock+0x58/0x110 [libcfs] [ 1193.187839] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1193.195811] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1193.204358] [] ? wake_up_atomic_t+0x30/0x30 [ 1193.211354] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1193.218774] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1193.227045] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1193.235027] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1193.243684] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1193.251566] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1193.258877] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1193.267053] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1193.275986] [] ? __getnstimeofday64+0x3f/0xd0 [ 1193.283190] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1193.292235] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1193.300295] [] ? __wake_up_common_lock+0x91/0xc0 [ 1193.307771] [] ? sched_feat_set+0xf0/0xf0 [ 1193.314588] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1193.322068] [] ? __switch_to+0xce/0x5a0 [ 1193.328694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1193.337424] [] kthread+0xd1/0xe0 [ 1193.343352] [] ? insert_kthread_work+0x40/0x40 [ 1193.350637] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1193.358405] [] ? insert_kthread_work+0x40/0x40 [ 1193.365686] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1199.759861] LNet: 30707:0:(o2iblnd_cb.c:3432:kiblnd_check_conns()) Timed out tx for 172.19.3.101@o2ib600: 10 seconds [ 1201.976842] INFO: task mount.lustre:2109 blocked for more than 120 seconds. [ 1201.985102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1201.994336] mount.lustre D ffff8852f4c51660 0 2109 2028 0x00000080 [ 1202.002724] Call Trace: [ 1202.005949] [] schedule+0x29/0x70 [ 1202.011984] [] schedule_timeout+0x271/0x320 [ 1202.018988] [] ? select_task_rq_fair+0x690/0x780 [ 1202.026474] [] wait_for_completion+0x12d/0x170 [ 1202.033772] [] ? wake_up_state+0x20/0x20 [ 1202.040518] [] llog_process_or_fork+0x244/0x450 [obdclass] [ 1202.048983] [] llog_process+0x14/0x20 [obdclass] [ 1202.056485] [] class_config_parse_llog+0x125/0x350 [obdclass] [ 1202.065236] [] mgc_process_cfg_log+0x7c0/0xc70 [mgc] [ 1202.073114] [] mgc_process_log+0x3ec/0x920 [mgc] [ 1202.080591] [] ? config_recover_log_add+0x13f/0x280 [mgc] [ 1202.088960] [] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [ 1202.098095] [] mgc_process_config+0x8bb/0x1420 [mgc] [ 1202.105981] [] lustre_process_log+0x2d8/0xad0 [obdclass] [ 1202.114252] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 1202.122135] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 1202.130606] [] server_start_targets+0x13a4/0x2a20 [obdclass] [ 1202.139266] [] ? lustre_start_mgc+0x260/0x2510 [obdclass] [ 1202.147635] [] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [ 1202.156785] [] server_fill_super+0x10cc/0x1890 [obdclass] [ 1202.165146] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 1202.173031] [] lustre_fill_super+0x468/0x960 [obdclass] [ 1202.181205] [] ? lustre_common_put_super+0x270/0x270 [obdclass] [ 1202.190147] [] mount_nodev+0x4f/0xb0 [ 1202.196480] [] lustre_mount+0x38/0x60 [obdclass] [ 1202.203964] [] mount_fs+0x3e/0x1b0 [ 1202.210095] [] vfs_kern_mount+0x67/0x120 [ 1202.216808] [] do_mount+0x1ef/0xd50 [ 1202.223036] [] ? kmem_cache_alloc_trace+0x4f/0x250 [ 1202.230717] [] SyS_mount+0x83/0xd0 [ 1202.236849] [] system_call_fastpath+0x25/0x2a [ 1208.947390] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1208.956418] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1209.036262] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1209.096560] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1209.110824] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1209.123148] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1209.131982] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1209.143545] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1209.149954] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1209.158400] RDX: ffff8852ffb9b8c0 RSI: 0000000000310000 RDI: ffff8852f5420480 [ 1209.166847] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1209.175294] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1209.183740] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1209.192188] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1209.201702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1209.208597] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1209.217042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1209.225488] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1209.233933] Call Trace: [ 1209.237147] [] queued_spin_lock_slowpath+0xb/0xf [ 1209.244625] [] _raw_spin_lock+0x30/0x40 [ 1209.251239] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1209.258920] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1209.267367] [] ? wake_up_atomic_t+0x30/0x30 [ 1209.274365] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1209.281784] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1209.290057] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1209.298037] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1209.306694] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1209.314576] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1209.321889] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1209.330065] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1209.338999] [] ? __getnstimeofday64+0x3f/0xd0 [ 1209.346205] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1209.355253] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1209.363314] [] ? __wake_up_common_lock+0x91/0xc0 [ 1209.370792] [] ? sched_feat_set+0xf0/0xf0 [ 1209.377609] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1209.385089] [] ? __switch_to+0xce/0x5a0 [ 1209.391712] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1209.400450] [] kthread+0xd1/0xe0 [ 1209.406376] [] ? insert_kthread_work+0x40/0x40 [ 1209.413661] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1209.421431] [] ? insert_kthread_work+0x40/0x40 [ 1209.428712] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1215.898069] INFO: rcu_sched self-detected stall on CPU [ 1215.898074] INFO: rcu_sched self-detected stall on CPU { 11} (t=60001 jiffies g=25118 c=25117 q=34329) [ 1215.898075] Task dump for CPU 11: [ 1215.898076] mdt01_006 R running task 0 3034 2 0x00000088 [ 1215.898077] Call Trace: [ 1215.898083] [] sched_show_task+0xbf/0x120 [ 1215.898086] [] dump_cpu_task+0x39/0x70 [ 1215.898090] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 1215.898093] [] rcu_check_callbacks+0x482/0x770 [ 1215.898097] [] update_process_times+0x46/0x80 [ 1215.898101] [] tick_sched_handle+0x30/0x70 [ 1215.898103] [] tick_sched_timer+0x39/0x80 [ 1215.898105] [] __hrtimer_run_queues+0x13e/0x2f0 [ 1215.898107] [] ? tick_sched_do_timer+0x50/0x50 [ 1215.898108] [] hrtimer_interrupt+0xb9/0x1f0 [ 1215.898110] [] local_apic_timer_interrupt+0x3b/0x60 [ 1215.898114] [] smp_apic_timer_interrupt+0x43/0x60 [ 1215.898116] [] apic_timer_interrupt+0x16a/0x170 [ 1215.898126] [] ? cfs_percpt_lock+0xcd/0x110 [libcfs] [ 1215.898137] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1215.898139] [] ? wake_up_atomic_t+0x30/0x30 [ 1215.898144] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1215.898179] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1215.898197] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1215.898216] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1215.898234] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1215.898250] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1215.898278] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1215.898299] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1215.898300] [] ? __getnstimeofday64+0x3f/0xd0 [ 1215.898319] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1215.898338] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1215.898340] [] ? __wake_up_common_lock+0x91/0xc0 [ 1215.898341] [] ? sched_feat_set+0xf0/0xf0 [ 1215.898359] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1215.898361] [] ? __switch_to+0xce/0x5a0 [ 1215.898379] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1215.898381] [] kthread+0xd1/0xe0 [ 1215.898382] [] ? insert_kthread_work+0x40/0x40 [ 1215.898384] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1215.898385] [] ? insert_kthread_work+0x40/0x40 [ 1215.898386] Task dump for CPU 29: [ 1215.898387] mdt01_013 R running task 0 4292 2 0x00000088 [ 1215.898388] Call Trace: [ 1215.898389] [] ? _raw_spin_lock+0x20/0x40 [ 1215.898396] [] ? lnet_peer_is_uptodate+0x1b/0xa0 [lnet] [ 1215.898402] [] ? lnet_discover_peer_locked+0x2b3/0x450 [lnet] [ 1215.898404] [] ? wake_up_atomic_t+0x30/0x30 [ 1215.898409] [] ? LNetPrimaryNID+0xd5/0x220 [lnet] [ 1215.898426] [] ? ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1215.898444] [] ? ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1215.898462] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1215.898479] [] ? ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1215.898495] [] ? ptlrpc_error+0x10/0x20 [ptlrpc] [ 1215.898518] [] ? tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1215.898546] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1215.898548] [] ? __getnstimeofday64+0x3f/0xd0 [ 1215.898574] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1215.898599] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1215.898600] [] ? __wake_up_common_lock+0x91/0xc0 [ 1215.898602] [] ? sched_feat_set+0xf0/0xf0 [ 1215.898627] [] ? ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1215.898629] [] ? __switch_to+0xce/0x5a0 [ 1215.898654] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1215.898656] [] ? kthread+0xd1/0xe0 [ 1215.898658] [] ? insert_kthread_work+0x40/0x40 [ 1215.898660] [] ? ret_from_fork_nospec_begin+0x21/0x21 [ 1215.898661] [] ? insert_kthread_work+0x40/0x40 [ 1216.405066] { 29} (t=60510 jiffies g=25118 c=25117 q=34333) [ 1216.411992] Task dump for CPU 11: [ 1216.416173] mdt01_006 R running task 0 3034 2 0x00000088 [ 1216.424544] Call Trace: [ 1216.427757] [] ? native_queued_spin_lock_slowpath+0x158/0x200 [ 1216.436496] [] ? queued_spin_lock_slowpath+0xb/0xf [ 1216.444167] [] ? _raw_spin_lock+0x30/0x40 [ 1216.450969] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1216.458839] [] ? lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1216.467480] [] ? wake_up_atomic_t+0x30/0x30 [ 1216.474477] [] ? LNetPrimaryNID+0xd5/0x220 [lnet] [ 1216.482068] [] ? ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1216.490532] [] ? ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1216.498704] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1216.507363] [] ? ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1216.515438] [] ? ptlrpc_error+0x10/0x20 [ptlrpc] [ 1216.522939] [] ? tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1216.531310] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1216.540232] [] ? __getnstimeofday64+0x3f/0xd0 [ 1216.547436] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1216.556675] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1216.564733] [] ? __wake_up_common_lock+0x91/0xc0 [ 1216.572211] [] ? sched_feat_set+0xf0/0xf0 [ 1216.579027] [] ? ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1216.586699] [] ? __switch_to+0xce/0x5a0 [ 1216.593320] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1216.602059] [] ? kthread+0xd1/0xe0 [ 1216.608180] [] ? insert_kthread_work+0x40/0x40 [ 1216.615464] [] ? ret_from_fork_nospec_begin+0x21/0x21 [ 1216.623427] [] ? insert_kthread_work+0x40/0x40 [ 1216.630709] Task dump for CPU 29: [ 1216.634888] mdt01_013 R running task 0 4292 2 0x00000088 [ 1216.643258] Call Trace: [ 1216.646469] [] sched_show_task+0xbf/0x120 [ 1216.653958] [] dump_cpu_task+0x39/0x70 [ 1216.660467] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 1216.667556] [] rcu_check_callbacks+0x482/0x770 [ 1216.674840] [] update_process_times+0x46/0x80 [ 1216.682027] [] tick_sched_handle+0x30/0x70 [ 1216.688923] [] tick_sched_timer+0x39/0x80 [ 1216.695721] [] __hrtimer_run_queues+0x13e/0x2f0 [ 1216.703101] [] ? tick_sched_do_timer+0x50/0x50 [ 1216.710385] [] hrtimer_interrupt+0xb9/0x1f0 [ 1216.717378] [] local_apic_timer_interrupt+0x3b/0x60 [ 1216.725145] [] smp_apic_timer_interrupt+0x43/0x60 [ 1216.732719] [] apic_timer_interrupt+0x16a/0x170 [ 1216.740097] [] ? _raw_spin_lock+0x20/0x40 [ 1216.747592] [] ? lnet_peer_is_uptodate+0x1b/0xa0 [lnet] [ 1216.755752] [] lnet_discover_peer_locked+0x2b3/0x450 [lnet] [ 1216.764297] [] ? wake_up_atomic_t+0x30/0x30 [ 1216.771293] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1216.778688] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1216.786958] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1216.794938] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1216.803595] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1216.811476] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1216.818782] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1216.826959] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1216.835883] [] ? __getnstimeofday64+0x3f/0xd0 [ 1216.843087] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1216.852134] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1216.860193] [] ? __wake_up_common_lock+0x91/0xc0 [ 1216.867671] [] ? sched_feat_set+0xf0/0xf0 [ 1216.874486] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1216.881965] [] ? __switch_to+0xce/0x5a0 [ 1216.888587] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1216.897326] [] kthread+0xd1/0xe0 [ 1216.903251] [] ? insert_kthread_work+0x40/0x40 [ 1216.910535] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1216.918304] [] ? insert_kthread_work+0x40/0x40 [ 1232.796287] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mdt00_009:4257] [ 1232.805220] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1232.938280] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [mdt01_002:2849] [ 1232.885010] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1232.938282] Modules linked in: [ 1232.938283] osp(OE) [ 1232.938284] mdd(OE) [ 1232.938284] lod(OE) [ 1232.938284] mdt(OE) [ 1232.938285] lfsck(OE) [ 1232.938285] mgc(OE) [ 1232.938286] osd_zfs(OE) [ 1232.938286] lquota(OE) [ 1232.938286] fid(OE) [ 1232.938287] fld(OE) [ 1232.938287] ptlrpc(OE) [ 1232.938288] obdclass(OE) [ 1232.938288] ko2iblnd(OE) [ 1232.938288] lnet(OE) [ 1232.938289] libcfs(OE) [ 1232.938289] iTCO_wdt [ 1232.938289] iTCO_vendor_support [ 1232.938290] rpcrdma [ 1232.938290] ib_iser [ 1232.938291] sb_edac [ 1232.938291] intel_powerclamp [ 1232.938292] coretemp [ 1232.938292] intel_rapl [ 1232.938292] iosf_mbi [ 1232.938293] kvm [ 1232.938294] irqbypass [ 1232.938294] pcspkr [ 1232.938294] joydev [ 1232.938295] lpc_ich [ 1232.938295] i2c_i801 [ 1232.938296] ioatdma [ 1232.938296] ses [ 1232.938296] enclosure [ 1232.938297] sg [ 1232.938297] ipmi_si [ 1232.938297] ipmi_devintf [ 1232.938298] ipmi_msghandler [ 1232.938298] dm_round_robin [ 1232.938299] acpi_cpufreq [ 1232.938300] acpi_power_meter [ 1232.938300] ib_ipoib [ 1232.938300] rdma_ucm [ 1232.938301] ib_umad [ 1232.938301] sch_fq_codel [ 1232.938302] iw_cxgb4 [ 1232.938302] rdma_cm [ 1232.938303] iw_cm [ 1232.938303] ib_cm [ 1232.938303] iw_cxgb3 [ 1232.938304] zfs(POE) [ 1232.938304] binfmt_misc [ 1232.938305] zunicode(POE) [ 1232.938305] zavl(POE) [ 1232.938306] icp(POE) [ 1232.938306] zcommon(POE) [ 1232.938306] znvpair(POE) [ 1232.938307] spl(OE) [ 1232.938307] msr_safe(OE) [ 1232.938308] ip_tables [ 1232.938308] nfsv3 [ 1232.938309] nfs_acl [ 1232.938309] rpcsec_gss_krb5 [ 1232.938310] auth_rpcgss [ 1232.938310] nfsv4 [ 1232.938310] dns_resolver [ 1232.938311] nfs [ 1232.938311] lockd [ 1232.938312] grace [ 1232.938312] fscache [ 1232.938313] overlay(T) [ 1232.938313] ext4 [ 1232.938314] mbcache [ 1232.938315] jbd2 [ 1232.938315] dm_service_time [ 1232.938315] sd_mod [ 1232.938316] crc_t10dif [ 1232.938316] crct10dif_generic [ 1232.938317] mlx5_ib [ 1232.938317] ib_uverbs [ 1232.938318] ib_core [ 1232.938318] be2iscsi [ 1232.938318] bnx2i [ 1232.938319] cnic [ 1232.938319] uio [ 1232.938320] cxgb4i [ 1232.938320] cxgb4 [ 1232.938321] cxgb3i [ 1232.938321] cxgb3 [ 1232.938321] mdio [ 1232.938322] libcxgbi [ 1232.938322] libcxgb [ 1232.938323] qla4xxx [ 1232.938324] iscsi_boot_sysfs [ 1232.938324] 8021q [ 1232.938324] garp [ 1232.938325] mrp [ 1232.938325] stp [ 1232.938326] llc [ 1232.938326] mgag200 [ 1232.938327] i2c_algo_bit [ 1232.938327] drm_kms_helper [ 1232.938327] syscopyarea [ 1232.938328] sysfillrect [ 1232.938328] sysimgblt [ 1232.938329] crct10dif_pclmul [ 1232.938329] fb_sys_fops [ 1232.938330] crct10dif_common [ 1232.938330] ttm [ 1232.938331] crc32_pclmul [ 1232.938331] crc32c_intel [ 1232.938332] mlx5_core [ 1232.938332] ahci [ 1232.938332] ghash_clmulni_intel [ 1232.938333] mxm_wmi [ 1232.938333] aesni_intel [ 1232.938334] mlxfw [ 1232.938334] drm [ 1232.938335] ixgbe(OE) [ 1232.938335] mpt3sas [ 1232.938335] lrw [ 1232.938336] devlink [ 1232.938336] libahci [ 1232.938337] gf128mul [ 1232.938338] dca [ 1232.938338] glue_helper [ 1232.938339] ablk_helper [ 1232.938339] ptp [ 1232.938339] raid_class [ 1232.938340] libata [ 1232.938340] drm_panel_orientation_quirks [ 1232.938340] cryptd [ 1232.938341] scsi_transport_sas [ 1232.938341] pps_core [ 1232.938342] dm_multipath [ 1232.938342] wmi [ 1232.938342] sunrpc [ 1232.938343] dm_mirror [ 1232.938343] dm_region_hash [ 1232.938344] dm_log [ 1232.938344] dm_mod [ 1232.938344] iscsi_tcp [ 1232.938345] libiscsi_tcp [ 1232.938345] libiscsi [ 1232.938345] scsi_transport_iscsi [ 1232.938346] fuse [ 1232.938346] [ 1232.938348] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1232.938349] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1232.938350] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1232.938351] RIP: 0010:[] [ 1232.938356] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1232.938356] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1232.938357] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1232.938358] RDX: ffff8892ff25b8c0 RSI: 0000000000490001 RDI: ffff8852f5420480 [ 1232.938358] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1232.938359] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1232.938359] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1232.938360] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1232.938361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1232.938362] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1232.938363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1232.938363] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1232.938364] Call Trace: [ 1232.938368] [] queued_spin_lock_slowpath+0xb/0xf [ 1232.938370] [] _raw_spin_lock+0x30/0x40 [ 1232.938380] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1232.938391] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1232.938393] [] ? wake_up_atomic_t+0x30/0x30 [ 1232.938399] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1232.938439] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1232.938458] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1232.938477] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1232.938494] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1232.938511] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1232.938540] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1232.938561] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1232.938563] [] ? __getnstimeofday64+0x3f/0xd0 [ 1232.938583] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1232.938601] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1232.938603] [] ? __wake_up_common_lock+0x91/0xc0 [ 1232.938604] [] ? sched_feat_set+0xf0/0xf0 [ 1232.938622] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1232.938624] [] ? __switch_to+0xce/0x5a0 [ 1232.938642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1232.938644] [] kthread+0xd1/0xe0 [ 1232.938645] [] ? insert_kthread_work+0x40/0x40 [ 1232.938647] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1232.938649] [] ? insert_kthread_work+0x40/0x40 [ 1232.938649] Code: [ 1232.938650] 0d [ 1232.938650] 48 [ 1232.938650] 98 [ 1232.938650] 83 [ 1232.938651] e2 [ 1232.938651] 30 [ 1232.938651] 48 [ 1232.938651] 81 [ 1232.938652] c2 [ 1232.938652] c0 [ 1232.938652] b8 [ 1232.938652] 01 [ 1232.938653] 00 [ 1232.938653] 48 [ 1232.938653] 03 [ 1232.938653] 14 [ 1232.938654] c5 [ 1232.938654] e0 [ 1232.938654] 17 [ 1232.938654] d5 [ 1232.938655] 93 [ 1232.938655] 4c [ 1232.938655] 89 [ 1232.938656] 02 [ 1232.938656] 41 [ 1232.938656] 8b [ 1232.938656] 40 [ 1232.938656] 08 [ 1232.938657] 85 [ 1232.938657] c0 [ 1232.938657] 75 [ 1232.938658] 0f [ 1232.938658] 0f [ 1232.938658] 1f [ 1232.938658] 44 [ 1232.938659] 00 [ 1232.938659] 00 [ 1232.938659] f3 [ 1232.938659] 90 [ 1232.938660] 41 [ 1232.938660] 8b [ 1232.938660] 40 [ 1232.938660] 08 [ 1232.938661] <85> [ 1232.938661] c0 [ 1232.938661] 74 [ 1232.938661] f6 [ 1232.938662] 4d [ 1232.938662] 8b [ 1232.938662] 08 [ 1232.938662] 4d [ 1232.938663] 85 [ 1232.938663] c9 [ 1232.938663] 74 [ 1232.938663] 04 [ 1232.938664] 41 [ 1232.938664] 0f [ 1232.938664] 18 [ 1232.938664] 09 [ 1232.938665] 8b [ 1232.938665] 17 [ 1232.938665] 0f [ 1232.938665] b7 [ 1232.938665] c2 [ 1232.938666] [ 1233.441656] dm_mod [ 1233.444488] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1233.450365] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1233.464531] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1233.476855] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1233.485689] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [ 1233.497251] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000202 [ 1233.503662] RAX: 0000000000000001 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1233.512106] RDX: 0000000000d10000 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1233.520551] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: ffff8892ff55b8c0 [ 1233.528997] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1233.537443] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1233.545891] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1233.555404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1233.562297] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1233.570742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1233.579187] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1233.587633] Call Trace: [ 1233.590848] [] queued_spin_lock_slowpath+0xb/0xf [ 1233.598327] [] _raw_spin_lock+0x30/0x40 [ 1233.604941] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1233.612616] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1233.620596] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1233.629141] [] ? wake_up_atomic_t+0x30/0x30 [ 1233.636136] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1233.643551] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1233.651821] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1233.659802] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1233.668459] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1233.676341] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1233.683651] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1233.691828] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1233.700752] [] ? __getnstimeofday64+0x3f/0xd0 [ 1233.707958] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1233.717004] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1233.725065] [] ? __wake_up_common_lock+0x91/0xc0 [ 1233.732543] [] ? sched_feat_set+0xf0/0xf0 [ 1233.739359] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1233.746838] [] ? __switch_to+0xce/0x5a0 [ 1233.753459] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1233.762197] [] kthread+0xd1/0xe0 [ 1233.768124] [] ? insert_kthread_work+0x40/0x40 [ 1233.775408] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1233.783176] [] ? insert_kthread_work+0x40/0x40 [ 1233.790458] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [ 1233.812636] sched: RT throttling activated [ 1240.889942] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1240.945939] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1240.898971] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1240.945939] Modules linked in: [ 1240.945940] osp(OE) [ 1240.945941] mdd(OE) [ 1240.945941] lod(OE) [ 1240.945941] mdt(OE) [ 1240.945942] lfsck(OE) [ 1240.945942] mgc(OE) [ 1240.945942] osd_zfs(OE) [ 1240.945943] lquota(OE) [ 1240.945943] fid(OE) [ 1240.945943] fld(OE) [ 1240.945944] ptlrpc(OE) [ 1240.945944] obdclass(OE) [ 1240.945944] ko2iblnd(OE) [ 1240.945945] lnet(OE) [ 1240.945945] libcfs(OE) [ 1240.945945] iTCO_wdt [ 1240.945946] iTCO_vendor_support [ 1240.945946] rpcrdma [ 1240.945946] ib_iser [ 1240.945946] sb_edac [ 1240.945947] intel_powerclamp [ 1240.945947] coretemp [ 1240.945947] intel_rapl [ 1240.945948] iosf_mbi [ 1240.945948] kvm [ 1240.945948] irqbypass [ 1240.945949] pcspkr [ 1240.945949] joydev [ 1240.945949] lpc_ich [ 1240.945950] i2c_i801 [ 1240.945950] ioatdma [ 1240.945950] ses [ 1240.945950] enclosure [ 1240.945951] sg [ 1240.945951] ipmi_si [ 1240.945951] ipmi_devintf [ 1240.945952] ipmi_msghandler [ 1240.945952] dm_round_robin [ 1240.945952] acpi_cpufreq [ 1240.945953] acpi_power_meter [ 1240.945953] ib_ipoib [ 1240.945953] rdma_ucm [ 1240.945953] ib_umad [ 1240.945954] sch_fq_codel [ 1240.945954] iw_cxgb4 [ 1240.945955] rdma_cm [ 1240.945955] iw_cm [ 1240.945955] ib_cm [ 1240.945956] iw_cxgb3 [ 1240.945956] zfs(POE) [ 1240.945957] binfmt_misc [ 1240.945957] zunicode(POE) [ 1240.945958] zavl(POE) [ 1240.945958] icp(POE) [ 1240.945958] zcommon(POE) [ 1240.945959] znvpair(POE) [ 1240.945959] spl(OE) [ 1240.945960] msr_safe(OE) [ 1240.945960] ip_tables [ 1240.945961] nfsv3 [ 1240.945961] nfs_acl [ 1240.945962] rpcsec_gss_krb5 [ 1240.945962] auth_rpcgss [ 1240.945962] nfsv4 [ 1240.945963] dns_resolver [ 1240.945963] nfs [ 1240.945964] lockd [ 1240.945964] grace [ 1240.945964] fscache [ 1240.945965] overlay(T) [ 1240.945965] ext4 [ 1240.945966] mbcache [ 1240.945966] jbd2 [ 1240.945967] dm_service_time [ 1240.945967] sd_mod [ 1240.945968] crc_t10dif [ 1240.945968] crct10dif_generic [ 1240.945968] mlx5_ib [ 1240.945969] ib_uverbs [ 1240.945969] ib_core [ 1240.945970] be2iscsi [ 1240.945970] bnx2i [ 1240.945970] cnic [ 1240.945971] uio [ 1240.945971] cxgb4i [ 1240.945971] cxgb4 [ 1240.945972] cxgb3i [ 1240.945972] cxgb3 [ 1240.945973] mdio [ 1240.945973] libcxgbi [ 1240.945974] libcxgb [ 1240.945974] qla4xxx [ 1240.945974] iscsi_boot_sysfs [ 1240.945975] 8021q [ 1240.945976] garp [ 1240.945976] mrp [ 1240.945976] stp [ 1240.945977] llc [ 1240.945977] mgag200 [ 1240.945977] i2c_algo_bit [ 1240.945978] drm_kms_helper [ 1240.945978] syscopyarea [ 1240.945979] sysfillrect [ 1240.945979] sysimgblt [ 1240.945980] crct10dif_pclmul [ 1240.945980] fb_sys_fops [ 1240.945980] crct10dif_common [ 1240.945981] ttm [ 1240.945981] crc32_pclmul [ 1240.945981] crc32c_intel [ 1240.945982] mlx5_core [ 1240.945982] ahci [ 1240.945983] ghash_clmulni_intel [ 1240.945983] mxm_wmi [ 1240.945983] aesni_intel [ 1240.945984] mlxfw [ 1240.945985] drm [ 1240.945985] ixgbe(OE) [ 1240.945985] mpt3sas [ 1240.945986] lrw [ 1240.945986] devlink [ 1240.945987] libahci [ 1240.945987] gf128mul [ 1240.945987] dca [ 1240.945988] glue_helper [ 1240.945989] ablk_helper [ 1240.945989] ptp [ 1240.945989] raid_class [ 1240.945990] libata [ 1240.945990] drm_panel_orientation_quirks [ 1240.945990] cryptd [ 1240.945991] scsi_transport_sas [ 1240.945991] pps_core [ 1240.945992] dm_multipath [ 1240.945992] wmi [ 1240.945992] sunrpc [ 1240.945993] dm_mirror [ 1240.945993] dm_region_hash [ 1240.945994] dm_log [ 1240.945994] dm_mod [ 1240.945994] iscsi_tcp [ 1240.945995] libiscsi_tcp [ 1240.945995] libiscsi [ 1240.945995] scsi_transport_iscsi [ 1240.945996] fuse [ 1240.945996] [ 1240.945998] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1240.945999] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1240.946000] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1240.946001] RIP: 0010:[] [ 1240.946004] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1240.946005] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1240.946006] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1240.946006] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1240.946007] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1240.946008] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1240.946008] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1240.946009] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1240.946010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1240.946011] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1240.946012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1240.946012] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1240.946012] Call Trace: [ 1240.946016] [] queued_spin_lock_slowpath+0xb/0xf [ 1240.946018] [] _raw_spin_lock+0x30/0x40 [ 1240.946028] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1240.946039] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1240.946042] [] ? wake_up_atomic_t+0x30/0x30 [ 1240.946047] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1240.946083] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1240.946102] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1240.946121] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1240.946138] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1240.946155] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1240.946186] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1240.946213] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1240.946214] [] ? __getnstimeofday64+0x3f/0xd0 [ 1240.946240] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1240.946265] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1240.946268] [] ? __wake_up_common_lock+0x91/0xc0 [ 1240.946269] [] ? sched_feat_set+0xf0/0xf0 [ 1240.946295] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1240.946297] [] ? __switch_to+0xce/0x5a0 [ 1240.946322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1240.946324] [] kthread+0xd1/0xe0 [ 1240.946325] [] ? insert_kthread_work+0x40/0x40 [ 1240.946327] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1240.946328] [] ? insert_kthread_work+0x40/0x40 [ 1240.946329] Code: [ 1240.946329] 0d [ 1240.946330] 48 [ 1240.946330] 98 [ 1240.946330] 83 [ 1240.946330] e2 [ 1240.946331] 30 [ 1240.946331] 48 [ 1240.946331] 81 [ 1240.946331] c2 [ 1240.946332] c0 [ 1240.946332] b8 [ 1240.946332] 01 [ 1240.946333] 00 [ 1240.946333] 48 [ 1240.946333] 03 [ 1240.946333] 14 [ 1240.946334] c5 [ 1240.946334] e0 [ 1240.946334] 17 [ 1240.946334] d5 [ 1240.946335] 93 [ 1240.946335] 4c [ 1240.946335] 89 [ 1240.946336] 02 [ 1240.946336] 41 [ 1240.946336] 8b [ 1240.946336] 40 [ 1240.946337] 08 [ 1240.946337] 85 [ 1240.946337] c0 [ 1240.946337] 75 [ 1240.946338] 0f [ 1240.946338] 0f [ 1240.946338] 1f [ 1240.946338] 44 [ 1240.946339] 00 [ 1240.946339] 00 [ 1240.946339] f3 [ 1240.946339] 90 [ 1240.946340] 41 [ 1240.946340] 8b [ 1240.946340] 40 [ 1240.946341] 08 [ 1240.946341] <85> [ 1240.946341] c0 [ 1240.946341] 74 [ 1240.946342] f6 [ 1240.946342] 4d [ 1240.946342] 8b [ 1240.946342] 08 [ 1240.946343] 4d [ 1240.946343] 85 [ 1240.946343] c9 [ 1240.946343] 74 [ 1240.946344] 04 [ 1240.946344] 41 [ 1240.946344] 0f [ 1240.946344] 18 [ 1240.946345] 09 [ 1240.946345] 8b [ 1240.946345] 17 [ 1240.946345] 0f [ 1240.946346] b7 [ 1240.946346] c2 [ 1240.946346] [ 1241.449426] rdma_cm [ 1241.452355] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1241.531453] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1241.544396] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1241.558660] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1241.570986] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1241.579820] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1241.591380] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1241.597791] RAX: 0000000000000001 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1241.606237] RDX: ffff8852ffb9b8c0 RSI: 0000000000310000 RDI: ffff8852f5420480 [ 1241.614683] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1241.623130] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1241.631575] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1241.640021] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1241.649534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1241.656429] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1241.664876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1241.673322] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1241.681768] Call Trace: [ 1241.684980] [] queued_spin_lock_slowpath+0xb/0xf [ 1241.692457] [] _raw_spin_lock+0x30/0x40 [ 1241.699067] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1241.706745] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1241.715193] [] ? wake_up_atomic_t+0x30/0x30 [ 1241.722191] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1241.729590] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1241.737860] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1241.745839] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1241.754499] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1241.762382] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1241.769689] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1241.777867] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1241.786799] [] ? __getnstimeofday64+0x3f/0xd0 [ 1241.794006] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1241.803054] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1241.811115] [] ? __wake_up_common_lock+0x91/0xc0 [ 1241.818593] [] ? sched_feat_set+0xf0/0xf0 [ 1241.825410] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1241.832889] [] ? __switch_to+0xce/0x5a0 [ 1241.839513] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1241.848250] [] kthread+0xd1/0xe0 [ 1241.854177] [] ? insert_kthread_work+0x40/0x40 [ 1241.861461] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1241.869230] [] ? insert_kthread_work+0x40/0x40 [ 1241.876513] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1241.898755] LNet: Service thread pid 3034 was inactive for 201.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1241.918070] Pid: 3034, comm: mdt01_006 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 1241.929627] Call Trace: [ 1241.932845] [] 0xffffffffffffffff [ 1241.938896] LustreError: dumping log to /tmp/lustre-log.1644346456.3034 [ 1256.880295] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1256.889229] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1256.969079] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1257.029368] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1257.043536] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1257.055860] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1257.064696] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [ 1257.076259] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000202 [ 1257.082670] RAX: 0000000000000001 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1257.091117] RDX: 0000000000590001 RSI: 0000000000d10001 RDI: ffff8852f5420480 [ 1257.099563] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: ffff8892ff55b8c0 [ 1257.108008] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1257.116454] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1257.124900] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1257.134414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1257.141309] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1257.149756] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1257.158202] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1257.166650] Call Trace: [ 1257.169864] [] queued_spin_lock_slowpath+0xb/0xf [ 1257.177344] [] _raw_spin_lock+0x30/0x40 [ 1257.183959] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1257.191644] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1257.200092] [] ? wake_up_atomic_t+0x30/0x30 [ 1257.207092] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1257.214517] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1257.222789] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1257.230772] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1257.239429] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1257.247310] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1257.254624] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1257.262792] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1257.271717] [] ? __getnstimeofday64+0x3f/0xd0 [ 1257.278922] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1257.287968] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1257.296029] [] ? __wake_up_common_lock+0x91/0xc0 [ 1257.303508] [] ? sched_feat_set+0xf0/0xf0 [ 1257.310331] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1257.317811] [] ? __switch_to+0xce/0x5a0 [ 1257.324435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1257.333173] [] kthread+0xd1/0xe0 [ 1257.339100] [] ? insert_kthread_work+0x40/0x40 [ 1257.346384] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1257.354152] [] ? insert_kthread_work+0x40/0x40 [ 1257.361435] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [ 1260.795137] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mdt00_009:4257] [ 1260.804071] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1260.937131] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [mdt01_002:2849] [ 1260.883958] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1260.937132] Modules linked in: [ 1260.937133] osp(OE) [ 1260.937134] mdd(OE) [ 1260.937134] lod(OE) [ 1260.937134] mdt(OE) [ 1260.937135] lfsck(OE) [ 1260.937135] mgc(OE) [ 1260.937135] osd_zfs(OE) [ 1260.937136] lquota(OE) [ 1260.937136] fid(OE) [ 1260.937136] fld(OE) [ 1260.937136] ptlrpc(OE) [ 1260.937137] obdclass(OE) [ 1260.937137] ko2iblnd(OE) [ 1260.937137] lnet(OE) [ 1260.937137] libcfs(OE) [ 1260.937138] iTCO_wdt [ 1260.937138] iTCO_vendor_support [ 1260.937138] rpcrdma [ 1260.937139] ib_iser [ 1260.937139] sb_edac [ 1260.937139] intel_powerclamp [ 1260.937140] coretemp [ 1260.937140] intel_rapl [ 1260.937140] iosf_mbi [ 1260.937140] kvm [ 1260.937141] irqbypass [ 1260.937141] pcspkr [ 1260.937141] joydev [ 1260.937142] lpc_ich [ 1260.937142] i2c_i801 [ 1260.937142] ioatdma [ 1260.937142] ses [ 1260.937143] enclosure [ 1260.937143] sg [ 1260.937143] ipmi_si [ 1260.937143] ipmi_devintf [ 1260.937144] ipmi_msghandler [ 1260.937144] dm_round_robin [ 1260.937144] acpi_cpufreq [ 1260.937144] acpi_power_meter [ 1260.937145] ib_ipoib [ 1260.937145] rdma_ucm [ 1260.937145] ib_umad [ 1260.937146] sch_fq_codel [ 1260.937146] iw_cxgb4 [ 1260.937146] rdma_cm [ 1260.937147] iw_cm [ 1260.937147] ib_cm [ 1260.937147] iw_cxgb3 [ 1260.937148] zfs(POE) [ 1260.937148] binfmt_misc [ 1260.937148] zunicode(POE) [ 1260.937149] zavl(POE) [ 1260.937149] icp(POE) [ 1260.937149] zcommon(POE) [ 1260.937150] znvpair(POE) [ 1260.937150] spl(OE) [ 1260.937150] msr_safe(OE) [ 1260.937151] ip_tables [ 1260.937151] nfsv3 [ 1260.937151] nfs_acl [ 1260.937152] rpcsec_gss_krb5 [ 1260.937152] auth_rpcgss [ 1260.937155] nfsv4 [ 1260.937155] dns_resolver [ 1260.937156] nfs [ 1260.937156] lockd [ 1260.937156] grace [ 1260.937156] fscache [ 1260.937157] overlay(T) [ 1260.937158] ext4 [ 1260.937158] mbcache [ 1260.937158] jbd2 [ 1260.937158] dm_service_time [ 1260.937159] sd_mod [ 1260.937159] crc_t10dif [ 1260.937159] crct10dif_generic [ 1260.937160] mlx5_ib [ 1260.937160] ib_uverbs [ 1260.937160] ib_core [ 1260.937161] be2iscsi [ 1260.937161] bnx2i [ 1260.937161] cnic [ 1260.937161] uio [ 1260.937162] cxgb4i [ 1260.937162] cxgb4 [ 1260.937162] cxgb3i [ 1260.937162] cxgb3 [ 1260.937163] mdio [ 1260.937163] libcxgbi [ 1260.937163] libcxgb [ 1260.937164] qla4xxx [ 1260.937164] iscsi_boot_sysfs [ 1260.937164] 8021q [ 1260.937165] garp [ 1260.937165] mrp [ 1260.937165] stp [ 1260.937165] llc [ 1260.937166] mgag200 [ 1260.937166] i2c_algo_bit [ 1260.937166] drm_kms_helper [ 1260.937167] syscopyarea [ 1260.937167] sysfillrect [ 1260.937167] sysimgblt [ 1260.937168] crct10dif_pclmul [ 1260.937168] fb_sys_fops [ 1260.937168] crct10dif_common [ 1260.937169] ttm [ 1260.937169] crc32_pclmul [ 1260.937169] crc32c_intel [ 1260.937170] mlx5_core [ 1260.937170] ahci [ 1260.937170] ghash_clmulni_intel [ 1260.937171] mxm_wmi [ 1260.937171] aesni_intel [ 1260.937171] mlxfw [ 1260.937172] drm [ 1260.937172] ixgbe(OE) [ 1260.937172] mpt3sas [ 1260.937173] lrw [ 1260.937173] devlink [ 1260.937173] libahci [ 1260.937173] gf128mul [ 1260.937174] dca [ 1260.937174] glue_helper [ 1260.937174] ablk_helper [ 1260.937175] ptp [ 1260.937175] raid_class [ 1260.937175] libata [ 1260.937176] drm_panel_orientation_quirks [ 1260.937176] cryptd [ 1260.937176] scsi_transport_sas [ 1260.937177] pps_core [ 1260.937177] dm_multipath [ 1260.937177] wmi [ 1260.937177] sunrpc [ 1260.937178] dm_mirror [ 1260.937178] dm_region_hash [ 1260.937178] dm_log [ 1260.937178] dm_mod [ 1260.937179] iscsi_tcp [ 1260.937179] libiscsi_tcp [ 1260.937179] libiscsi [ 1260.937180] scsi_transport_iscsi [ 1260.937180] fuse [ 1260.937180] [ 1260.937182] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1260.937182] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1260.937183] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1260.937184] RIP: 0010:[] [ 1260.937186] [] native_queued_spin_lock_slowpath+0x67/0x200 [ 1260.937187] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000202 [ 1260.937188] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000000001 [ 1260.937189] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8892fdfb3340 [ 1260.937189] RBP: ffff8852ebf03ab0 R08: 0000000000000101 R09: ffff8892ff25b8c0 [ 1260.937190] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1260.937191] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1260.937192] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1260.937192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1260.937193] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1260.937194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1260.937194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1260.937195] Call Trace: [ 1260.937197] [] queued_spin_lock_slowpath+0xb/0xf [ 1260.937199] [] _raw_spin_lock+0x30/0x40 [ 1260.937206] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1260.937214] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1260.937216] [] ? wake_up_atomic_t+0x30/0x30 [ 1260.937222] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1260.937249] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1260.937267] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1260.937286] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1260.937303] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1260.937320] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1260.937344] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1260.937365] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1260.937367] [] ? __getnstimeofday64+0x3f/0xd0 [ 1260.937386] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1260.937405] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1260.937406] [] ? __wake_up_common_lock+0x91/0xc0 [ 1260.937407] [] ? sched_feat_set+0xf0/0xf0 [ 1260.937426] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1260.937428] [] ? __switch_to+0xce/0x5a0 [ 1260.937445] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1260.937447] [] kthread+0xd1/0xe0 [ 1260.937449] [] ? insert_kthread_work+0x40/0x40 [ 1260.937450] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1260.937451] [] ? insert_kthread_work+0x40/0x40 [ 1260.937452] Code: [ 1260.937452] 00 [ 1260.937452] 00 [ 1260.937453] 0f [ 1260.937453] 84 [ 1260.937453] 5e [ 1260.937454] 01 [ 1260.937454] 00 [ 1260.937454] 00 [ 1260.937454] 41 [ 1260.937454] b8 [ 1260.937455] 01 [ 1260.937455] 01 [ 1260.937455] 00 [ 1260.937455] 00 [ 1260.937456] b9 [ 1260.937456] 01 [ 1260.937456] 00 [ 1260.937456] 00 [ 1260.937457] 00 [ 1260.937457] eb [ 1260.937457] 20 [ 1260.937457] 0f [ 1260.937458] 1f [ 1260.937458] 80 [ 1260.937458] 00 [ 1260.937458] 00 [ 1260.937459] 00 [ 1260.937459] 00 [ 1260.937459] 83 [ 1260.937459] fe [ 1260.937460] 01 [ 1260.937460] 89 [ 1260.937461] ca [ 1260.937461] 89 [ 1260.937461] f0 [ 1260.937461] 41 [ 1260.937462] 0f [ 1260.937462] 44 [ 1260.937462] d0 [ 1260.937462] f0 [ 1260.937463] 0f [ 1260.937463] b1 [ 1260.937463] 17 [ 1260.937464] <39> [ 1260.937464] c6 [ 1260.937464] 0f [ 1260.937464] 84 [ 1260.937465] 49 [ 1260.937465] 01 [ 1260.937465] 00 [ 1260.937465] 00 [ 1260.937466] 89 [ 1260.937466] c6 [ 1260.937466] f7 [ 1260.937467] c6 [ 1260.937467] 00 [ 1260.937467] ff [ 1260.937467] ff [ 1260.937468] ff [ 1260.937468] 74 [ 1260.937468] df [ 1260.937468] 48 [ 1260.937469] c7 [ 1260.937469] c0 [ 1260.937469] [ 1261.440846] dm_mod [ 1261.443677] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1261.449556] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1261.463721] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1261.476045] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1261.484879] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1261.494500] RSP: 0018:ffff8892ac49fad8 EFLAGS: 00000287 [ 1261.500911] RAX: 0000000000490000 RBX: ffff8852ffb9b8c0 RCX: ffff8892ac49ffd8 [ 1261.509356] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8852f5420480 [ 1261.517801] RBP: ffff8892ac49faf8 R08: ffff8892ac49fb48 R09: ffff8892ff2db8c0 [ 1261.526248] R10: 0000000000000001 R11: 00000000000024c0 R12: 0000000000000001 [ 1261.534695] R13: 00000000000024c0 R14: ffff8852ffb80000 R15: ffff8892ac49fab0 [ 1261.543140] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1261.552654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1261.559548] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1261.567994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1261.576440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1261.584885] Call Trace: [ 1261.588107] [] ? cfs_percpt_lock+0x58/0x110 [libcfs] [ 1261.595994] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1261.603960] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1261.612506] [] ? wake_up_atomic_t+0x30/0x30 [ 1261.619504] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1261.626917] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1261.635194] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1261.643174] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1261.651831] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1261.659712] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1261.667039] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1261.675216] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1261.684148] [] ? __getnstimeofday64+0x3f/0xd0 [ 1261.691354] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1261.700407] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1261.708468] [] ? __wake_up_common_lock+0x91/0xc0 [ 1261.715944] [] ? sched_feat_set+0xf0/0xf0 [ 1261.722759] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1261.730239] [] ? __switch_to+0xce/0x5a0 [ 1261.736861] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1261.745599] [] kthread+0xd1/0xe0 [ 1261.751540] [] ? insert_kthread_work+0x40/0x40 [ 1261.758825] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1261.766592] [] ? insert_kthread_work+0x40/0x40 [ 1261.773873] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1268.888810] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1268.944808] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1268.897840] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1268.944809] Modules linked in: [ 1268.944809] osp(OE) [ 1268.944810] mdd(OE) [ 1268.944810] lod(OE) [ 1268.944811] mdt(OE) [ 1268.944811] lfsck(OE) [ 1268.944811] mgc(OE) [ 1268.944812] osd_zfs(OE) [ 1268.944812] lquota(OE) [ 1268.944812] fid(OE) [ 1268.944813] fld(OE) [ 1268.944813] ptlrpc(OE) [ 1268.944813] obdclass(OE) [ 1268.944814] ko2iblnd(OE) [ 1268.944814] lnet(OE) [ 1268.944814] libcfs(OE) [ 1268.944814] iTCO_wdt [ 1268.944815] iTCO_vendor_support [ 1268.944815] rpcrdma [ 1268.944815] ib_iser [ 1268.944816] sb_edac [ 1268.944816] intel_powerclamp [ 1268.944816] coretemp [ 1268.944817] intel_rapl [ 1268.944817] iosf_mbi [ 1268.944817] kvm [ 1268.944818] irqbypass [ 1268.944818] pcspkr [ 1268.944818] joydev [ 1268.944818] lpc_ich [ 1268.944819] i2c_i801 [ 1268.944819] ioatdma [ 1268.944819] ses [ 1268.944820] enclosure [ 1268.944820] sg [ 1268.944820] ipmi_si [ 1268.944820] ipmi_devintf [ 1268.944821] ipmi_msghandler [ 1268.944821] dm_round_robin [ 1268.944821] acpi_cpufreq [ 1268.944822] acpi_power_meter [ 1268.944822] ib_ipoib [ 1268.944822] rdma_ucm [ 1268.944823] ib_umad [ 1268.944823] sch_fq_codel [ 1268.944823] iw_cxgb4 [ 1268.944824] rdma_cm [ 1268.944824] iw_cm [ 1268.944824] ib_cm [ 1268.944824] iw_cxgb3 [ 1268.944825] zfs(POE) [ 1268.944825] binfmt_misc [ 1268.944826] zunicode(POE) [ 1268.944826] zavl(POE) [ 1268.944827] icp(POE) [ 1268.944827] zcommon(POE) [ 1268.944827] znvpair(POE) [ 1268.944828] spl(OE) [ 1268.944828] msr_safe(OE) [ 1268.944828] ip_tables [ 1268.944829] nfsv3 [ 1268.944829] nfs_acl [ 1268.944830] rpcsec_gss_krb5 [ 1268.944830] auth_rpcgss [ 1268.944830] nfsv4 [ 1268.944831] dns_resolver [ 1268.944831] nfs [ 1268.944832] lockd [ 1268.944832] grace [ 1268.944832] fscache [ 1268.944833] overlay(T) [ 1268.944833] ext4 [ 1268.944834] mbcache [ 1268.944834] jbd2 [ 1268.944834] dm_service_time [ 1268.944834] sd_mod [ 1268.944835] crc_t10dif [ 1268.944835] crct10dif_generic [ 1268.944835] mlx5_ib [ 1268.944836] ib_uverbs [ 1268.944836] ib_core [ 1268.944837] be2iscsi [ 1268.944837] bnx2i [ 1268.944837] cnic [ 1268.944838] uio [ 1268.944838] cxgb4i [ 1268.944838] cxgb4 [ 1268.944839] cxgb3i [ 1268.944839] cxgb3 [ 1268.944839] mdio [ 1268.944840] libcxgbi [ 1268.944840] libcxgb [ 1268.944841] qla4xxx [ 1268.944841] iscsi_boot_sysfs [ 1268.944841] 8021q [ 1268.944842] garp [ 1268.944842] mrp [ 1268.944843] stp [ 1268.944843] llc [ 1268.944843] mgag200 [ 1268.944843] i2c_algo_bit [ 1268.944844] drm_kms_helper [ 1268.944844] syscopyarea [ 1268.944845] sysfillrect [ 1268.944845] sysimgblt [ 1268.944845] crct10dif_pclmul [ 1268.944846] fb_sys_fops [ 1268.944846] crct10dif_common [ 1268.944847] ttm [ 1268.944847] crc32_pclmul [ 1268.944847] crc32c_intel [ 1268.944848] mlx5_core [ 1268.944848] ahci [ 1268.944849] ghash_clmulni_intel [ 1268.944849] mxm_wmi [ 1268.944849] aesni_intel [ 1268.944850] mlxfw [ 1268.944850] drm [ 1268.944850] ixgbe(OE) [ 1268.944851] mpt3sas [ 1268.944851] lrw [ 1268.944852] devlink [ 1268.944852] libahci [ 1268.944853] gf128mul [ 1268.944853] dca [ 1268.944853] glue_helper [ 1268.944854] ablk_helper [ 1268.944854] ptp [ 1268.944854] raid_class [ 1268.944855] libata [ 1268.944855] drm_panel_orientation_quirks [ 1268.944855] cryptd [ 1268.944856] scsi_transport_sas [ 1268.944856] pps_core [ 1268.944856] dm_multipath [ 1268.944857] wmi [ 1268.944857] sunrpc [ 1268.944857] dm_mirror [ 1268.944858] dm_region_hash [ 1268.944858] dm_log [ 1268.944858] dm_mod [ 1268.944859] iscsi_tcp [ 1268.944859] libiscsi_tcp [ 1268.944859] libiscsi [ 1268.944860] scsi_transport_iscsi [ 1268.944860] fuse [ 1268.944860] [ 1268.944862] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1268.944863] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1268.944864] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1268.944865] RIP: 0010:[] [ 1268.944868] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1268.944868] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1268.944869] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1268.944870] RDX: ffff8892ff49b8c0 RSI: 0000000000d10000 RDI: ffff8852f5420480 [ 1268.944870] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1268.944871] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1268.944872] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1268.944873] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1268.944873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1268.944874] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1268.944874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1268.944875] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1268.944875] Call Trace: [ 1268.944878] [] queued_spin_lock_slowpath+0xb/0xf [ 1268.944880] [] _raw_spin_lock+0x30/0x40 [ 1268.944889] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1268.944897] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1268.944899] [] ? wake_up_atomic_t+0x30/0x30 [ 1268.944905] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1268.944936] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1268.944954] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1268.944973] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1268.944990] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1268.945007] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1268.945033] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1268.945055] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1268.945056] [] ? __getnstimeofday64+0x3f/0xd0 [ 1268.945076] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1268.945094] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1268.945096] [] ? __wake_up_common_lock+0x91/0xc0 [ 1268.945097] [] ? sched_feat_set+0xf0/0xf0 [ 1268.945115] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1268.945117] [] ? __switch_to+0xce/0x5a0 [ 1268.945134] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1268.945136] [] kthread+0xd1/0xe0 [ 1268.945138] [] ? insert_kthread_work+0x40/0x40 [ 1268.945140] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1268.945141] [] ? insert_kthread_work+0x40/0x40 [ 1268.945142] Code: [ 1268.945142] 0d [ 1268.945142] 48 [ 1268.945143] 98 [ 1268.945143] 83 [ 1268.945143] e2 [ 1268.945144] 30 [ 1268.945144] 48 [ 1268.945144] 81 [ 1268.945144] c2 [ 1268.945145] c0 [ 1268.945145] b8 [ 1268.945145] 01 [ 1268.945146] 00 [ 1268.945146] 48 [ 1268.945146] 03 [ 1268.945147] 14 [ 1268.945147] c5 [ 1268.945147] e0 [ 1268.945147] 17 [ 1268.945148] d5 [ 1268.945148] 93 [ 1268.945148] 4c [ 1268.945148] 89 [ 1268.945149] 02 [ 1268.945149] 41 [ 1268.945149] 8b [ 1268.945149] 40 [ 1268.945150] 08 [ 1268.945150] 85 [ 1268.945150] c0 [ 1268.945150] 75 [ 1268.945151] 0f [ 1268.945151] 0f [ 1268.945151] 1f [ 1268.945152] 44 [ 1268.945152] 00 [ 1268.945152] 00 [ 1268.945152] f3 [ 1268.945153] 90 [ 1268.945153] 41 [ 1268.945153] 8b [ 1268.945154] 40 [ 1268.945154] 08 [ 1268.945154] <85> [ 1268.945154] c0 [ 1268.945155] 74 [ 1268.945155] f6 [ 1268.945155] 4d [ 1268.945155] 8b [ 1268.945156] 08 [ 1268.945156] 4d [ 1268.945156] 85 [ 1268.945156] c9 [ 1268.945157] 74 [ 1268.945157] 04 [ 1268.945157] 41 [ 1268.945157] 0f [ 1268.945158] 18 [ 1268.945158] 09 [ 1268.945158] 8b [ 1268.945158] 17 [ 1268.945159] 0f [ 1268.945159] b7 [ 1268.945159] c2 [ 1268.945159] [ 1269.448304] rdma_cm [ 1269.451232] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1269.530335] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1269.543276] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1269.557539] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1269.569866] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1269.578699] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1269.590260] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1269.596670] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1269.605115] RDX: ffff8852ffb9b8c0 RSI: 0000000000310001 RDI: ffff8852f5420480 [ 1269.613561] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1269.622008] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1269.630455] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1269.638902] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1269.648415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1269.655311] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1269.663757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1269.672203] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1269.680648] Call Trace: [ 1269.683862] [] queued_spin_lock_slowpath+0xb/0xf [ 1269.691339] [] _raw_spin_lock+0x30/0x40 [ 1269.697950] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1269.705628] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1269.714076] [] ? wake_up_atomic_t+0x30/0x30 [ 1269.721075] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1269.728475] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1269.736746] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1269.744726] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1269.753385] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1269.761266] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1269.768573] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1269.776749] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1269.785682] [] ? __getnstimeofday64+0x3f/0xd0 [ 1269.792888] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1269.801936] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1269.809997] [] ? __wake_up_common_lock+0x91/0xc0 [ 1269.817473] [] ? sched_feat_set+0xf0/0xf0 [ 1269.824290] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1269.831767] [] ? __switch_to+0xce/0x5a0 [ 1269.838390] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1269.847128] [] kthread+0xd1/0xe0 [ 1269.853055] [] ? insert_kthread_work+0x40/0x40 [ 1269.860339] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1269.868107] [] ? insert_kthread_work+0x40/0x40 [ 1269.875391] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1284.879164] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1284.888098] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1284.967929] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1285.028232] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1285.042397] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1285.054722] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1285.063555] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [ 1285.075109] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000202 [ 1285.081518] RAX: 0000000000000001 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1285.089964] RDX: 0000000000d10001 RSI: 0000000000e90000 RDI: ffff8852f5420480 [ 1285.098410] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: ffff8852ffb9b8c0 [ 1285.106856] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1285.115303] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1285.123751] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1285.133262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1285.140158] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1285.148605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1285.157052] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1285.165496] Call Trace: [ 1285.168710] [] queued_spin_lock_slowpath+0xb/0xf [ 1285.176188] [] _raw_spin_lock+0x30/0x40 [ 1285.182801] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1285.190482] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1285.198932] [] ? wake_up_atomic_t+0x30/0x30 [ 1285.205929] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1285.213349] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1285.221621] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1285.229602] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1285.238260] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1285.246143] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1285.253458] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1285.261634] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1285.270567] [] ? __getnstimeofday64+0x3f/0xd0 [ 1285.277774] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1285.286822] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1285.294881] [] ? __wake_up_common_lock+0x91/0xc0 [ 1285.302359] [] ? sched_feat_set+0xf0/0xf0 [ 1285.309176] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1285.316655] [] ? __switch_to+0xce/0x5a0 [ 1285.323277] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1285.332016] [] kthread+0xd1/0xe0 [ 1285.337943] [] ? insert_kthread_work+0x40/0x40 [ 1285.345226] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1285.352996] [] ? insert_kthread_work+0x40/0x40 [ 1285.360280] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [ 1288.794006] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mdt00_009:4257] [ 1288.802959] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1288.936000] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mdt01_002:2849] [ 1288.882744] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1288.936001] Modules linked in: [ 1288.936002] osp(OE) [ 1288.936002] mdd(OE) [ 1288.936003] lod(OE) [ 1288.936003] mdt(OE) [ 1288.936003] lfsck(OE) [ 1288.936003] mgc(OE) [ 1288.936004] osd_zfs(OE) [ 1288.936004] lquota(OE) [ 1288.936004] fid(OE) [ 1288.936005] fld(OE) [ 1288.936005] ptlrpc(OE) [ 1288.936005] obdclass(OE) [ 1288.936005] ko2iblnd(OE) [ 1288.936006] lnet(OE) [ 1288.936006] libcfs(OE) [ 1288.936006] iTCO_wdt [ 1288.936007] iTCO_vendor_support [ 1288.936007] rpcrdma [ 1288.936007] ib_iser [ 1288.936008] sb_edac [ 1288.936008] intel_powerclamp [ 1288.936008] coretemp [ 1288.936009] intel_rapl [ 1288.936009] iosf_mbi [ 1288.936009] kvm [ 1288.936010] irqbypass [ 1288.936010] pcspkr [ 1288.936010] joydev [ 1288.936010] lpc_ich [ 1288.936011] i2c_i801 [ 1288.936011] ioatdma [ 1288.936011] ses [ 1288.936012] enclosure [ 1288.936012] sg [ 1288.936012] ipmi_si [ 1288.936012] ipmi_devintf [ 1288.936013] ipmi_msghandler [ 1288.936013] dm_round_robin [ 1288.936013] acpi_cpufreq [ 1288.936014] acpi_power_meter [ 1288.936014] ib_ipoib [ 1288.936014] rdma_ucm [ 1288.936014] ib_umad [ 1288.936015] sch_fq_codel [ 1288.936015] iw_cxgb4 [ 1288.936015] rdma_cm [ 1288.936015] iw_cm [ 1288.936016] ib_cm [ 1288.936016] iw_cxgb3 [ 1288.936016] zfs(POE) [ 1288.936017] binfmt_misc [ 1288.936017] zunicode(POE) [ 1288.936017] zavl(POE) [ 1288.936018] icp(POE) [ 1288.936018] zcommon(POE) [ 1288.936018] znvpair(POE) [ 1288.936019] spl(OE) [ 1288.936019] msr_safe(OE) [ 1288.936020] ip_tables [ 1288.936020] nfsv3 [ 1288.936020] nfs_acl [ 1288.936021] rpcsec_gss_krb5 [ 1288.936021] auth_rpcgss [ 1288.936021] nfsv4 [ 1288.936022] dns_resolver [ 1288.936022] nfs [ 1288.936022] lockd [ 1288.936023] grace [ 1288.936023] fscache [ 1288.936023] overlay(T) [ 1288.936024] ext4 [ 1288.936024] mbcache [ 1288.936025] jbd2 [ 1288.936025] dm_service_time [ 1288.936025] sd_mod [ 1288.936026] crc_t10dif [ 1288.936026] crct10dif_generic [ 1288.936026] mlx5_ib [ 1288.936027] ib_uverbs [ 1288.936027] ib_core [ 1288.936027] be2iscsi [ 1288.936028] bnx2i [ 1288.936028] cnic [ 1288.936028] uio [ 1288.936029] cxgb4i [ 1288.936029] cxgb4 [ 1288.936029] cxgb3i [ 1288.936029] cxgb3 [ 1288.936030] mdio [ 1288.936030] libcxgbi [ 1288.936030] libcxgb [ 1288.936031] qla4xxx [ 1288.936031] iscsi_boot_sysfs [ 1288.936031] 8021q [ 1288.936032] garp [ 1288.936032] mrp [ 1288.936033] stp [ 1288.936033] llc [ 1288.936033] mgag200 [ 1288.936033] i2c_algo_bit [ 1288.936034] drm_kms_helper [ 1288.936034] syscopyarea [ 1288.936034] sysfillrect [ 1288.936035] sysimgblt [ 1288.936035] crct10dif_pclmul [ 1288.936036] fb_sys_fops [ 1288.936036] crct10dif_common [ 1288.936036] ttm [ 1288.936036] crc32_pclmul [ 1288.936037] crc32c_intel [ 1288.936037] mlx5_core [ 1288.936038] ahci [ 1288.936038] ghash_clmulni_intel [ 1288.936038] mxm_wmi [ 1288.936039] aesni_intel [ 1288.936039] mlxfw [ 1288.936039] drm [ 1288.936040] ixgbe(OE) [ 1288.936040] mpt3sas [ 1288.936040] lrw [ 1288.936041] devlink [ 1288.936041] libahci [ 1288.936041] gf128mul [ 1288.936041] dca [ 1288.936042] glue_helper [ 1288.936042] ablk_helper [ 1288.936042] ptp [ 1288.936043] raid_class [ 1288.936043] libata [ 1288.936043] drm_panel_orientation_quirks [ 1288.936044] cryptd [ 1288.936044] scsi_transport_sas [ 1288.936044] pps_core [ 1288.936045] dm_multipath [ 1288.936045] wmi [ 1288.936045] sunrpc [ 1288.936046] dm_mirror [ 1288.936046] dm_region_hash [ 1288.936046] dm_log [ 1288.936046] dm_mod [ 1288.936047] iscsi_tcp [ 1288.936047] libiscsi_tcp [ 1288.936047] libiscsi [ 1288.936048] scsi_transport_iscsi [ 1288.936048] fuse [ 1288.936048] [ 1288.936050] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1288.936051] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1288.936051] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1288.936052] RIP: 0010:[] [ 1288.936055] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1288.936055] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1288.936056] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1288.936057] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1288.936058] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1288.936058] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1288.936059] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1288.936060] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1288.936060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1288.936061] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1288.936062] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1288.936062] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1288.936062] Call Trace: [ 1288.936065] [] queued_spin_lock_slowpath+0xb/0xf [ 1288.936067] [] _raw_spin_lock+0x30/0x40 [ 1288.936076] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1288.936085] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1288.936087] [] ? wake_up_atomic_t+0x30/0x30 [ 1288.936092] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1288.936122] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1288.936140] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1288.936164] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1288.936188] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1288.936212] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1288.936242] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1288.936269] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1288.936270] [] ? __getnstimeofday64+0x3f/0xd0 [ 1288.936296] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1288.936321] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1288.936323] [] ? __wake_up_common_lock+0x91/0xc0 [ 1288.936324] [] ? sched_feat_set+0xf0/0xf0 [ 1288.936349] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1288.936351] [] ? __switch_to+0xce/0x5a0 [ 1288.936376] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1288.936378] [] kthread+0xd1/0xe0 [ 1288.936379] [] ? insert_kthread_work+0x40/0x40 [ 1288.936381] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1288.936382] [] ? insert_kthread_work+0x40/0x40 [ 1288.936383] Code: [ 1288.936383] 0d [ 1288.936384] 48 [ 1288.936384] 98 [ 1288.936384] 83 [ 1288.936385] e2 [ 1288.936385] 30 [ 1288.936385] 48 [ 1288.936385] 81 [ 1288.936386] c2 [ 1288.936386] c0 [ 1288.936386] b8 [ 1288.936386] 01 [ 1288.936387] 00 [ 1288.936387] 48 [ 1288.936387] 03 [ 1288.936387] 14 [ 1288.936388] c5 [ 1288.936388] e0 [ 1288.936388] 17 [ 1288.936389] d5 [ 1288.936389] 93 [ 1288.936389] 4c [ 1288.936389] 89 [ 1288.936390] 02 [ 1288.936390] 41 [ 1288.936390] 8b [ 1288.936390] 40 [ 1288.936391] 08 [ 1288.936391] 85 [ 1288.936391] c0 [ 1288.936391] 75 [ 1288.936392] 0f [ 1288.936392] 0f [ 1288.936392] 1f [ 1288.936393] 44 [ 1288.936393] 00 [ 1288.936393] 00 [ 1288.936393] f3 [ 1288.936394] 90 [ 1288.936394] 41 [ 1288.936394] 8b [ 1288.936394] 40 [ 1288.936395] 08 [ 1288.936395] <85> [ 1288.936395] c0 [ 1288.936396] 74 [ 1288.936396] f6 [ 1288.936396] 4d [ 1288.936396] 8b [ 1288.936396] 08 [ 1288.936397] 4d [ 1288.936397] 85 [ 1288.936397] c9 [ 1288.936397] 74 [ 1288.936398] 04 [ 1288.936398] 41 [ 1288.936398] 0f [ 1288.936398] 18 [ 1288.936399] 09 [ 1288.936399] 8b [ 1288.936399] 17 [ 1288.936399] 0f [ 1288.936400] b7 [ 1288.936400] c2 [ 1288.936400] [ 1289.439379] dm_mod [ 1289.442209] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1289.448086] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1289.462253] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1289.474576] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1289.483410] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1289.494965] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000246 [ 1289.501373] RAX: 0000000000000000 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1289.509819] RDX: ffff8892ff55b8c0 RSI: 0000000000e90000 RDI: ffff8852f5420480 [ 1289.518264] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: 0000000000000000 [ 1289.526709] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1289.535153] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1289.543599] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1289.553111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1289.560004] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1289.568450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1289.576895] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1289.585341] Call Trace: [ 1289.588556] [] queued_spin_lock_slowpath+0xb/0xf [ 1289.596026] [] _raw_spin_lock+0x30/0x40 [ 1289.602639] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1289.610312] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1289.618288] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1289.626832] [] ? wake_up_atomic_t+0x30/0x30 [ 1289.633830] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1289.641246] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1289.649515] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1289.657496] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1289.666154] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1289.674035] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1289.681346] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1289.689523] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1289.698455] [] ? __getnstimeofday64+0x3f/0xd0 [ 1289.705660] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1289.714713] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1289.722773] [] ? __wake_up_common_lock+0x91/0xc0 [ 1289.730250] [] ? sched_feat_set+0xf0/0xf0 [ 1289.737065] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1289.744544] [] ? __switch_to+0xce/0x5a0 [ 1289.751165] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1289.759902] [] kthread+0xd1/0xe0 [ 1289.765829] [] ? insert_kthread_work+0x40/0x40 [ 1289.773114] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1289.780881] [] ? insert_kthread_work+0x40/0x40 [ 1289.788164] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1291.694897] LNet: Service thread pid 4292 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1291.714226] Pid: 4292, comm: mdt01_013 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 1291.725782] Call Trace: [ 1291.728999] [] 0xffffffffffffffff [ 1291.735059] LustreError: dumping log to /tmp/lustre-log.1644346506.4292 [ 1296.887679] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1296.943677] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1296.896710] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1296.943678] Modules linked in: [ 1296.943678] osp(OE) [ 1296.943679] mdd(OE) [ 1296.943679] lod(OE) [ 1296.943680] mdt(OE) [ 1296.943680] lfsck(OE) [ 1296.943680] mgc(OE) [ 1296.943681] osd_zfs(OE) [ 1296.943681] lquota(OE) [ 1296.943681] fid(OE) [ 1296.943682] fld(OE) [ 1296.943682] ptlrpc(OE) [ 1296.943682] obdclass(OE) [ 1296.943683] ko2iblnd(OE) [ 1296.943683] lnet(OE) [ 1296.943683] libcfs(OE) [ 1296.943684] iTCO_wdt [ 1296.943684] iTCO_vendor_support [ 1296.943684] rpcrdma [ 1296.943685] ib_iser [ 1296.943685] sb_edac [ 1296.943685] intel_powerclamp [ 1296.943685] coretemp [ 1296.943686] intel_rapl [ 1296.943686] iosf_mbi [ 1296.943686] kvm [ 1296.943687] irqbypass [ 1296.943687] pcspkr [ 1296.943687] joydev [ 1296.943688] lpc_ich [ 1296.943688] i2c_i801 [ 1296.943688] ioatdma [ 1296.943688] ses [ 1296.943689] enclosure [ 1296.943689] sg [ 1296.943689] ipmi_si [ 1296.943690] ipmi_devintf [ 1296.943690] ipmi_msghandler [ 1296.943690] dm_round_robin [ 1296.943691] acpi_cpufreq [ 1296.943691] acpi_power_meter [ 1296.943691] ib_ipoib [ 1296.943691] rdma_ucm [ 1296.943692] ib_umad [ 1296.943692] sch_fq_codel [ 1296.943692] iw_cxgb4 [ 1296.943693] rdma_cm [ 1296.943693] iw_cm [ 1296.943693] ib_cm [ 1296.943693] iw_cxgb3 [ 1296.943694] zfs(POE) [ 1296.943694] binfmt_misc [ 1296.943695] zunicode(POE) [ 1296.943695] zavl(POE) [ 1296.943696] icp(POE) [ 1296.943696] zcommon(POE) [ 1296.943697] znvpair(POE) [ 1296.943697] spl(OE) [ 1296.943697] msr_safe(OE) [ 1296.943698] ip_tables [ 1296.943698] nfsv3 [ 1296.943699] nfs_acl [ 1296.943699] rpcsec_gss_krb5 [ 1296.943700] auth_rpcgss [ 1296.943700] nfsv4 [ 1296.943700] dns_resolver [ 1296.943700] nfs [ 1296.943701] lockd [ 1296.943701] grace [ 1296.943702] fscache [ 1296.943702] overlay(T) [ 1296.943703] ext4 [ 1296.943703] mbcache [ 1296.943703] jbd2 [ 1296.943704] dm_service_time [ 1296.943704] sd_mod [ 1296.943704] crc_t10dif [ 1296.943705] crct10dif_generic [ 1296.943705] mlx5_ib [ 1296.943705] ib_uverbs [ 1296.943706] ib_core [ 1296.943706] be2iscsi [ 1296.943706] bnx2i [ 1296.943706] cnic [ 1296.943707] uio [ 1296.943707] cxgb4i [ 1296.943707] cxgb4 [ 1296.943708] cxgb3i [ 1296.943708] cxgb3 [ 1296.943708] mdio [ 1296.943709] libcxgbi [ 1296.943709] libcxgb [ 1296.943710] qla4xxx [ 1296.943710] iscsi_boot_sysfs [ 1296.943711] 8021q [ 1296.943711] garp [ 1296.943712] mrp [ 1296.943712] stp [ 1296.943712] llc [ 1296.943713] mgag200 [ 1296.943713] i2c_algo_bit [ 1296.943713] drm_kms_helper [ 1296.943714] syscopyarea [ 1296.943714] sysfillrect [ 1296.943714] sysimgblt [ 1296.943715] crct10dif_pclmul [ 1296.943715] fb_sys_fops [ 1296.943715] crct10dif_common [ 1296.943716] ttm [ 1296.943716] crc32_pclmul [ 1296.943717] crc32c_intel [ 1296.943717] mlx5_core [ 1296.943718] ahci [ 1296.943718] ghash_clmulni_intel [ 1296.943718] mxm_wmi [ 1296.943719] aesni_intel [ 1296.943719] mlxfw [ 1296.943719] drm [ 1296.943720] ixgbe(OE) [ 1296.943720] mpt3sas [ 1296.943720] lrw [ 1296.943721] devlink [ 1296.943721] libahci [ 1296.943722] gf128mul [ 1296.943722] dca [ 1296.943722] glue_helper [ 1296.943723] ablk_helper [ 1296.943723] ptp [ 1296.943724] raid_class [ 1296.943724] libata [ 1296.943724] drm_panel_orientation_quirks [ 1296.943725] cryptd [ 1296.943725] scsi_transport_sas [ 1296.943726] pps_core [ 1296.943726] dm_multipath [ 1296.943726] wmi [ 1296.943727] sunrpc [ 1296.943727] dm_mirror [ 1296.943728] dm_region_hash [ 1296.943728] dm_log [ 1296.943729] dm_mod [ 1296.943729] iscsi_tcp [ 1296.943730] libiscsi_tcp [ 1296.943730] libiscsi [ 1296.943730] scsi_transport_iscsi [ 1296.943731] fuse [ 1296.943731] [ 1296.943735] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1296.943736] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1296.943737] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1296.943738] RIP: 0010:[] [ 1296.943743] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1296.943744] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1296.943745] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1296.943746] RDX: ffff8852ffb9b8c0 RSI: 0000000000310000 RDI: ffff8852f5420480 [ 1296.943747] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1296.943748] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1296.943749] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1296.943751] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1296.943752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1296.943752] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1296.943754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1296.943755] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1296.943755] Call Trace: [ 1296.943760] [] queued_spin_lock_slowpath+0xb/0xf [ 1296.943762] [] _raw_spin_lock+0x30/0x40 [ 1296.943773] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1296.943784] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1296.943786] [] ? wake_up_atomic_t+0x30/0x30 [ 1296.943792] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1296.943831] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1296.943849] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1296.943868] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1296.943886] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1296.943903] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1296.943931] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1296.943953] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1296.943955] [] ? __getnstimeofday64+0x3f/0xd0 [ 1296.943974] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1296.943993] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1296.943994] [] ? __wake_up_common_lock+0x91/0xc0 [ 1296.943995] [] ? sched_feat_set+0xf0/0xf0 [ 1296.944014] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1296.944016] [] ? __switch_to+0xce/0x5a0 [ 1296.944034] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1296.944037] [] kthread+0xd1/0xe0 [ 1296.944038] [] ? insert_kthread_work+0x40/0x40 [ 1296.944040] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1296.944041] [] ? insert_kthread_work+0x40/0x40 [ 1296.944042] Code: [ 1296.944042] c1 [ 1296.944043] e8 [ 1296.944043] 13 [ 1296.944043] 48 [ 1296.944043] c1 [ 1296.944044] ea [ 1296.944044] 0d [ 1296.944044] 48 [ 1296.944044] 98 [ 1296.944045] 83 [ 1296.944045] e2 [ 1296.944045] 30 [ 1296.944046] 48 [ 1296.944046] 81 [ 1296.944046] c2 [ 1296.944046] c0 [ 1296.944047] b8 [ 1296.944047] 01 [ 1296.944047] 00 [ 1296.944047] 48 [ 1296.944048] 03 [ 1296.944048] 14 [ 1296.944048] c5 [ 1296.944049] e0 [ 1296.944049] 17 [ 1296.944049] d5 [ 1296.944049] 93 [ 1296.944050] 4c [ 1296.944050] 89 [ 1296.944050] 02 [ 1296.944050] 41 [ 1296.944051] 8b [ 1296.944051] 40 [ 1296.944051] 08 [ 1296.944052] 85 [ 1296.944052] c0 [ 1296.944052] 75 [ 1296.944052] 0f [ 1296.944053] 0f [ 1296.944053] 1f [ 1296.944053] 44 [ 1296.944053] 00 [ 1296.944054] 00 [ 1296.944054] [ 1296.944054] 90 [ 1296.944054] 41 [ 1296.944055] 8b [ 1296.944055] 40 [ 1296.944055] 08 [ 1296.944055] 85 [ 1296.944056] c0 [ 1296.944056] 74 [ 1296.944056] f6 [ 1296.944056] 4d [ 1296.944057] 8b [ 1296.944057] 08 [ 1296.944057] 4d [ 1296.944057] 85 [ 1296.944058] c9 [ 1296.944058] 74 [ 1296.944058] 04 [ 1296.944058] 41 [ 1296.944059] 0f [ 1296.944059] 18 [ 1296.944059] [ 1297.447212] rdma_cm [ 1297.450142] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1297.529255] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1297.542193] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1297.556456] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1297.568780] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1297.577614] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1297.589175] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1297.595586] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1297.604031] RDX: ffff8892ff49b8c0 RSI: 0000000000d10001 RDI: ffff8852f5420480 [ 1297.612478] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1297.620923] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1297.629369] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1297.637816] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1297.647329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1297.654223] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1297.662670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1297.671118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1297.679565] Call Trace: [ 1297.682777] [] queued_spin_lock_slowpath+0xb/0xf [ 1297.690254] [] _raw_spin_lock+0x30/0x40 [ 1297.696865] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1297.704544] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1297.712991] [] ? wake_up_atomic_t+0x30/0x30 [ 1297.719989] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1297.727389] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1297.735661] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1297.743642] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1297.752302] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1297.760185] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1297.767492] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1297.775667] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1297.784600] [] ? __getnstimeofday64+0x3f/0xd0 [ 1297.791805] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1297.800852] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1297.808912] [] ? __wake_up_common_lock+0x91/0xc0 [ 1297.816390] [] ? sched_feat_set+0xf0/0xf0 [ 1297.823209] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1297.830687] [] ? __switch_to+0xce/0x5a0 [ 1297.837312] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1297.846049] [] kthread+0xd1/0xe0 [ 1297.851975] [] ? insert_kthread_work+0x40/0x40 [ 1297.859260] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1297.867030] [] ? insert_kthread_work+0x40/0x40 [ 1297.874313] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1312.878249] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1312.887183] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1312.967032] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1313.027319] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1313.041485] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1313.053811] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1313.062645] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1313.074206] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000246 [ 1313.080616] RAX: 0000000000000000 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1313.089062] RDX: ffff8852ffadb8c0 RSI: 0000000000190000 RDI: ffff8852f5420480 [ 1313.097508] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: 0000000000000000 [ 1313.105953] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1313.114398] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1313.122844] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1313.132356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1313.139251] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1313.147698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1313.156145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1313.164591] Call Trace: [ 1313.167806] [] queued_spin_lock_slowpath+0xb/0xf [ 1313.175275] [] _raw_spin_lock+0x30/0x40 [ 1313.181890] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1313.189572] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1313.198021] [] ? wake_up_atomic_t+0x30/0x30 [ 1313.205019] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1313.212441] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1313.220714] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1313.228695] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1313.237352] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1313.245234] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1313.252546] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1313.260723] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1313.269655] [] ? __getnstimeofday64+0x3f/0xd0 [ 1313.276861] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1313.285908] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1313.293969] [] ? __wake_up_common_lock+0x91/0xc0 [ 1313.301447] [] ? sched_feat_set+0xf0/0xf0 [ 1313.308263] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1313.315742] [] ? __switch_to+0xce/0x5a0 [ 1313.322365] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1313.331103] [] kthread+0xd1/0xe0 [ 1313.337029] [] ? insert_kthread_work+0x40/0x40 [ 1313.344312] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1313.352081] [] ? insert_kthread_work+0x40/0x40 [ 1313.359365] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1316.793169] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mdt00_009:4257] [ 1316.802120] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1316.935165] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mdt01_002:2849] [ 1316.881902] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1316.935166] Modules linked in: [ 1316.935167] osp(OE) [ 1316.935167] mdd(OE) [ 1316.935168] lod(OE) [ 1316.935168] mdt(OE) [ 1316.935168] lfsck(OE) [ 1316.935169] mgc(OE) [ 1316.935169] osd_zfs(OE) [ 1316.935170] lquota(OE) [ 1316.935170] fid(OE) [ 1316.935171] fld(OE) [ 1316.935171] ptlrpc(OE) [ 1316.935171] obdclass(OE) [ 1316.935172] ko2iblnd(OE) [ 1316.935172] lnet(OE) [ 1316.935172] libcfs(OE) [ 1316.935173] iTCO_wdt [ 1316.935173] iTCO_vendor_support [ 1316.935173] rpcrdma [ 1316.935174] ib_iser [ 1316.935174] sb_edac [ 1316.935174] intel_powerclamp [ 1316.935175] coretemp [ 1316.935175] intel_rapl [ 1316.935175] iosf_mbi [ 1316.935176] kvm [ 1316.935176] irqbypass [ 1316.935176] pcspkr [ 1316.935176] joydev [ 1316.935177] lpc_ich [ 1316.935177] i2c_i801 [ 1316.935177] ioatdma [ 1316.935177] ses [ 1316.935178] enclosure [ 1316.935178] sg [ 1316.935178] ipmi_si [ 1316.935179] ipmi_devintf [ 1316.935179] ipmi_msghandler [ 1316.935179] dm_round_robin [ 1316.935179] acpi_cpufreq [ 1316.935180] acpi_power_meter [ 1316.935180] ib_ipoib [ 1316.935180] rdma_ucm [ 1316.935181] ib_umad [ 1316.935181] sch_fq_codel [ 1316.935181] iw_cxgb4 [ 1316.935181] rdma_cm [ 1316.935182] iw_cm [ 1316.935182] ib_cm [ 1316.935182] iw_cxgb3 [ 1316.935183] zfs(POE) [ 1316.935183] binfmt_misc [ 1316.935183] zunicode(POE) [ 1316.935184] zavl(POE) [ 1316.935184] icp(POE) [ 1316.935184] zcommon(POE) [ 1316.935185] znvpair(POE) [ 1316.935185] spl(OE) [ 1316.935185] msr_safe(OE) [ 1316.935186] ip_tables [ 1316.935186] nfsv3 [ 1316.935186] nfs_acl [ 1316.935187] rpcsec_gss_krb5 [ 1316.935187] auth_rpcgss [ 1316.935187] nfsv4 [ 1316.935187] dns_resolver [ 1316.935188] nfs [ 1316.935188] lockd [ 1316.935188] grace [ 1316.935188] fscache [ 1316.935189] overlay(T) [ 1316.935189] ext4 [ 1316.935190] mbcache [ 1316.935190] jbd2 [ 1316.935190] dm_service_time [ 1316.935191] sd_mod [ 1316.935191] crc_t10dif [ 1316.935191] crct10dif_generic [ 1316.935192] mlx5_ib [ 1316.935192] ib_uverbs [ 1316.935192] ib_core [ 1316.935193] be2iscsi [ 1316.935193] bnx2i [ 1316.935193] cnic [ 1316.935193] uio [ 1316.935194] cxgb4i [ 1316.935194] cxgb4 [ 1316.935194] cxgb3i [ 1316.935194] cxgb3 [ 1316.935195] mdio [ 1316.935195] libcxgbi [ 1316.935195] libcxgb [ 1316.935195] qla4xxx [ 1316.935196] iscsi_boot_sysfs [ 1316.935196] 8021q [ 1316.935196] garp [ 1316.935197] mrp [ 1316.935197] stp [ 1316.935197] llc [ 1316.935198] mgag200 [ 1316.935198] i2c_algo_bit [ 1316.935198] drm_kms_helper [ 1316.935199] syscopyarea [ 1316.935199] sysfillrect [ 1316.935199] sysimgblt [ 1316.935199] crct10dif_pclmul [ 1316.935200] fb_sys_fops [ 1316.935200] crct10dif_common [ 1316.935201] ttm [ 1316.935201] crc32_pclmul [ 1316.935201] crc32c_intel [ 1316.935201] mlx5_core [ 1316.935202] ahci [ 1316.935202] ghash_clmulni_intel [ 1316.935202] mxm_wmi [ 1316.935203] aesni_intel [ 1316.935203] mlxfw [ 1316.935203] drm [ 1316.935204] ixgbe(OE) [ 1316.935204] mpt3sas [ 1316.935205] lrw [ 1316.935205] devlink [ 1316.935205] libahci [ 1316.935206] gf128mul [ 1316.935206] dca [ 1316.935206] glue_helper [ 1316.935206] ablk_helper [ 1316.935207] ptp [ 1316.935207] raid_class [ 1316.935207] libata [ 1316.935208] drm_panel_orientation_quirks [ 1316.935208] cryptd [ 1316.935208] scsi_transport_sas [ 1316.935208] pps_core [ 1316.935209] dm_multipath [ 1316.935209] wmi [ 1316.935209] sunrpc [ 1316.935210] dm_mirror [ 1316.935210] dm_region_hash [ 1316.935210] dm_log [ 1316.935211] dm_mod [ 1316.935211] iscsi_tcp [ 1316.935211] libiscsi_tcp [ 1316.935212] libiscsi [ 1316.935212] scsi_transport_iscsi [ 1316.935212] fuse [ 1316.935213] [ 1316.935214] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1316.935215] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1316.935215] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1316.935216] RIP: 0010:[] [ 1316.935219] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1316.935219] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1316.935220] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1316.935221] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1316.935221] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1316.935222] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1316.935223] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1316.935223] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1316.935224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1316.935225] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1316.935225] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1316.935226] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1316.935226] Call Trace: [ 1316.935229] [] queued_spin_lock_slowpath+0xb/0xf [ 1316.935230] [] _raw_spin_lock+0x30/0x40 [ 1316.935239] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1316.935247] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1316.935249] [] ? wake_up_atomic_t+0x30/0x30 [ 1316.935255] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1316.935283] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1316.935301] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1316.935320] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1316.935337] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1316.935355] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1316.935380] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1316.935401] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1316.935402] [] ? __getnstimeofday64+0x3f/0xd0 [ 1316.935422] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1316.935440] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1316.935441] [] ? __wake_up_common_lock+0x91/0xc0 [ 1316.935442] [] ? sched_feat_set+0xf0/0xf0 [ 1316.935461] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1316.935462] [] ? __switch_to+0xce/0x5a0 [ 1316.935480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1316.935482] [] kthread+0xd1/0xe0 [ 1316.935483] [] ? insert_kthread_work+0x40/0x40 [ 1316.935484] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1316.935486] [] ? insert_kthread_work+0x40/0x40 [ 1316.935486] Code: [ 1316.935487] 0d [ 1316.935487] 48 [ 1316.935487] 98 [ 1316.935488] 83 [ 1316.935488] e2 [ 1316.935488] 30 [ 1316.935489] 48 [ 1316.935489] 81 [ 1316.935489] c2 [ 1316.935489] c0 [ 1316.935490] b8 [ 1316.935490] 01 [ 1316.935490] 00 [ 1316.935490] 48 [ 1316.935491] 03 [ 1316.935491] 14 [ 1316.935491] c5 [ 1316.935492] e0 [ 1316.935492] 17 [ 1316.935492] d5 [ 1316.935492] 93 [ 1316.935493] 4c [ 1316.935493] 89 [ 1316.935493] 02 [ 1316.935493] 41 [ 1316.935494] 8b [ 1316.935494] 40 [ 1316.935494] 08 [ 1316.935494] 85 [ 1316.935495] c0 [ 1316.935495] 75 [ 1316.935495] 0f [ 1316.935496] 0f [ 1316.935496] 1f [ 1316.935496] 44 [ 1316.935496] 00 [ 1316.935497] 00 [ 1316.935497] f3 [ 1316.935497] 90 [ 1316.935497] 41 [ 1316.935498] 8b [ 1316.935498] 40 [ 1316.935498] 08 [ 1316.935498] <85> [ 1316.935499] c0 [ 1316.935499] 74 [ 1316.935499] f6 [ 1316.935499] 4d [ 1316.935500] 8b [ 1316.935500] 08 [ 1316.935500] 4d [ 1316.935500] 85 [ 1316.935501] c9 [ 1316.935501] 74 [ 1316.935501] 04 [ 1316.935501] 41 [ 1316.935501] 0f [ 1316.935502] 18 [ 1316.935502] 09 [ 1316.935502] 8b [ 1316.935502] 17 [ 1316.935503] 0f [ 1316.935503] b7 [ 1316.935503] c2 [ 1316.935503] [ 1317.438552] dm_mod [ 1317.441384] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1317.447261] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1317.461425] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1317.473748] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1317.482581] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1317.494142] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000246 [ 1317.500551] RAX: 0000000000000001 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1317.508996] RDX: ffff8892ff55b8c0 RSI: 0000000000e90000 RDI: ffff8852f5420480 [ 1317.517442] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: 0000000000000000 [ 1317.525887] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1317.534333] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1317.542780] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1317.552293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1317.559186] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1317.567632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1317.576077] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1317.584522] Call Trace: [ 1317.587737] [] queued_spin_lock_slowpath+0xb/0xf [ 1317.595215] [] _raw_spin_lock+0x30/0x40 [ 1317.601827] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1317.609502] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1317.617467] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1317.626012] [] ? wake_up_atomic_t+0x30/0x30 [ 1317.633008] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1317.640422] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1317.648692] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1317.656672] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1317.665330] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1317.673211] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1317.680521] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1317.688696] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1317.697630] [] ? __getnstimeofday64+0x3f/0xd0 [ 1317.704836] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1317.713881] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1317.721940] [] ? __wake_up_common_lock+0x91/0xc0 [ 1317.729416] [] ? sched_feat_set+0xf0/0xf0 [ 1317.736233] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1317.743711] [] ? __switch_to+0xce/0x5a0 [ 1317.750332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1317.759069] [] kthread+0xd1/0xe0 [ 1317.764996] [] ? insert_kthread_work+0x40/0x40 [ 1317.772280] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1317.780048] [] ? insert_kthread_work+0x40/0x40 [ 1317.787330] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1322.240150] INFO: task mount.lustre:2109 blocked for more than 120 seconds. [ 1322.248412] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1322.257643] mount.lustre D ffff8852f4c51660 0 2109 2028 0x00000080 [ 1322.266030] Call Trace: [ 1322.269242] [] schedule+0x29/0x70 [ 1322.275274] [] schedule_timeout+0x271/0x320 [ 1322.282279] [] ? select_task_rq_fair+0x690/0x780 [ 1322.289754] [] wait_for_completion+0x12d/0x170 [ 1322.297047] [] ? wake_up_state+0x20/0x20 [ 1322.303787] [] llog_process_or_fork+0x244/0x450 [obdclass] [ 1322.312248] [] llog_process+0x14/0x20 [obdclass] [ 1322.319767] [] class_config_parse_llog+0x125/0x350 [obdclass] [ 1322.328516] [] mgc_process_cfg_log+0x7c0/0xc70 [mgc] [ 1322.336391] [] mgc_process_log+0x3ec/0x920 [mgc] [ 1322.343886] [] ? config_recover_log_add+0x13f/0x280 [mgc] [ 1322.352261] [] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [ 1322.361398] [] mgc_process_config+0x8bb/0x1420 [mgc] [ 1322.369285] [] lustre_process_log+0x2d8/0xad0 [obdclass] [ 1322.377571] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 1322.385464] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 1322.393942] [] server_start_targets+0x13a4/0x2a20 [obdclass] [ 1322.402607] [] ? lustre_start_mgc+0x260/0x2510 [obdclass] [ 1322.410982] [] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [ 1322.420132] [] server_fill_super+0x10cc/0x1890 [obdclass] [ 1322.428495] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 1322.436384] [] lustre_fill_super+0x468/0x960 [obdclass] [ 1322.444560] [] ? lustre_common_put_super+0x270/0x270 [obdclass] [ 1322.453504] [] mount_nodev+0x4f/0xb0 [ 1322.459839] [] lustre_mount+0x38/0x60 [obdclass] [ 1322.467325] [] mount_fs+0x3e/0x1b0 [ 1322.473456] [] vfs_kern_mount+0x67/0x120 [ 1322.480168] [] do_mount+0x1ef/0xd50 [ 1322.486399] [] ? kmem_cache_alloc_trace+0x4f/0x250 [ 1322.494083] [] SyS_mount+0x83/0xd0 [ 1322.500214] [] system_call_fastpath+0x25/0x2a [ 1324.886999] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1324.942997] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1324.896028] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1324.942998] Modules linked in: [ 1324.942999] osp(OE) [ 1324.942999] mdd(OE) [ 1324.943000] lod(OE) [ 1324.943000] mdt(OE) [ 1324.943001] lfsck(OE) [ 1324.943001] mgc(OE) [ 1324.943002] osd_zfs(OE) [ 1324.943002] lquota(OE) [ 1324.943002] fid(OE) [ 1324.943003] fld(OE) [ 1324.943003] ptlrpc(OE) [ 1324.943004] obdclass(OE) [ 1324.943004] ko2iblnd(OE) [ 1324.943005] lnet(OE) [ 1324.943005] libcfs(OE) [ 1324.943005] iTCO_wdt [ 1324.943006] iTCO_vendor_support [ 1324.943006] rpcrdma [ 1324.943007] ib_iser [ 1324.943007] sb_edac [ 1324.943007] intel_powerclamp [ 1324.943008] coretemp [ 1324.943008] intel_rapl [ 1324.943009] iosf_mbi [ 1324.943009] kvm [ 1324.943009] irqbypass [ 1324.943010] pcspkr [ 1324.943010] joydev [ 1324.943011] lpc_ich [ 1324.943011] i2c_i801 [ 1324.943011] ioatdma [ 1324.943012] ses [ 1324.943012] enclosure [ 1324.943012] sg [ 1324.943013] ipmi_si [ 1324.943013] ipmi_devintf [ 1324.943014] ipmi_msghandler [ 1324.943014] dm_round_robin [ 1324.943015] acpi_cpufreq [ 1324.943015] acpi_power_meter [ 1324.943016] ib_ipoib [ 1324.943016] rdma_ucm [ 1324.943017] ib_umad [ 1324.943017] sch_fq_codel [ 1324.943018] iw_cxgb4 [ 1324.943018] rdma_cm [ 1324.943018] iw_cm [ 1324.943019] ib_cm [ 1324.943019] iw_cxgb3 [ 1324.943020] zfs(POE) [ 1324.943020] binfmt_misc [ 1324.943021] zunicode(POE) [ 1324.943021] zavl(POE) [ 1324.943022] icp(POE) [ 1324.943023] zcommon(POE) [ 1324.943023] znvpair(POE) [ 1324.943024] spl(OE) [ 1324.943024] msr_safe(OE) [ 1324.943025] ip_tables [ 1324.943026] nfsv3 [ 1324.943026] nfs_acl [ 1324.943027] rpcsec_gss_krb5 [ 1324.943027] auth_rpcgss [ 1324.943028] nfsv4 [ 1324.943028] dns_resolver [ 1324.943029] nfs [ 1324.943029] lockd [ 1324.943030] grace [ 1324.943030] fscache [ 1324.943031] overlay(T) [ 1324.943031] ext4 [ 1324.943032] mbcache [ 1324.943032] jbd2 [ 1324.943033] dm_service_time [ 1324.943034] sd_mod [ 1324.943034] crc_t10dif [ 1324.943035] crct10dif_generic [ 1324.943035] mlx5_ib [ 1324.943036] ib_uverbs [ 1324.943036] ib_core [ 1324.943037] be2iscsi [ 1324.943037] bnx2i [ 1324.943038] cnic [ 1324.943038] uio [ 1324.943038] cxgb4i [ 1324.943039] cxgb4 [ 1324.943039] cxgb3i [ 1324.943040] cxgb3 [ 1324.943040] mdio [ 1324.943041] libcxgbi [ 1324.943042] libcxgb [ 1324.943042] qla4xxx [ 1324.943043] iscsi_boot_sysfs [ 1324.943043] 8021q [ 1324.943044] garp [ 1324.943044] mrp [ 1324.943045] stp [ 1324.943045] llc [ 1324.943046] mgag200 [ 1324.943046] i2c_algo_bit [ 1324.943046] drm_kms_helper [ 1324.943047] syscopyarea [ 1324.943048] sysfillrect [ 1324.943048] sysimgblt [ 1324.943049] crct10dif_pclmul [ 1324.943049] fb_sys_fops [ 1324.943050] crct10dif_common [ 1324.943050] ttm [ 1324.943051] crc32_pclmul [ 1324.943051] crc32c_intel [ 1324.943052] mlx5_core [ 1324.943052] ahci [ 1324.943053] ghash_clmulni_intel [ 1324.943053] mxm_wmi [ 1324.943054] aesni_intel [ 1324.943054] mlxfw [ 1324.943055] drm [ 1324.943056] ixgbe(OE) [ 1324.943056] mpt3sas [ 1324.943057] lrw [ 1324.943057] devlink [ 1324.943058] libahci [ 1324.943058] gf128mul [ 1324.943059] dca [ 1324.943059] glue_helper [ 1324.943059] ablk_helper [ 1324.943060] ptp [ 1324.943060] raid_class [ 1324.943061] libata [ 1324.943061] drm_panel_orientation_quirks [ 1324.943062] cryptd [ 1324.943062] scsi_transport_sas [ 1324.943062] pps_core [ 1324.943063] dm_multipath [ 1324.943063] wmi [ 1324.943064] sunrpc [ 1324.943064] dm_mirror [ 1324.943065] dm_region_hash [ 1324.943065] dm_log [ 1324.943065] dm_mod [ 1324.943066] iscsi_tcp [ 1324.943066] libiscsi_tcp [ 1324.943067] libiscsi [ 1324.943067] scsi_transport_iscsi [ 1324.943068] fuse [ 1324.943068] [ 1324.943071] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1324.943072] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1324.943073] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1324.943074] RIP: 0010:[] [ 1324.943077] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1324.943078] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1324.943079] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1324.943080] RDX: ffff8892ff49b8c0 RSI: 0000000000d10000 RDI: ffff8852f5420480 [ 1324.943081] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1324.943082] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1324.943083] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1324.943084] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1324.943085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1324.943086] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1324.943087] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1324.943087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1324.943088] Call Trace: [ 1324.943091] [] queued_spin_lock_slowpath+0xb/0xf [ 1324.943094] [] _raw_spin_lock+0x30/0x40 [ 1324.943102] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1324.943117] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1324.943119] [] ? wake_up_atomic_t+0x30/0x30 [ 1324.943127] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1324.943169] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1324.943193] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1324.943218] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1324.943242] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1324.943264] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1324.943300] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1324.943328] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1324.943331] [] ? __getnstimeofday64+0x3f/0xd0 [ 1324.943357] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1324.943381] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1324.943383] [] ? __wake_up_common_lock+0x91/0xc0 [ 1324.943385] [] ? sched_feat_set+0xf0/0xf0 [ 1324.943409] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1324.943411] [] ? __switch_to+0xce/0x5a0 [ 1324.943435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1324.943438] [] kthread+0xd1/0xe0 [ 1324.943440] [] ? insert_kthread_work+0x40/0x40 [ 1324.943442] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1324.943443] [] ? insert_kthread_work+0x40/0x40 [ 1324.943444] Code: [ 1324.943444] 0d [ 1324.943445] 48 [ 1324.943445] 98 [ 1324.943446] 83 [ 1324.943446] e2 [ 1324.943446] 30 [ 1324.943447] 48 [ 1324.943447] 81 [ 1324.943447] c2 [ 1324.943448] c0 [ 1324.943448] b8 [ 1324.943448] 01 [ 1324.943449] 00 [ 1324.943449] 48 [ 1324.943450] 03 [ 1324.943450] 14 [ 1324.943450] c5 [ 1324.943451] e0 [ 1324.943451] 17 [ 1324.943451] d5 [ 1324.943452] 93 [ 1324.943452] 4c [ 1324.943453] 89 [ 1324.943453] 02 [ 1324.943453] 41 [ 1324.943454] 8b [ 1324.943454] 40 [ 1324.943454] 08 [ 1324.943455] 85 [ 1324.943455] c0 [ 1324.943456] 75 [ 1324.943456] 0f [ 1324.943456] 0f [ 1324.943457] 1f [ 1324.943457] 44 [ 1324.943457] 00 [ 1324.943458] 00 [ 1324.943458] f3 [ 1324.943459] 90 [ 1324.943459] 41 [ 1324.943459] 8b [ 1324.943460] 40 [ 1324.943460] 08 [ 1324.943461] <85> [ 1324.943461] c0 [ 1324.943461] 74 [ 1324.943462] f6 [ 1324.943462] 4d [ 1324.943462] 8b [ 1324.943463] 08 [ 1324.943463] 4d [ 1324.943463] 85 [ 1324.943464] c9 [ 1324.943464] 74 [ 1324.943465] 04 [ 1324.943465] 41 [ 1324.943465] 0f [ 1324.943466] 18 [ 1324.943466] 09 [ 1324.943466] 8b [ 1324.943467] 17 [ 1324.943467] 0f [ 1324.943467] b7 [ 1324.943468] c2 [ 1324.943468] [ 1325.446495] rdma_cm [ 1325.449425] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1325.528539] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1325.541486] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1325.555748] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1325.568073] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1325.576908] RIP: 0010:[] [] _raw_spin_lock+0x20/0x40 [ 1325.586529] RSP: 0018:ffff88528ac83ad8 EFLAGS: 00000246 [ 1325.592941] RAX: 0000000000000000 RBX: ffff8892ff2db8c0 RCX: 0000000000590000 [ 1325.601388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8892fdfb3340 [ 1325.609833] RBP: ffff88528ac83af8 R08: ffff8892ff2db8c0 R09: ffff8892ff49b8c0 [ 1325.618279] R10: 0000000000000001 R11: 0000000000000005 R12: 0000000000000001 [ 1325.626726] R13: 0000000000000005 R14: ffff8892ff2c0000 R15: ffff88528ac83ab0 [ 1325.635172] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1325.644686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1325.651581] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1325.660027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1325.668475] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1325.676921] Call Trace: [ 1325.680138] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1325.688011] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1325.696459] [] ? wake_up_atomic_t+0x30/0x30 [ 1325.703459] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1325.710864] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1325.719140] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1325.727126] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1325.735790] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1325.743679] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1325.750991] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1325.759175] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1325.768106] [] ? __getnstimeofday64+0x3f/0xd0 [ 1325.775316] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1325.784368] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1325.792427] [] ? __wake_up_common_lock+0x91/0xc0 [ 1325.799905] [] ? sched_feat_set+0xf0/0xf0 [ 1325.806726] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1325.814205] [] ? __switch_to+0xce/0x5a0 [ 1325.820832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1325.829572] [] kthread+0xd1/0xe0 [ 1325.835500] [] ? insert_kthread_work+0x40/0x40 [ 1325.842784] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1325.850551] [] ? insert_kthread_work+0x40/0x40 [ 1325.857833] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 b8 0e 01 00 83 80 44 c0 ff ff 01 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 db 0f ff ff 5d c3 0f 1f 40 [ 1332.780829] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [mdt00_011:4259] [ 1332.789762] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1332.869619] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1332.929894] CPU: 3 PID: 4259 Comm: mdt00_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1332.944060] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1332.956384] task: ffff8892c975b180 ti: ffff8892f75f0000 task.ti: ffff8892f75f0000 [ 1332.965219] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1332.976780] RSP: 0018:ffff8892f75f3ab0 EFLAGS: 00000246 [ 1332.983190] RAX: 0000000000000000 RBX: ffff8852ffac0000 RCX: 0000000000190000 [ 1332.991636] RDX: ffff8892ff39b8c0 RSI: 0000000000710000 RDI: ffff8852f5420480 [ 1333.000082] RBP: ffff8892f75f3ab0 R08: ffff8852ffadb8c0 R09: 0000000000000000 [ 1333.008528] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f75f3a78 [ 1333.016973] R13: ffffffff930ec9b1 R14: ffff8892f75f3a58 R15: ffff8892f75f3a88 [ 1333.025419] FS: 0000000000000000(0000) GS:ffff8852ffac0000(0000) knlGS:0000000000000000 [ 1333.034932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1333.041827] CR2: 00007ffff1096160 CR3: 0000007f7e794000 CR4: 00000000003607e0 [ 1333.050273] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1333.058719] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1333.067166] Call Trace: [ 1333.070378] [] queued_spin_lock_slowpath+0xb/0xf [ 1333.077856] [] _raw_spin_lock+0x30/0x40 [ 1333.084469] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1333.092142] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1333.100115] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1333.108659] [] ? wake_up_atomic_t+0x30/0x30 [ 1333.115656] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1333.123068] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1333.131338] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1333.139318] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1333.147974] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1333.155854] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1333.163163] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1333.171338] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1333.180270] [] ? __getnstimeofday64+0x3f/0xd0 [ 1333.187475] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1333.196521] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1333.204580] [] ? __wake_up_common_lock+0x91/0xc0 [ 1333.212057] [] ? sched_feat_set+0xf0/0xf0 [ 1333.218874] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1333.226354] [] ? __switch_to+0xce/0x5a0 [ 1333.232975] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1333.241714] [] kthread+0xd1/0xe0 [ 1333.247641] [] ? insert_kthread_work+0x40/0x40 [ 1333.254925] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1333.262692] [] ? insert_kthread_work+0x40/0x40 [ 1333.269974] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1340.877650] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1340.886585] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1340.966431] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1341.026723] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1341.040889] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1341.053214] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1341.062048] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1341.073610] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000246 [ 1341.080020] RAX: 0000000000000000 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1341.088467] RDX: ffff8852ffb9b8c0 RSI: 0000000000310001 RDI: ffff8852f5420480 [ 1341.096913] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: 0000000000000000 [ 1341.105359] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1341.113806] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1341.122253] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1341.131765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1341.138660] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1341.147106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1341.155551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1341.163997] Call Trace: [ 1341.167210] [] queued_spin_lock_slowpath+0xb/0xf [ 1341.174688] [] _raw_spin_lock+0x30/0x40 [ 1341.181302] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1341.188981] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1341.197430] [] ? wake_up_atomic_t+0x30/0x30 [ 1341.204426] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1341.211848] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1341.220118] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1341.228098] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1341.236755] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1341.244637] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1341.251948] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1341.260126] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1341.269061] [] ? __getnstimeofday64+0x3f/0xd0 [ 1341.276265] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1341.285311] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1341.293372] [] ? __wake_up_common_lock+0x91/0xc0 [ 1341.300849] [] ? sched_feat_set+0xf0/0xf0 [ 1341.307667] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1341.315146] [] ? __switch_to+0xce/0x5a0 [ 1341.321770] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1341.330509] [] kthread+0xd1/0xe0 [ 1341.336435] [] ? insert_kthread_work+0x40/0x40 [ 1341.343720] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1341.351488] [] ? insert_kthread_work+0x40/0x40 [ 1341.358772] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [ 1344.792562] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mdt00_009:4257] [ 1344.801494] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1344.934559] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mdt01_002:2849] [ 1344.881270] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1344.934560] Modules linked in: [ 1344.934561] osp(OE) [ 1344.934561] mdd(OE) [ 1344.934562] lod(OE) [ 1344.934562] mdt(OE) [ 1344.934562] lfsck(OE) [ 1344.934563] mgc(OE) [ 1344.934563] osd_zfs(OE) [ 1344.934563] lquota(OE) [ 1344.934563] fid(OE) [ 1344.934564] fld(OE) [ 1344.934564] ptlrpc(OE) [ 1344.934564] obdclass(OE) [ 1344.934565] ko2iblnd(OE) [ 1344.934565] lnet(OE) [ 1344.934565] libcfs(OE) [ 1344.934566] iTCO_wdt [ 1344.934566] iTCO_vendor_support [ 1344.934566] rpcrdma [ 1344.934567] ib_iser [ 1344.934567] sb_edac [ 1344.934567] intel_powerclamp [ 1344.934568] coretemp [ 1344.934568] intel_rapl [ 1344.934568] iosf_mbi [ 1344.934569] kvm [ 1344.934569] irqbypass [ 1344.934569] pcspkr [ 1344.934569] joydev [ 1344.934570] lpc_ich [ 1344.934570] i2c_i801 [ 1344.934570] ioatdma [ 1344.934571] ses [ 1344.934571] enclosure [ 1344.934571] sg [ 1344.934572] ipmi_si [ 1344.934572] ipmi_devintf [ 1344.934572] ipmi_msghandler [ 1344.934573] dm_round_robin [ 1344.934573] acpi_cpufreq [ 1344.934573] acpi_power_meter [ 1344.934573] ib_ipoib [ 1344.934574] rdma_ucm [ 1344.934574] ib_umad [ 1344.934575] sch_fq_codel [ 1344.934575] iw_cxgb4 [ 1344.934575] rdma_cm [ 1344.934576] iw_cm [ 1344.934576] ib_cm [ 1344.934576] iw_cxgb3 [ 1344.934577] zfs(POE) [ 1344.934577] binfmt_misc [ 1344.934578] zunicode(POE) [ 1344.934578] zavl(POE) [ 1344.934578] icp(POE) [ 1344.934579] zcommon(POE) [ 1344.934579] znvpair(POE) [ 1344.934579] spl(OE) [ 1344.934580] msr_safe(OE) [ 1344.934580] ip_tables [ 1344.934581] nfsv3 [ 1344.934581] nfs_acl [ 1344.934582] rpcsec_gss_krb5 [ 1344.934582] auth_rpcgss [ 1344.934582] nfsv4 [ 1344.934582] dns_resolver [ 1344.934583] nfs [ 1344.934583] lockd [ 1344.934583] grace [ 1344.934584] fscache [ 1344.934584] overlay(T) [ 1344.934585] ext4 [ 1344.934585] mbcache [ 1344.934585] jbd2 [ 1344.934586] dm_service_time [ 1344.934586] sd_mod [ 1344.934586] crc_t10dif [ 1344.934587] crct10dif_generic [ 1344.934587] mlx5_ib [ 1344.934587] ib_uverbs [ 1344.934588] ib_core [ 1344.934588] be2iscsi [ 1344.934588] bnx2i [ 1344.934589] cnic [ 1344.934589] uio [ 1344.934589] cxgb4i [ 1344.934589] cxgb4 [ 1344.934590] cxgb3i [ 1344.934590] cxgb3 [ 1344.934590] mdio [ 1344.934591] libcxgbi [ 1344.934591] libcxgb [ 1344.934591] qla4xxx [ 1344.934592] iscsi_boot_sysfs [ 1344.934592] 8021q [ 1344.934592] garp [ 1344.934592] mrp [ 1344.934593] stp [ 1344.934593] llc [ 1344.934593] mgag200 [ 1344.934594] i2c_algo_bit [ 1344.934594] drm_kms_helper [ 1344.934594] syscopyarea [ 1344.934595] sysfillrect [ 1344.934595] sysimgblt [ 1344.934595] crct10dif_pclmul [ 1344.934596] fb_sys_fops [ 1344.934596] crct10dif_common [ 1344.934596] ttm [ 1344.934597] crc32_pclmul [ 1344.934597] crc32c_intel [ 1344.934598] mlx5_core [ 1344.934598] ahci [ 1344.934598] ghash_clmulni_intel [ 1344.934598] mxm_wmi [ 1344.934599] aesni_intel [ 1344.934599] mlxfw [ 1344.934600] drm [ 1344.934600] ixgbe(OE) [ 1344.934600] mpt3sas [ 1344.934601] lrw [ 1344.934601] devlink [ 1344.934601] libahci [ 1344.934602] gf128mul [ 1344.934602] dca [ 1344.934602] glue_helper [ 1344.934603] ablk_helper [ 1344.934603] ptp [ 1344.934603] raid_class [ 1344.934603] libata [ 1344.934604] drm_panel_orientation_quirks [ 1344.934604] cryptd [ 1344.934605] scsi_transport_sas [ 1344.934605] pps_core [ 1344.934605] dm_multipath [ 1344.934606] wmi [ 1344.934606] sunrpc [ 1344.934607] dm_mirror [ 1344.934607] dm_region_hash [ 1344.934607] dm_log [ 1344.934607] dm_mod [ 1344.934608] iscsi_tcp [ 1344.934608] libiscsi_tcp [ 1344.934608] libiscsi [ 1344.934609] scsi_transport_iscsi [ 1344.934609] fuse [ 1344.934609] [ 1344.934611] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1344.934612] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1344.934613] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1344.934613] RIP: 0010:[] [ 1344.934616] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1344.934617] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1344.934618] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1344.934618] RDX: ffff8892ff59b8c0 RSI: 0000000000f10000 RDI: ffff8852f5420480 [ 1344.934619] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1344.934619] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1344.934620] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1344.934621] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1344.934622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1344.934623] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1344.934623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1344.934624] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1344.934624] Call Trace: [ 1344.934627] [] queued_spin_lock_slowpath+0xb/0xf [ 1344.934628] [] _raw_spin_lock+0x30/0x40 [ 1344.934637] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1344.934645] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1344.934647] [] ? wake_up_atomic_t+0x30/0x30 [ 1344.934653] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1344.934685] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1344.934703] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1344.934722] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1344.934739] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1344.934756] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1344.934781] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1344.934803] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1344.934804] [] ? __getnstimeofday64+0x3f/0xd0 [ 1344.934823] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1344.934842] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1344.934843] [] ? __wake_up_common_lock+0x91/0xc0 [ 1344.934844] [] ? sched_feat_set+0xf0/0xf0 [ 1344.934863] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1344.934864] [] ? __switch_to+0xce/0x5a0 [ 1344.934882] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1344.934883] [] kthread+0xd1/0xe0 [ 1344.934885] [] ? insert_kthread_work+0x40/0x40 [ 1344.934886] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1344.934887] [] ? insert_kthread_work+0x40/0x40 [ 1344.934888] Code: [ 1344.934888] c1 [ 1344.934889] e8 [ 1344.934889] 13 [ 1344.934889] 48 [ 1344.934889] c1 [ 1344.934890] ea [ 1344.934890] 0d [ 1344.934890] 48 [ 1344.934890] 98 [ 1344.934891] 83 [ 1344.934891] e2 [ 1344.934891] 30 [ 1344.934891] 48 [ 1344.934892] 81 [ 1344.934892] c2 [ 1344.934892] c0 [ 1344.934892] b8 [ 1344.934893] 01 [ 1344.934893] 00 [ 1344.934893] 48 [ 1344.934894] 03 [ 1344.934894] 14 [ 1344.934894] c5 [ 1344.934894] e0 [ 1344.934895] 17 [ 1344.934895] d5 [ 1344.934895] 93 [ 1344.934895] 4c [ 1344.934896] 89 [ 1344.934896] 02 [ 1344.934896] 41 [ 1344.934896] 8b [ 1344.934897] 40 [ 1344.934897] 08 [ 1344.934897] 85 [ 1344.934897] c0 [ 1344.934898] 75 [ 1344.934898] 0f [ 1344.934898] 0f [ 1344.934899] 1f [ 1344.934899] 44 [ 1344.934899] 00 [ 1344.934899] 00 [ 1344.934900] [ 1344.934900] 90 [ 1344.934900] 41 [ 1344.934901] 8b [ 1344.934901] 40 [ 1344.934901] 08 [ 1344.934901] 85 [ 1344.934902] c0 [ 1344.934902] 74 [ 1344.934902] f6 [ 1344.934902] 4d [ 1344.934903] 8b [ 1344.934903] 08 [ 1344.934903] 4d [ 1344.934903] 85 [ 1344.934904] c9 [ 1344.934904] 74 [ 1344.934904] 04 [ 1344.934904] 41 [ 1344.934905] 0f [ 1344.934905] 18 [ 1344.934905] [ 1345.437922] dm_mod [ 1345.440754] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1345.446631] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1345.460796] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1345.473120] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1345.481952] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1345.493513] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000246 [ 1345.499922] RAX: 0000000000000000 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1345.508366] RDX: ffff8892ff55b8c0 RSI: 0000000000e90000 RDI: ffff8852f5420480 [ 1345.516812] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: 0000000000000000 [ 1345.525257] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1345.533703] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1345.542148] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1345.551662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1345.558557] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1345.567004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1345.575450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1345.583895] Call Trace: [ 1345.587108] [] queued_spin_lock_slowpath+0xb/0xf [ 1345.594586] [] _raw_spin_lock+0x30/0x40 [ 1345.601199] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1345.608881] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1345.617330] [] ? wake_up_atomic_t+0x30/0x30 [ 1345.624327] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1345.631736] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1345.640011] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1345.647993] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1345.656651] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1345.664534] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1345.671847] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1345.680014] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1345.688945] [] ? __getnstimeofday64+0x3f/0xd0 [ 1345.696151] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1345.705196] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1345.713255] [] ? __wake_up_common_lock+0x91/0xc0 [ 1345.720733] [] ? sched_feat_set+0xf0/0xf0 [ 1345.727551] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1345.735030] [] ? __switch_to+0xce/0x5a0 [ 1345.741652] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1345.750389] [] kthread+0xd1/0xe0 [ 1345.756315] [] ? insert_kthread_work+0x40/0x40 [ 1345.763599] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1345.771366] [] ? insert_kthread_work+0x40/0x40 [ 1345.778649] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [ 1350.754467] LNet: 30707:0:(o2iblnd_cb.c:3432:kiblnd_check_conns()) Timed out tx for 172.19.3.103@o2ib600: 0 seconds [ 1351.597417] LNet: Service thread pid 4257 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1351.616727] Pid: 4257, comm: mdt00_009 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 1351.628280] Call Trace: [ 1351.631507] [] 0xffffffffffffffff [ 1351.637570] LustreError: dumping log to /tmp/lustre-log.1644346566.4257 [ 1352.886377] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1352.942375] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 23s! [mdt01_013:4292] [ 1352.895408] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1352.942376] Modules linked in: [ 1352.942376] osp(OE) [ 1352.942377] mdd(OE) [ 1352.942377] lod(OE) [ 1352.942378] mdt(OE) [ 1352.942378] lfsck(OE) [ 1352.942379] mgc(OE) [ 1352.942379] osd_zfs(OE) [ 1352.942380] lquota(OE) [ 1352.942380] fid(OE) [ 1352.942380] fld(OE) [ 1352.942380] ptlrpc(OE) [ 1352.942381] obdclass(OE) [ 1352.942381] ko2iblnd(OE) [ 1352.942382] lnet(OE) [ 1352.942382] libcfs(OE) [ 1352.942382] iTCO_wdt [ 1352.942383] iTCO_vendor_support [ 1352.942383] rpcrdma [ 1352.942383] ib_iser [ 1352.942383] sb_edac [ 1352.942384] intel_powerclamp [ 1352.942384] coretemp [ 1352.942384] intel_rapl [ 1352.942385] iosf_mbi [ 1352.942385] kvm [ 1352.942385] irqbypass [ 1352.942385] pcspkr [ 1352.942386] joydev [ 1352.942386] lpc_ich [ 1352.942386] i2c_i801 [ 1352.942387] ioatdma [ 1352.942387] ses [ 1352.942387] enclosure [ 1352.942388] sg [ 1352.942388] ipmi_si [ 1352.942388] ipmi_devintf [ 1352.942388] ipmi_msghandler [ 1352.942389] dm_round_robin [ 1352.942389] acpi_cpufreq [ 1352.942389] acpi_power_meter [ 1352.942390] ib_ipoib [ 1352.942390] rdma_ucm [ 1352.942390] ib_umad [ 1352.942391] sch_fq_codel [ 1352.942391] iw_cxgb4 [ 1352.942391] rdma_cm [ 1352.942392] iw_cm [ 1352.942392] ib_cm [ 1352.942392] iw_cxgb3 [ 1352.942393] zfs(POE) [ 1352.942393] binfmt_misc [ 1352.942394] zunicode(POE) [ 1352.942394] zavl(POE) [ 1352.942395] icp(POE) [ 1352.942395] zcommon(POE) [ 1352.942396] znvpair(POE) [ 1352.942396] spl(OE) [ 1352.942397] msr_safe(OE) [ 1352.942397] ip_tables [ 1352.942398] nfsv3 [ 1352.942398] nfs_acl [ 1352.942399] rpcsec_gss_krb5 [ 1352.942399] auth_rpcgss [ 1352.942399] nfsv4 [ 1352.942400] dns_resolver [ 1352.942400] nfs [ 1352.942401] lockd [ 1352.942401] grace [ 1352.942401] fscache [ 1352.942402] overlay(T) [ 1352.942402] ext4 [ 1352.942403] mbcache [ 1352.942403] jbd2 [ 1352.942404] dm_service_time [ 1352.942404] sd_mod [ 1352.942404] crc_t10dif [ 1352.942405] crct10dif_generic [ 1352.942405] mlx5_ib [ 1352.942405] ib_uverbs [ 1352.942406] ib_core [ 1352.942406] be2iscsi [ 1352.942407] bnx2i [ 1352.942407] cnic [ 1352.942408] uio [ 1352.942408] cxgb4i [ 1352.942408] cxgb4 [ 1352.942409] cxgb3i [ 1352.942409] cxgb3 [ 1352.942410] mdio [ 1352.942410] libcxgbi [ 1352.942411] libcxgb [ 1352.942411] qla4xxx [ 1352.942412] iscsi_boot_sysfs [ 1352.942412] 8021q [ 1352.942413] garp [ 1352.942413] mrp [ 1352.942413] stp [ 1352.942414] llc [ 1352.942414] mgag200 [ 1352.942414] i2c_algo_bit [ 1352.942415] drm_kms_helper [ 1352.942415] syscopyarea [ 1352.942416] sysfillrect [ 1352.942416] sysimgblt [ 1352.942417] crct10dif_pclmul [ 1352.942417] fb_sys_fops [ 1352.942417] crct10dif_common [ 1352.942418] ttm [ 1352.942418] crc32_pclmul [ 1352.942419] crc32c_intel [ 1352.942419] mlx5_core [ 1352.942419] ahci [ 1352.942420] ghash_clmulni_intel [ 1352.942420] mxm_wmi [ 1352.942421] aesni_intel [ 1352.942421] mlxfw [ 1352.942422] drm [ 1352.942422] ixgbe(OE) [ 1352.942422] mpt3sas [ 1352.942423] lrw [ 1352.942423] devlink [ 1352.942424] libahci [ 1352.942424] gf128mul [ 1352.942425] dca [ 1352.942425] glue_helper [ 1352.942426] ablk_helper [ 1352.942426] ptp [ 1352.942426] raid_class [ 1352.942427] libata [ 1352.942427] drm_panel_orientation_quirks [ 1352.942427] cryptd [ 1352.942428] scsi_transport_sas [ 1352.942428] pps_core [ 1352.942428] dm_multipath [ 1352.942429] wmi [ 1352.942429] sunrpc [ 1352.942429] dm_mirror [ 1352.942430] dm_region_hash [ 1352.942430] dm_log [ 1352.942431] dm_mod [ 1352.942431] iscsi_tcp [ 1352.942431] libiscsi_tcp [ 1352.942432] libiscsi [ 1352.942432] scsi_transport_iscsi [ 1352.942432] fuse [ 1352.942433] [ 1352.942435] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1352.942435] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1352.942437] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1352.942437] RIP: 0010:[] [ 1352.942441] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1352.942442] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1352.942443] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1352.942443] RDX: ffff8892ff49b8c0 RSI: 0000000000d10000 RDI: ffff8852f5420480 [ 1352.942444] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1352.942444] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1352.942445] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1352.942446] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1352.942447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1352.942448] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1352.942448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1352.942449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1352.942449] Call Trace: [ 1352.942454] [] queued_spin_lock_slowpath+0xb/0xf [ 1352.942457] [] _raw_spin_lock+0x30/0x40 [ 1352.942468] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1352.942479] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1352.942481] [] ? wake_up_atomic_t+0x30/0x30 [ 1352.942486] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1352.942528] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1352.942547] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1352.942567] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1352.942584] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1352.942601] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1352.942630] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1352.942651] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1352.942652] [] ? __getnstimeofday64+0x3f/0xd0 [ 1352.942672] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1352.942690] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1352.942692] [] ? __wake_up_common_lock+0x91/0xc0 [ 1352.942693] [] ? sched_feat_set+0xf0/0xf0 [ 1352.942711] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1352.942713] [] ? __switch_to+0xce/0x5a0 [ 1352.942731] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1352.942734] [] kthread+0xd1/0xe0 [ 1352.942735] [] ? insert_kthread_work+0x40/0x40 [ 1352.942738] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1352.942739] [] ? insert_kthread_work+0x40/0x40 [ 1352.942740] Code: [ 1352.942740] 0d [ 1352.942741] 48 [ 1352.942741] 98 [ 1352.942741] 83 [ 1352.942741] e2 [ 1352.942742] 30 [ 1352.942742] 48 [ 1352.942742] 81 [ 1352.942743] c2 [ 1352.942743] c0 [ 1352.942743] b8 [ 1352.942743] 01 [ 1352.942744] 00 [ 1352.942744] 48 [ 1352.942744] 03 [ 1352.942744] 14 [ 1352.942745] c5 [ 1352.942745] e0 [ 1352.942745] 17 [ 1352.942745] d5 [ 1352.942746] 93 [ 1352.942746] 4c [ 1352.942746] 89 [ 1352.942746] 02 [ 1352.942747] 41 [ 1352.942747] 8b [ 1352.942747] 40 [ 1352.942748] 08 [ 1352.942748] 85 [ 1352.942748] c0 [ 1352.942748] 75 [ 1352.942749] 0f [ 1352.942749] 0f [ 1352.942749] 1f [ 1352.942749] 44 [ 1352.942750] 00 [ 1352.942750] 00 [ 1352.942750] f3 [ 1352.942750] 90 [ 1352.942751] 41 [ 1352.942751] 8b [ 1352.942751] 40 [ 1352.942751] 08 [ 1352.942752] <85> [ 1352.942752] c0 [ 1352.942752] 74 [ 1352.942752] f6 [ 1352.942753] 4d [ 1352.942753] 8b [ 1352.942753] 08 [ 1352.942753] 4d [ 1352.942754] 85 [ 1352.942754] c9 [ 1352.942754] 74 [ 1352.942754] 04 [ 1352.942755] 41 [ 1352.942755] 0f [ 1352.942755] 18 [ 1352.942755] 09 [ 1352.942756] 8b [ 1352.942756] 17 [ 1352.942756] 0f [ 1352.942756] b7 [ 1352.942756] c2 [ 1352.942757] [ 1353.445891] rdma_cm [ 1353.448820] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1353.527924] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1353.540863] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1353.555127] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1353.567453] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1353.576288] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1353.587849] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1353.594260] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1353.602707] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1353.611153] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1353.619599] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1353.628047] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1353.636492] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1353.646004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1353.652901] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1353.661348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1353.669796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1353.678242] Call Trace: [ 1353.681455] [] queued_spin_lock_slowpath+0xb/0xf [ 1353.688934] [] _raw_spin_lock+0x30/0x40 [ 1353.695545] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1353.703221] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1353.711670] [] ? wake_up_atomic_t+0x30/0x30 [ 1353.718669] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1353.726070] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1353.734340] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1353.742322] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1353.750981] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1353.758862] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1353.766170] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1353.774348] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1353.783281] [] ? __getnstimeofday64+0x3f/0xd0 [ 1353.790486] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1353.799534] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1353.807595] [] ? __wake_up_common_lock+0x91/0xc0 [ 1353.815072] [] ? sched_feat_set+0xf0/0xf0 [ 1353.821889] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1353.829369] [] ? __switch_to+0xce/0x5a0 [ 1353.835992] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1353.844730] [] kthread+0xd1/0xe0 [ 1353.850658] [] ? insert_kthread_work+0x40/0x40 [ 1353.857941] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1353.865710] [] ? insert_kthread_work+0x40/0x40 [ 1353.872992] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1356.900283] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mdt01_007:3041] [ 1356.944281] NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [mdt01_005:2899] [ 1356.909312] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si [ 1356.944282] Modules linked in: [ 1356.944282] osp(OE) [ 1356.944283] mdd(OE) [ 1356.944283] lod(OE) [ 1356.944283] mdt(OE) [ 1356.944284] lfsck(OE) [ 1356.944284] mgc(OE) [ 1356.944285] osd_zfs(OE) [ 1356.944285] lquota(OE) [ 1356.944286] fid(OE) [ 1356.944286] fld(OE) [ 1356.944286] ptlrpc(OE) [ 1356.944287] obdclass(OE) [ 1356.944287] ko2iblnd(OE) [ 1356.944287] lnet(OE) [ 1356.944288] libcfs(OE) [ 1356.944288] iTCO_wdt [ 1356.944288] iTCO_vendor_support [ 1356.944289] rpcrdma [ 1356.944289] ib_iser [ 1356.944289] sb_edac [ 1356.944290] intel_powerclamp [ 1356.944290] coretemp [ 1356.944290] intel_rapl [ 1356.944291] iosf_mbi [ 1356.944291] kvm [ 1356.944291] irqbypass [ 1356.944292] pcspkr [ 1356.944292] joydev [ 1356.944292] lpc_ich [ 1356.944293] i2c_i801 [ 1356.944293] ioatdma [ 1356.944293] ses [ 1356.944293] enclosure [ 1356.944294] sg [ 1356.944294] ipmi_si [ 1356.944295] ipmi_devintf [ 1356.944295] ipmi_msghandler [ 1356.944295] dm_round_robin [ 1356.944296] acpi_cpufreq [ 1356.944296] acpi_power_meter [ 1356.944296] ib_ipoib [ 1356.944297] rdma_ucm [ 1356.944297] ib_umad [ 1356.944297] sch_fq_codel [ 1356.944298] iw_cxgb4 [ 1356.944298] rdma_cm [ 1356.944298] iw_cm [ 1356.944299] ib_cm [ 1356.944299] iw_cxgb3 [ 1356.944299] zfs(POE) [ 1356.944300] binfmt_misc [ 1356.944300] zunicode(POE) [ 1356.944301] zavl(POE) [ 1356.944301] icp(POE) [ 1356.944302] zcommon(POE) [ 1356.944302] znvpair(POE) [ 1356.944302] spl(OE) [ 1356.944303] msr_safe(OE) [ 1356.944303] ip_tables [ 1356.944303] nfsv3 [ 1356.944304] nfs_acl [ 1356.944304] rpcsec_gss_krb5 [ 1356.944305] auth_rpcgss [ 1356.944305] nfsv4 [ 1356.944305] dns_resolver [ 1356.944306] nfs [ 1356.944306] lockd [ 1356.944307] grace [ 1356.944307] fscache [ 1356.944308] overlay(T) [ 1356.944308] ext4 [ 1356.944308] mbcache [ 1356.944309] jbd2 [ 1356.944309] dm_service_time [ 1356.944309] sd_mod [ 1356.944310] crc_t10dif [ 1356.944310] crct10dif_generic [ 1356.944311] mlx5_ib [ 1356.944311] ib_uverbs [ 1356.944311] ib_core [ 1356.944312] be2iscsi [ 1356.944312] bnx2i [ 1356.944312] cnic [ 1356.944313] uio [ 1356.944313] cxgb4i [ 1356.944313] cxgb4 [ 1356.944314] cxgb3i [ 1356.944314] cxgb3 [ 1356.944315] mdio [ 1356.944315] libcxgbi [ 1356.944316] libcxgb [ 1356.944316] qla4xxx [ 1356.944317] iscsi_boot_sysfs [ 1356.944317] 8021q [ 1356.944317] garp [ 1356.944318] mrp [ 1356.944318] stp [ 1356.944319] llc [ 1356.944319] mgag200 [ 1356.944319] i2c_algo_bit [ 1356.944320] drm_kms_helper [ 1356.944320] syscopyarea [ 1356.944321] sysfillrect [ 1356.944321] sysimgblt [ 1356.944322] crct10dif_pclmul [ 1356.944322] fb_sys_fops [ 1356.944322] crct10dif_common [ 1356.944323] ttm [ 1356.944323] crc32_pclmul [ 1356.944324] crc32c_intel [ 1356.944324] mlx5_core [ 1356.944325] ahci [ 1356.944325] ghash_clmulni_intel [ 1356.944325] mxm_wmi [ 1356.944326] aesni_intel [ 1356.944326] mlxfw [ 1356.944326] drm [ 1356.944327] ixgbe(OE) [ 1356.944327] mpt3sas [ 1356.944328] lrw [ 1356.944328] devlink [ 1356.944329] libahci [ 1356.944329] gf128mul [ 1356.944329] dca [ 1356.944330] glue_helper [ 1356.944330] ablk_helper [ 1356.944330] ptp [ 1356.944331] raid_class [ 1356.944331] libata [ 1356.944332] drm_panel_orientation_quirks [ 1356.944332] cryptd [ 1356.944332] scsi_transport_sas [ 1356.944333] pps_core [ 1356.944333] dm_multipath [ 1356.944334] wmi [ 1356.944334] sunrpc [ 1356.944334] dm_mirror [ 1356.944335] dm_region_hash [ 1356.944335] dm_log [ 1356.944336] dm_mod [ 1356.944336] iscsi_tcp [ 1356.944336] libiscsi_tcp [ 1356.944337] libiscsi [ 1356.944337] scsi_transport_iscsi [ 1356.944338] fuse [ 1356.944338] [ 1356.944340] CPU: 30 PID: 2899 Comm: mdt01_005 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1356.944340] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1356.944341] task: ffff88151ecc3180 ti: ffff88528ac90000 task.ti: ffff88528ac90000 [ 1356.944342] RIP: 0010:[] [ 1356.944345] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1356.944346] RSP: 0018:ffff88528ac93ab0 EFLAGS: 00000246 [ 1356.944346] RAX: 0000000000000000 RBX: ffff8892ff580000 RCX: 0000000000f10000 [ 1356.944347] RDX: ffff8852ffadb8c0 RSI: 0000000000190000 RDI: ffff8852f5420480 [ 1356.944348] RBP: ffff88528ac93ab0 R08: ffff8892ff59b8c0 R09: 0000000000000000 [ 1356.944349] R10: 0000000000000001 R11: fffff300bca63500 R12: ffff88528ac93a78 [ 1356.944350] R13: ffffffff930ec9b1 R14: ffff88528ac93a58 R15: ffff88528ac93a88 [ 1356.944351] FS: 0000000000000000(0000) GS:ffff8892ff580000(0000) knlGS:0000000000000000 [ 1356.944352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1356.944352] CR2: 00007ffff67b4dd0 CR3: 0000003f6bd52000 CR4: 00000000003607e0 [ 1356.944353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1356.944354] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1356.944354] Call Trace: [ 1356.944357] [] queued_spin_lock_slowpath+0xb/0xf [ 1356.944359] [] _raw_spin_lock+0x30/0x40 [ 1356.944368] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1356.944379] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1356.944381] [] ? wake_up_atomic_t+0x30/0x30 [ 1356.944388] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1356.944425] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1356.944447] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1356.944471] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1356.944493] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1356.944514] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1356.944545] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1356.944571] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1356.944573] [] ? __getnstimeofday64+0x3f/0xd0 [ 1356.944597] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1356.944619] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1356.944621] [] ? __wake_up_common_lock+0x91/0xc0 [ 1356.944622] [] ? sched_feat_set+0xf0/0xf0 [ 1356.944645] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1356.944647] [] ? __switch_to+0xce/0x5a0 [ 1356.944670] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1356.944671] [] kthread+0xd1/0xe0 [ 1356.944673] [] ? insert_kthread_work+0x40/0x40 [ 1356.944675] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1356.944677] [] ? insert_kthread_work+0x40/0x40 [ 1356.944677] Code: [ 1356.944678] 0d [ 1356.944678] 48 [ 1356.944679] 98 [ 1356.944679] 83 [ 1356.944679] e2 [ 1356.944679] 30 [ 1356.944680] 48 [ 1356.944680] 81 [ 1356.944680] c2 [ 1356.944681] c0 [ 1356.944681] b8 [ 1356.944681] 01 [ 1356.944682] 00 [ 1356.944682] 48 [ 1356.944682] 03 [ 1356.944683] 14 [ 1356.944683] c5 [ 1356.944683] e0 [ 1356.944684] 17 [ 1356.944684] d5 [ 1356.944684] 93 [ 1356.944685] 4c [ 1356.944686] 89 [ 1356.944686] 02 [ 1356.944687] 41 [ 1356.944687] 8b [ 1356.944687] 40 [ 1356.944688] 08 [ 1356.944688] 85 [ 1356.944688] c0 [ 1356.944688] 75 [ 1356.944689] 0f [ 1356.944689] 0f [ 1356.944690] 1f [ 1356.944690] 44 [ 1356.944690] 00 [ 1356.944690] 00 [ 1356.944691] f3 [ 1356.944691] 90 [ 1356.944691] 41 [ 1356.944692] 8b [ 1356.944692] 40 [ 1356.944692] 08 [ 1356.944693] <85> [ 1356.944693] c0 [ 1356.944693] 74 [ 1356.944694] f6 [ 1356.944694] 4d [ 1356.944694] 8b [ 1356.944694] 08 [ 1356.944695] 4d [ 1356.944695] 85 [ 1356.944695] c9 [ 1356.944695] 74 [ 1356.944696] 04 [ 1356.944696] 41 [ 1356.944696] 0f [ 1356.944697] 18 [ 1356.944697] 09 [ 1356.944697] 8b [ 1356.944697] 17 [ 1356.944698] 0f [ 1356.944698] b7 [ 1356.944698] c2 [ 1356.944699] [ 1357.447893] ipmi_devintf [ 1357.451306] ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw [ 1357.529133] devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1357.554804] CPU: 14 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1357.569069] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1357.581394] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1357.590229] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1357.601792] RSP: 0018:ffff88528ad4fab0 EFLAGS: 00000246 [ 1357.608202] RAX: 0000000000000000 RBX: ffff8892ff380000 RCX: 0000000000710000 [ 1357.616647] RDX: ffff8892ff49b8c0 RSI: 0000000000d10000 RDI: ffff8852f5420480 [ 1357.625094] RBP: ffff88528ad4fab0 R08: ffff8892ff39b8c0 R09: 0000000000000000 [ 1357.633541] R10: 0000000000000001 R11: ffff8852a386f800 R12: ffff88528ad4fa78 [ 1357.641987] R13: ffffffff930ec9b1 R14: ffff88528ad4fa58 R15: ffff88528ad4fa88 [ 1357.650433] FS: 0000000000000000(0000) GS:ffff8892ff380000(0000) knlGS:0000000000000000 [ 1357.659946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1357.666839] CR2: 00007fb699688008 CR3: 0000003f6bd52000 CR4: 00000000003607e0 [ 1357.675287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1357.683734] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1357.692180] Call Trace: [ 1357.695394] [] queued_spin_lock_slowpath+0xb/0xf [ 1357.702872] [] _raw_spin_lock+0x30/0x40 [ 1357.709483] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1357.717162] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1357.725609] [] ? wake_up_atomic_t+0x30/0x30 [ 1357.732609] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1357.740013] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1357.742273] LNet: Service thread pid 2849 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 1357.742274] Pid: 2849, comm: mdt01_002 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 1357.742275] Call Trace: [ 1357.742294] [] 0xffffffffffffffff [ 1357.742295] LustreError: dumping log to /tmp/lustre-log.1644346572.2849 [ 1357.796234] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1357.804222] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1357.812884] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1357.820773] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1357.828087] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1357.836279] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1357.845211] [] ? __getnstimeofday64+0x3f/0xd0 [ 1357.852435] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1357.861501] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1357.869560] [] ? __wake_up_common_lock+0x91/0xc0 [ 1357.877036] [] ? sched_feat_set+0xf0/0xf0 [ 1357.883871] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1357.891350] [] ? __switch_to+0xce/0x5a0 [ 1357.897993] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1357.906722] [] kthread+0xd1/0xe0 [ 1357.912648] [] ? insert_kthread_work+0x40/0x40 [ 1357.919931] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1357.927699] [] ? insert_kthread_work+0x40/0x40 [ 1357.934982] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1360.780191] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [mdt00_011:4259] [ 1360.789139] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1360.868988] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1360.929271] CPU: 3 PID: 4259 Comm: mdt00_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1360.943438] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1360.955761] task: ffff8892c975b180 ti: ffff8892f75f0000 task.ti: ffff8892f75f0000 [ 1360.964594] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1360.976155] RSP: 0018:ffff8892f75f3ab0 EFLAGS: 00000246 [ 1360.982563] RAX: 0000000000000000 RBX: ffff8852ffac0000 RCX: 0000000000190000 [ 1360.991009] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1360.999455] RBP: ffff8892f75f3ab0 R08: ffff8852ffadb8c0 R09: 0000000000000000 [ 1361.007902] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f75f3a78 [ 1361.016348] R13: ffffffff930ec9b1 R14: ffff8892f75f3a58 R15: ffff8892f75f3a88 [ 1361.024795] FS: 0000000000000000(0000) GS:ffff8852ffac0000(0000) knlGS:0000000000000000 [ 1361.034309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1361.041202] CR2: 00007ffff1096160 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1361.049649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1361.058094] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1361.066540] Call Trace: [ 1361.069754] [] queued_spin_lock_slowpath+0xb/0xf [ 1361.077232] [] _raw_spin_lock+0x30/0x40 [ 1361.083844] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1361.091531] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1361.099980] [] ? wake_up_atomic_t+0x30/0x30 [ 1361.106975] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1361.114389] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1361.122659] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1361.130638] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1361.139295] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1361.147176] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1361.154486] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1361.162663] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1361.171596] [] ? __getnstimeofday64+0x3f/0xd0 [ 1361.178801] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1361.187848] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1361.195907] [] ? __wake_up_common_lock+0x91/0xc0 [ 1361.203383] [] ? sched_feat_set+0xf0/0xf0 [ 1361.210201] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1361.217681] [] ? __switch_to+0xce/0x5a0 [ 1361.224307] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1361.233047] [] kthread+0xd1/0xe0 [ 1361.238972] [] ? insert_kthread_work+0x40/0x40 [ 1361.246257] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1361.254024] [] ? insert_kthread_work+0x40/0x40 [ 1361.261306] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1368.876997] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1368.885932] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1368.965772] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1369.026065] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1369.040231] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1369.052556] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1369.061389] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1369.072954] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000246 [ 1369.079364] RAX: 0000000000000000 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1369.087810] RDX: ffff8852ffb9b8c0 RSI: 0000000000310001 RDI: ffff8852f5420480 [ 1369.096256] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: 0000000000000000 [ 1369.104703] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1369.113149] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1369.121596] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1369.131109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1369.138003] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1369.146448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1369.154893] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1369.163340] Call Trace: [ 1369.166554] [] queued_spin_lock_slowpath+0xb/0xf [ 1369.174032] [] _raw_spin_lock+0x30/0x40 [ 1369.180645] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1369.188327] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1369.196775] [] ? wake_up_atomic_t+0x30/0x30 [ 1369.203774] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1369.211195] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1369.219466] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1369.227446] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1369.236105] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1369.243988] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1369.251299] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1369.259475] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1369.268408] [] ? __getnstimeofday64+0x3f/0xd0 [ 1369.275614] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1369.284661] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1369.292720] [] ? __wake_up_common_lock+0x91/0xc0 [ 1369.300196] [] ? sched_feat_set+0xf0/0xf0 [ 1369.307013] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1369.314492] [] ? __switch_to+0xce/0x5a0 [ 1369.321115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1369.329855] [] kthread+0xd1/0xe0 [ 1369.335781] [] ? insert_kthread_work+0x40/0x40 [ 1369.343065] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1369.350833] [] ? insert_kthread_work+0x40/0x40 [ 1369.358116] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1372.791910] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mdt00_009:4257] [ 1372.800842] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1372.933907] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mdt01_002:2849] [ 1372.880636] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1372.933908] Modules linked in: [ 1372.933908] osp(OE) [ 1372.933909] mdd(OE) [ 1372.933909] lod(OE) [ 1372.933909] mdt(OE) [ 1372.933910] lfsck(OE) [ 1372.933910] mgc(OE) [ 1372.933910] osd_zfs(OE) [ 1372.933911] lquota(OE) [ 1372.933911] fid(OE) [ 1372.933911] fld(OE) [ 1372.933911] ptlrpc(OE) [ 1372.933912] obdclass(OE) [ 1372.933912] ko2iblnd(OE) [ 1372.933912] lnet(OE) [ 1372.933912] libcfs(OE) [ 1372.933913] iTCO_wdt [ 1372.933913] iTCO_vendor_support [ 1372.933913] rpcrdma [ 1372.933914] ib_iser [ 1372.933914] sb_edac [ 1372.933914] intel_powerclamp [ 1372.933915] coretemp [ 1372.933915] intel_rapl [ 1372.933915] iosf_mbi [ 1372.933915] kvm [ 1372.933916] irqbypass [ 1372.933916] pcspkr [ 1372.933916] joydev [ 1372.933917] lpc_ich [ 1372.933917] i2c_i801 [ 1372.933917] ioatdma [ 1372.933917] ses [ 1372.933918] enclosure [ 1372.933918] sg [ 1372.933918] ipmi_si [ 1372.933918] ipmi_devintf [ 1372.933919] ipmi_msghandler [ 1372.933919] dm_round_robin [ 1372.933919] acpi_cpufreq [ 1372.933919] acpi_power_meter [ 1372.933920] ib_ipoib [ 1372.933920] rdma_ucm [ 1372.933920] ib_umad [ 1372.933920] sch_fq_codel [ 1372.933921] iw_cxgb4 [ 1372.933921] rdma_cm [ 1372.933921] iw_cm [ 1372.933921] ib_cm [ 1372.933922] iw_cxgb3 [ 1372.933922] zfs(POE) [ 1372.933922] binfmt_misc [ 1372.933923] zunicode(POE) [ 1372.933923] zavl(POE) [ 1372.933924] icp(POE) [ 1372.933924] zcommon(POE) [ 1372.933924] znvpair(POE) [ 1372.933925] spl(OE) [ 1372.933925] msr_safe(OE) [ 1372.933925] ip_tables [ 1372.933926] nfsv3 [ 1372.933926] nfs_acl [ 1372.933926] rpcsec_gss_krb5 [ 1372.933927] auth_rpcgss [ 1372.933927] nfsv4 [ 1372.933927] dns_resolver [ 1372.933927] nfs [ 1372.933928] lockd [ 1372.933928] grace [ 1372.933928] fscache [ 1372.933929] overlay(T) [ 1372.933929] ext4 [ 1372.933930] mbcache [ 1372.933930] jbd2 [ 1372.933930] dm_service_time [ 1372.933931] sd_mod [ 1372.933931] crc_t10dif [ 1372.933931] crct10dif_generic [ 1372.933931] mlx5_ib [ 1372.933932] ib_uverbs [ 1372.933932] ib_core [ 1372.933932] be2iscsi [ 1372.933933] bnx2i [ 1372.933933] cnic [ 1372.933933] uio [ 1372.933933] cxgb4i [ 1372.933934] cxgb4 [ 1372.933934] cxgb3i [ 1372.933934] cxgb3 [ 1372.933934] mdio [ 1372.933935] libcxgbi [ 1372.933935] libcxgb [ 1372.933935] qla4xxx [ 1372.933936] iscsi_boot_sysfs [ 1372.933936] 8021q [ 1372.933936] garp [ 1372.933937] mrp [ 1372.933937] stp [ 1372.933937] llc [ 1372.933938] mgag200 [ 1372.933938] i2c_algo_bit [ 1372.933938] drm_kms_helper [ 1372.933938] syscopyarea [ 1372.933939] sysfillrect [ 1372.933939] sysimgblt [ 1372.933939] crct10dif_pclmul [ 1372.933940] fb_sys_fops [ 1372.933940] crct10dif_common [ 1372.933941] ttm [ 1372.933941] crc32_pclmul [ 1372.933941] crc32c_intel [ 1372.933942] mlx5_core [ 1372.933942] ahci [ 1372.933942] ghash_clmulni_intel [ 1372.933942] mxm_wmi [ 1372.933943] aesni_intel [ 1372.933943] mlxfw [ 1372.933943] drm [ 1372.933944] ixgbe(OE) [ 1372.933944] mpt3sas [ 1372.933944] lrw [ 1372.933945] devlink [ 1372.933945] libahci [ 1372.933945] gf128mul [ 1372.933945] dca [ 1372.933946] glue_helper [ 1372.933946] ablk_helper [ 1372.933946] ptp [ 1372.933946] raid_class [ 1372.933947] libata [ 1372.933947] drm_panel_orientation_quirks [ 1372.933947] cryptd [ 1372.933948] scsi_transport_sas [ 1372.933948] pps_core [ 1372.933948] dm_multipath [ 1372.933949] wmi [ 1372.933949] sunrpc [ 1372.933949] dm_mirror [ 1372.933950] dm_region_hash [ 1372.933950] dm_log [ 1372.933950] dm_mod [ 1372.933950] iscsi_tcp [ 1372.933951] libiscsi_tcp [ 1372.933951] libiscsi [ 1372.933951] scsi_transport_iscsi [ 1372.933952] fuse [ 1372.933952] [ 1372.933953] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1372.933954] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1372.933955] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1372.933956] RIP: 0010:[] [ 1372.933958] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1372.933959] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1372.933959] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1372.933960] RDX: ffff8892ff55b8c0 RSI: 0000000000e90000 RDI: ffff8852f5420480 [ 1372.933961] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1372.933961] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1372.933962] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1372.933963] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1372.933963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1372.933964] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1372.933965] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1372.933965] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1372.933966] Call Trace: [ 1372.933968] [] queued_spin_lock_slowpath+0xb/0xf [ 1372.933970] [] _raw_spin_lock+0x30/0x40 [ 1372.933977] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1372.933986] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1372.933988] [] ? wake_up_atomic_t+0x30/0x30 [ 1372.933994] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1372.934020] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1372.934038] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1372.934057] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1372.934074] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1372.934091] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1372.934115] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1372.934136] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1372.934138] [] ? __getnstimeofday64+0x3f/0xd0 [ 1372.934157] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1372.934175] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1372.934177] [] ? __wake_up_common_lock+0x91/0xc0 [ 1372.934178] [] ? sched_feat_set+0xf0/0xf0 [ 1372.934196] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1372.934197] [] ? __switch_to+0xce/0x5a0 [ 1372.934215] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1372.934217] [] kthread+0xd1/0xe0 [ 1372.934218] [] ? insert_kthread_work+0x40/0x40 [ 1372.934220] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1372.934221] [] ? insert_kthread_work+0x40/0x40 [ 1372.934222] Code: [ 1372.934222] 0d [ 1372.934222] 48 [ 1372.934223] 98 [ 1372.934223] 83 [ 1372.934223] e2 [ 1372.934223] 30 [ 1372.934224] 48 [ 1372.934224] 81 [ 1372.934224] c2 [ 1372.934224] c0 [ 1372.934225] b8 [ 1372.934225] 01 [ 1372.934225] 00 [ 1372.934225] 48 [ 1372.934226] 03 [ 1372.934226] 14 [ 1372.934226] c5 [ 1372.934227] e0 [ 1372.934227] 17 [ 1372.934227] d5 [ 1372.934227] 93 [ 1372.934228] 4c [ 1372.934228] 89 [ 1372.934228] 02 [ 1372.934228] 41 [ 1372.934229] 8b [ 1372.934229] 40 [ 1372.934230] 08 [ 1372.934230] 85 [ 1372.934230] c0 [ 1372.934230] 75 [ 1372.934231] 0f [ 1372.934231] 0f [ 1372.934231] 1f [ 1372.934231] 44 [ 1372.934232] 00 [ 1372.934232] 00 [ 1372.934232] f3 [ 1372.934232] 90 [ 1372.934233] 41 [ 1372.934233] 8b [ 1372.934233] 40 [ 1372.934234] 08 [ 1372.934234] <85> [ 1372.934234] c0 [ 1372.934234] 74 [ 1372.934235] f6 [ 1372.934235] 4d [ 1372.934235] 8b [ 1372.934235] 08 [ 1372.934236] 4d [ 1372.934236] 85 [ 1372.934236] c9 [ 1372.934236] 74 [ 1372.934236] 04 [ 1372.934237] 41 [ 1372.934237] 0f [ 1372.934237] 18 [ 1372.934237] 09 [ 1372.934238] 8b [ 1372.934238] 17 [ 1372.934238] 0f [ 1372.934238] b7 [ 1372.934239] c2 [ 1372.934239] [ 1373.437283] dm_mod [ 1373.440115] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1373.445988] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1373.460155] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1373.472479] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1373.481313] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1373.492874] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000246 [ 1373.499284] RAX: 0000000000000000 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1373.507728] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1373.516174] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: 0000000000000000 [ 1373.524619] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1373.533065] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1373.541512] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1373.551024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1373.557920] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1373.566365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1373.574810] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1373.583256] Call Trace: [ 1373.586469] [] queued_spin_lock_slowpath+0xb/0xf [ 1373.593947] [] _raw_spin_lock+0x30/0x40 [ 1373.600559] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1373.608242] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1373.616689] [] ? wake_up_atomic_t+0x30/0x30 [ 1373.623685] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1373.631093] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1373.639364] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1373.647344] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1373.656002] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1373.663889] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1373.671198] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1373.679374] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1373.688306] [] ? __getnstimeofday64+0x3f/0xd0 [ 1373.695516] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1373.704569] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1373.712630] [] ? __wake_up_common_lock+0x91/0xc0 [ 1373.720108] [] ? sched_feat_set+0xf0/0xf0 [ 1373.726925] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1373.734402] [] ? __switch_to+0xce/0x5a0 [ 1373.741024] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1373.749763] [] kthread+0xd1/0xe0 [ 1373.755688] [] ? insert_kthread_work+0x40/0x40 [ 1373.762974] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1373.770742] [] ? insert_kthread_work+0x40/0x40 [ 1373.778023] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [ 1380.885729] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [mdt01_006:3034] [ 1380.941728] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 23s! [mdt01_013:4292] [ 1380.894760] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1380.941728] Modules linked in: [ 1380.941729] osp(OE) [ 1380.941729] mdd(OE) [ 1380.941730] lod(OE) [ 1380.941730] mdt(OE) [ 1380.941731] lfsck(OE) [ 1380.941731] mgc(OE) [ 1380.941731] osd_zfs(OE) [ 1380.941732] lquota(OE) [ 1380.941732] fid(OE) [ 1380.941732] fld(OE) [ 1380.941733] ptlrpc(OE) [ 1380.941733] obdclass(OE) [ 1380.941733] ko2iblnd(OE) [ 1380.941733] lnet(OE) [ 1380.941734] libcfs(OE) [ 1380.941734] iTCO_wdt [ 1380.941735] iTCO_vendor_support [ 1380.941735] rpcrdma [ 1380.941735] ib_iser [ 1380.941736] sb_edac [ 1380.941736] intel_powerclamp [ 1380.941736] coretemp [ 1380.941736] intel_rapl [ 1380.941737] iosf_mbi [ 1380.941737] kvm [ 1380.941738] irqbypass [ 1380.941738] pcspkr [ 1380.941738] joydev [ 1380.941738] lpc_ich [ 1380.941739] i2c_i801 [ 1380.941739] ioatdma [ 1380.941739] ses [ 1380.941740] enclosure [ 1380.941740] sg [ 1380.941740] ipmi_si [ 1380.941740] ipmi_devintf [ 1380.941741] ipmi_msghandler [ 1380.941741] dm_round_robin [ 1380.941741] acpi_cpufreq [ 1380.941742] acpi_power_meter [ 1380.941742] ib_ipoib [ 1380.941742] rdma_ucm [ 1380.941743] ib_umad [ 1380.941743] sch_fq_codel [ 1380.941743] iw_cxgb4 [ 1380.941744] rdma_cm [ 1380.941744] iw_cm [ 1380.941744] ib_cm [ 1380.941745] iw_cxgb3 [ 1380.941745] zfs(POE) [ 1380.941745] binfmt_misc [ 1380.941746] zunicode(POE) [ 1380.941747] zavl(POE) [ 1380.941747] icp(POE) [ 1380.941747] zcommon(POE) [ 1380.941748] znvpair(POE) [ 1380.941748] spl(OE) [ 1380.941748] msr_safe(OE) [ 1380.941749] ip_tables [ 1380.941749] nfsv3 [ 1380.941750] nfs_acl [ 1380.941750] rpcsec_gss_krb5 [ 1380.941750] auth_rpcgss [ 1380.941751] nfsv4 [ 1380.941751] dns_resolver [ 1380.941752] nfs [ 1380.941752] lockd [ 1380.941752] grace [ 1380.941753] fscache [ 1380.941753] overlay(T) [ 1380.941753] ext4 [ 1380.941754] mbcache [ 1380.941754] jbd2 [ 1380.941755] dm_service_time [ 1380.941755] sd_mod [ 1380.941755] crc_t10dif [ 1380.941756] crct10dif_generic [ 1380.941756] mlx5_ib [ 1380.941756] ib_uverbs [ 1380.941757] ib_core [ 1380.941757] be2iscsi [ 1380.941758] bnx2i [ 1380.941758] cnic [ 1380.941758] uio [ 1380.941758] cxgb4i [ 1380.941759] cxgb4 [ 1380.941759] cxgb3i [ 1380.941759] cxgb3 [ 1380.941760] mdio [ 1380.941760] libcxgbi [ 1380.941761] libcxgb [ 1380.941761] qla4xxx [ 1380.941761] iscsi_boot_sysfs [ 1380.941762] 8021q [ 1380.941762] garp [ 1380.941763] mrp [ 1380.941763] stp [ 1380.941763] llc [ 1380.941764] mgag200 [ 1380.941764] i2c_algo_bit [ 1380.941764] drm_kms_helper [ 1380.941765] syscopyarea [ 1380.941765] sysfillrect [ 1380.941765] sysimgblt [ 1380.941766] crct10dif_pclmul [ 1380.941766] fb_sys_fops [ 1380.941766] crct10dif_common [ 1380.941767] ttm [ 1380.941767] crc32_pclmul [ 1380.941768] crc32c_intel [ 1380.941768] mlx5_core [ 1380.941769] ahci [ 1380.941769] ghash_clmulni_intel [ 1380.941769] mxm_wmi [ 1380.941770] aesni_intel [ 1380.941770] mlxfw [ 1380.941770] drm [ 1380.941771] ixgbe(OE) [ 1380.941771] mpt3sas [ 1380.941772] lrw [ 1380.941772] devlink [ 1380.941772] libahci [ 1380.941773] gf128mul [ 1380.941773] dca [ 1380.941773] glue_helper [ 1380.941774] ablk_helper [ 1380.941774] ptp [ 1380.941775] raid_class [ 1380.941775] libata [ 1380.941776] drm_panel_orientation_quirks [ 1380.941776] cryptd [ 1380.941776] scsi_transport_sas [ 1380.941776] pps_core [ 1380.941777] dm_multipath [ 1380.941777] wmi [ 1380.941778] sunrpc [ 1380.941778] dm_mirror [ 1380.941778] dm_region_hash [ 1380.941778] dm_log [ 1380.941779] dm_mod [ 1380.941779] iscsi_tcp [ 1380.941779] libiscsi_tcp [ 1380.941780] libiscsi [ 1380.941780] scsi_transport_iscsi [ 1380.941781] fuse [ 1380.941781] [ 1380.941783] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1380.941784] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1380.941785] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1380.941785] RIP: 0010:[] [ 1380.941788] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1380.941789] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1380.941789] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1380.941790] RDX: ffff8892ff51b8c0 RSI: 0000000000e10000 RDI: ffff8852f5420480 [ 1380.941791] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1380.941791] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1380.941792] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1380.941793] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1380.941793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1380.941794] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1380.941795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1380.941796] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1380.941796] Call Trace: [ 1380.941799] [] queued_spin_lock_slowpath+0xb/0xf [ 1380.941801] [] _raw_spin_lock+0x30/0x40 [ 1380.941810] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1380.941820] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1380.941822] [] ? wake_up_atomic_t+0x30/0x30 [ 1380.941829] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1380.941864] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1380.941888] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1380.941913] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1380.941937] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1380.941960] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1380.941992] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1380.942019] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1380.942020] [] ? __getnstimeofday64+0x3f/0xd0 [ 1380.942046] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1380.942071] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1380.942074] [] ? __wake_up_common_lock+0x91/0xc0 [ 1380.942075] [] ? sched_feat_set+0xf0/0xf0 [ 1380.942100] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1380.942102] [] ? __switch_to+0xce/0x5a0 [ 1380.942127] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1380.942129] [] kthread+0xd1/0xe0 [ 1380.942131] [] ? insert_kthread_work+0x40/0x40 [ 1380.942132] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1380.942133] [] ? insert_kthread_work+0x40/0x40 [ 1380.942134] Code: [ 1380.942134] c1 [ 1380.942135] e8 [ 1380.942135] 13 [ 1380.942135] 48 [ 1380.942135] c1 [ 1380.942136] ea [ 1380.942136] 0d [ 1380.942136] 48 [ 1380.942136] 98 [ 1380.942137] 83 [ 1380.942137] e2 [ 1380.942137] 30 [ 1380.942137] 48 [ 1380.942138] 81 [ 1380.942138] c2 [ 1380.942138] c0 [ 1380.942138] b8 [ 1380.942139] 01 [ 1380.942139] 00 [ 1380.942139] 48 [ 1380.942140] 03 [ 1380.942140] 14 [ 1380.942140] c5 [ 1380.942140] e0 [ 1380.942141] 17 [ 1380.942141] d5 [ 1380.942141] 93 [ 1380.942141] 4c [ 1380.942142] 89 [ 1380.942142] 02 [ 1380.942142] 41 [ 1380.942142] 8b [ 1380.942143] 40 [ 1380.942143] 08 [ 1380.942143] 85 [ 1380.942144] c0 [ 1380.942144] 75 [ 1380.942144] 0f [ 1380.942144] 0f [ 1380.942145] 1f [ 1380.942145] 44 [ 1380.942145] 00 [ 1380.942145] 00 [ 1380.942146] [ 1380.942146] 90 [ 1380.942146] 41 [ 1380.942147] 8b [ 1380.942147] 40 [ 1380.942147] 08 [ 1380.942147] 85 [ 1380.942147] c0 [ 1380.942148] 74 [ 1380.942148] f6 [ 1380.942148] 4d [ 1380.942148] 8b [ 1380.942149] 08 [ 1380.942149] 4d [ 1380.942149] 85 [ 1380.942149] c9 [ 1380.942150] 74 [ 1380.942150] 04 [ 1380.942150] 41 [ 1380.942150] 0f [ 1380.942151] 18 [ 1380.942151] [ 1381.445227] rdma_cm [ 1381.448156] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1381.527261] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1381.540204] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1381.554469] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1381.566795] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1381.575629] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1381.587188] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1381.593599] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1381.602046] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1381.610492] RBP: ffff88528ac83ab0 R08: ffff8892ff2db8c0 R09: 0000000000000000 [ 1381.618939] R10: 0000000000000001 R11: 0000000000000005 R12: ffff88528ac83a78 [ 1381.627385] R13: ffffffff930ec9b1 R14: ffff88528ac83a58 R15: ffff88528ac83a88 [ 1381.635832] FS: 0000000000000000(0000) GS:ffff8892ff2c0000(0000) knlGS:0000000000000000 [ 1381.645346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1381.652241] CR2: 00007f37450e7504 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1381.660688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1381.669136] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1381.677581] Call Trace: [ 1381.680792] [] queued_spin_lock_slowpath+0xb/0xf [ 1381.688271] [] _raw_spin_lock+0x30/0x40 [ 1381.694880] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1381.702557] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1381.711004] [] ? wake_up_atomic_t+0x30/0x30 [ 1381.718003] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1381.725403] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1381.733673] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1381.741654] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1381.750312] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1381.758199] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1381.765507] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1381.773685] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1381.782617] [] ? __getnstimeofday64+0x3f/0xd0 [ 1381.789822] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1381.798871] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1381.806931] [] ? __wake_up_common_lock+0x91/0xc0 [ 1381.814408] [] ? sched_feat_set+0xf0/0xf0 [ 1381.821229] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1381.828708] [] ? __switch_to+0xce/0x5a0 [ 1381.835331] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1381.844070] [] kthread+0xd1/0xe0 [ 1381.849996] [] ? insert_kthread_work+0x40/0x40 [ 1381.857279] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1381.865049] [] ? insert_kthread_work+0x40/0x40 [ 1381.872332] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1384.899638] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mdt01_007:3041] [ 1384.943636] NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [mdt01_005:2899] [ 1384.908668] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si [ 1384.943637] Modules linked in: [ 1384.943637] osp(OE) [ 1384.943637] mdd(OE) [ 1384.943638] lod(OE) [ 1384.943638] mdt(OE) [ 1384.943639] lfsck(OE) [ 1384.943639] mgc(OE) [ 1384.943639] osd_zfs(OE) [ 1384.943640] lquota(OE) [ 1384.943640] fid(OE) [ 1384.943640] fld(OE) [ 1384.943641] ptlrpc(OE) [ 1384.943641] obdclass(OE) [ 1384.943642] ko2iblnd(OE) [ 1384.943642] lnet(OE) [ 1384.943642] libcfs(OE) [ 1384.943643] iTCO_wdt [ 1384.943643] iTCO_vendor_support [ 1384.943643] rpcrdma [ 1384.943644] ib_iser [ 1384.943644] sb_edac [ 1384.943644] intel_powerclamp [ 1384.943645] coretemp [ 1384.943645] intel_rapl [ 1384.943645] iosf_mbi [ 1384.943645] kvm [ 1384.943646] irqbypass [ 1384.943646] pcspkr [ 1384.943647] joydev [ 1384.943647] lpc_ich [ 1384.943647] i2c_i801 [ 1384.943647] ioatdma [ 1384.943648] ses [ 1384.943648] enclosure [ 1384.943648] sg [ 1384.943649] ipmi_si [ 1384.943649] ipmi_devintf [ 1384.943649] ipmi_msghandler [ 1384.943650] dm_round_robin [ 1384.943650] acpi_cpufreq [ 1384.943650] acpi_power_meter [ 1384.943651] ib_ipoib [ 1384.943651] rdma_ucm [ 1384.943651] ib_umad [ 1384.943652] sch_fq_codel [ 1384.943652] iw_cxgb4 [ 1384.943652] rdma_cm [ 1384.943653] iw_cm [ 1384.943653] ib_cm [ 1384.943653] iw_cxgb3 [ 1384.943654] zfs(POE) [ 1384.943654] binfmt_misc [ 1384.943655] zunicode(POE) [ 1384.943655] zavl(POE) [ 1384.943656] icp(POE) [ 1384.943656] zcommon(POE) [ 1384.943656] znvpair(POE) [ 1384.943657] spl(OE) [ 1384.943657] msr_safe(OE) [ 1384.943658] ip_tables [ 1384.943658] nfsv3 [ 1384.943658] nfs_acl [ 1384.943659] rpcsec_gss_krb5 [ 1384.943659] auth_rpcgss [ 1384.943659] nfsv4 [ 1384.943660] dns_resolver [ 1384.943660] nfs [ 1384.943661] lockd [ 1384.943661] grace [ 1384.943661] fscache [ 1384.943662] overlay(T) [ 1384.943662] ext4 [ 1384.943662] mbcache [ 1384.943663] jbd2 [ 1384.943663] dm_service_time [ 1384.943664] sd_mod [ 1384.943664] crc_t10dif [ 1384.943664] crct10dif_generic [ 1384.943665] mlx5_ib [ 1384.943665] ib_uverbs [ 1384.943665] ib_core [ 1384.943666] be2iscsi [ 1384.943666] bnx2i [ 1384.943666] cnic [ 1384.943667] uio [ 1384.943667] cxgb4i [ 1384.943667] cxgb4 [ 1384.943668] cxgb3i [ 1384.943668] cxgb3 [ 1384.943668] mdio [ 1384.943669] libcxgbi [ 1384.943669] libcxgb [ 1384.943670] qla4xxx [ 1384.943670] iscsi_boot_sysfs [ 1384.943671] 8021q [ 1384.943671] garp [ 1384.943671] mrp [ 1384.943672] stp [ 1384.943672] llc [ 1384.943672] mgag200 [ 1384.943673] i2c_algo_bit [ 1384.943673] drm_kms_helper [ 1384.943673] syscopyarea [ 1384.943674] sysfillrect [ 1384.943674] sysimgblt [ 1384.943674] crct10dif_pclmul [ 1384.943675] fb_sys_fops [ 1384.943675] crct10dif_common [ 1384.943675] ttm [ 1384.943676] crc32_pclmul [ 1384.943676] crc32c_intel [ 1384.943677] mlx5_core [ 1384.943677] ahci [ 1384.943677] ghash_clmulni_intel [ 1384.943678] mxm_wmi [ 1384.943678] aesni_intel [ 1384.943678] mlxfw [ 1384.943679] drm [ 1384.943679] ixgbe(OE) [ 1384.943680] mpt3sas [ 1384.943680] lrw [ 1384.943680] devlink [ 1384.943681] libahci [ 1384.943681] gf128mul [ 1384.943682] dca [ 1384.943682] glue_helper [ 1384.943682] ablk_helper [ 1384.943683] ptp [ 1384.943683] raid_class [ 1384.943683] libata [ 1384.943684] drm_panel_orientation_quirks [ 1384.943684] cryptd [ 1384.943685] scsi_transport_sas [ 1384.943685] pps_core [ 1384.943686] dm_multipath [ 1384.943686] wmi [ 1384.943686] sunrpc [ 1384.943687] dm_mirror [ 1384.943687] dm_region_hash [ 1384.943687] dm_log [ 1384.943688] dm_mod [ 1384.943688] iscsi_tcp [ 1384.943689] libiscsi_tcp [ 1384.943689] libiscsi [ 1384.943689] scsi_transport_iscsi [ 1384.943690] fuse [ 1384.943690] [ 1384.943692] CPU: 30 PID: 2899 Comm: mdt01_005 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1384.943693] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1384.943694] task: ffff88151ecc3180 ti: ffff88528ac90000 task.ti: ffff88528ac90000 [ 1384.943695] RIP: 0010:[] [ 1384.943697] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 1384.943698] RSP: 0018:ffff88528ac93ab0 EFLAGS: 00000246 [ 1384.943699] RAX: 0000000000000000 RBX: ffff8892ff580000 RCX: 0000000000f10000 [ 1384.943699] RDX: ffff8852ffb9b8c0 RSI: 0000000000310000 RDI: ffff8852f5420480 [ 1384.943700] RBP: ffff88528ac93ab0 R08: ffff8892ff59b8c0 R09: 0000000000000000 [ 1384.943701] R10: 0000000000000001 R11: fffff300bca63500 R12: ffff88528ac93a78 [ 1384.943702] R13: ffffffff930ec9b1 R14: ffff88528ac93a58 R15: ffff88528ac93a88 [ 1384.943703] FS: 0000000000000000(0000) GS:ffff8892ff580000(0000) knlGS:0000000000000000 [ 1384.943704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1384.943705] CR2: 00007ffff67b4dd0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1384.943705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1384.943706] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1384.943706] Call Trace: [ 1384.943709] [] queued_spin_lock_slowpath+0xb/0xf [ 1384.943711] [] _raw_spin_lock+0x30/0x40 [ 1384.943719] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1384.943729] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1384.943731] [] ? wake_up_atomic_t+0x30/0x30 [ 1384.943738] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1384.943772] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1384.943795] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1384.943818] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1384.943840] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1384.943862] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1384.943893] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1384.943919] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1384.943921] [] ? __getnstimeofday64+0x3f/0xd0 [ 1384.943945] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1384.943968] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1384.943969] [] ? __wake_up_common_lock+0x91/0xc0 [ 1384.943970] [] ? sched_feat_set+0xf0/0xf0 [ 1384.943993] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1384.943995] [] ? __switch_to+0xce/0x5a0 [ 1384.944017] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1384.944019] [] kthread+0xd1/0xe0 [ 1384.944021] [] ? insert_kthread_work+0x40/0x40 [ 1384.944022] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1384.944024] [] ? insert_kthread_work+0x40/0x40 [ 1384.944025] Code: [ 1384.944025] c1 [ 1384.944025] e8 [ 1384.944026] 13 [ 1384.944026] 48 [ 1384.944026] c1 [ 1384.944026] ea [ 1384.944027] 0d [ 1384.944027] 48 [ 1384.944028] 98 [ 1384.944028] 83 [ 1384.944028] e2 [ 1384.944028] 30 [ 1384.944029] 48 [ 1384.944029] 81 [ 1384.944029] c2 [ 1384.944030] c0 [ 1384.944030] b8 [ 1384.944031] 01 [ 1384.944031] 00 [ 1384.944031] 48 [ 1384.944031] 03 [ 1384.944032] 14 [ 1384.944032] c5 [ 1384.944032] e0 [ 1384.944033] 17 [ 1384.944033] d5 [ 1384.944033] 93 [ 1384.944034] 4c [ 1384.944034] 89 [ 1384.944034] 02 [ 1384.944035] 41 [ 1384.944035] 8b [ 1384.944036] 40 [ 1384.944036] 08 [ 1384.944036] 85 [ 1384.944037] c0 [ 1384.944037] 75 [ 1384.944037] 0f [ 1384.944037] 0f [ 1384.944038] 1f [ 1384.944038] 44 [ 1384.944039] 00 [ 1384.944039] 00 [ 1384.944039] [ 1384.944040] 90 [ 1384.944040] 41 [ 1384.944040] 8b [ 1384.944040] 40 [ 1384.944041] 08 [ 1384.944041] 85 [ 1384.944041] c0 [ 1384.944042] 74 [ 1384.944043] f6 [ 1384.944043] 4d [ 1384.944044] 8b [ 1384.944044] 08 [ 1384.944044] 4d [ 1384.944045] 85 [ 1384.944045] c9 [ 1384.944045] 74 [ 1384.944046] 04 [ 1384.944046] 41 [ 1384.944046] 0f [ 1384.944046] 18 [ 1384.944047] [ 1385.447252] ipmi_devintf [ 1385.450668] ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw [ 1385.528493] devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1385.554165] CPU: 14 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1385.568430] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1385.580754] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1385.589587] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [ 1385.601149] RSP: 0018:ffff88528ad4fab0 EFLAGS: 00000202 [ 1385.607561] RAX: 0000000000000001 RBX: ffff8892ff380000 RCX: 0000000000710000 [ 1385.616006] RDX: 0000000000190000 RSI: 0000000000e10000 RDI: ffff8852f5420480 [ 1385.624454] RBP: ffff88528ad4fab0 R08: ffff8892ff39b8c0 R09: ffff8892ff2db8c0 [ 1385.632900] R10: 0000000000000001 R11: ffff8852a386f800 R12: ffff88528ad4fa78 [ 1385.641346] R13: ffffffff930ec9b1 R14: ffff88528ad4fa58 R15: ffff88528ad4fa88 [ 1385.649793] FS: 0000000000000000(0000) GS:ffff8892ff380000(0000) knlGS:0000000000000000 [ 1385.659307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1385.666201] CR2: 00007fb699688008 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1385.674649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1385.683096] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1385.691542] Call Trace: [ 1385.694756] [] queued_spin_lock_slowpath+0xb/0xf [ 1385.702234] [] _raw_spin_lock+0x30/0x40 [ 1385.708843] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1385.716522] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1385.724969] [] ? wake_up_atomic_t+0x30/0x30 [ 1385.731968] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1385.739374] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1385.747649] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1385.755636] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1385.764298] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1385.772186] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1385.779500] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1385.787683] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1385.796614] [] ? __getnstimeofday64+0x3f/0xd0 [ 1385.803826] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1385.812878] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1385.820937] [] ? __wake_up_common_lock+0x91/0xc0 [ 1385.828415] [] ? sched_feat_set+0xf0/0xf0 [ 1385.835239] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1385.842718] [] ? __switch_to+0xce/0x5a0 [ 1385.849346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1385.858085] [] kthread+0xd1/0xe0 [ 1385.864012] [] ? insert_kthread_work+0x40/0x40 [ 1385.871297] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1385.879067] [] ? insert_kthread_work+0x40/0x40 [ 1385.886351] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [ 1388.779548] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [mdt00_011:4259] [ 1388.788499] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1388.868350] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1388.928624] CPU: 3 PID: 4259 Comm: mdt00_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1388.942790] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1388.955113] task: ffff8892c975b180 ti: ffff8892f75f0000 task.ti: ffff8892f75f0000 [ 1388.963947] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1388.975510] RSP: 0018:ffff8892f75f3ab0 EFLAGS: 00000246 [ 1388.981918] RAX: 0000000000000000 RBX: ffff8852ffac0000 RCX: 0000000000190000 [ 1388.990365] RDX: ffff8892ff59b8c0 RSI: 0000000000f10000 RDI: ffff8852f5420480 [ 1388.998811] RBP: ffff8892f75f3ab0 R08: ffff8852ffadb8c0 R09: 0000000000000000 [ 1389.007257] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f75f3a78 [ 1389.015703] R13: ffffffff930ec9b1 R14: ffff8892f75f3a58 R15: ffff8892f75f3a88 [ 1389.024150] FS: 0000000000000000(0000) GS:ffff8852ffac0000(0000) knlGS:0000000000000000 [ 1389.033661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1389.040556] CR2: 00007ffff1096160 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1389.049002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1389.057449] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1389.065895] Call Trace: [ 1389.069108] [] queued_spin_lock_slowpath+0xb/0xf [ 1389.076586] [] _raw_spin_lock+0x30/0x40 [ 1389.083201] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1389.090875] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1389.098852] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1389.107397] [] ? wake_up_atomic_t+0x30/0x30 [ 1389.114397] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1389.121808] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1389.130080] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1389.138059] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1389.146716] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1389.154596] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1389.161905] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1389.170083] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1389.179016] [] ? __getnstimeofday64+0x3f/0xd0 [ 1389.186220] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1389.195266] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1389.203325] [] ? __wake_up_common_lock+0x91/0xc0 [ 1389.210802] [] ? sched_feat_set+0xf0/0xf0 [ 1389.217619] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1389.225096] [] ? __switch_to+0xce/0x5a0 [ 1389.231717] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1389.240455] [] kthread+0xd1/0xe0 [ 1389.246381] [] ? insert_kthread_work+0x40/0x40 [ 1389.253664] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1389.261432] [] ? insert_kthread_work+0x40/0x40 [ 1389.268714] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1395.351770] LustreError: 137-5: ls1-MDT0000_UUID: not available for connect from 192.168.130.66@o2ib27 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.372050] LustreError: Skipped 12443 previous similar messages [ 1395.896382] INFO: rcu_sched self-detected stall on CPU [ 1395.896387] INFO: rcu_sched self-detected stall on CPU { 29} (t=240005 jiffies g=25118 c=25117 q=98099) [ 1395.896388] Task dump for CPU 11: [ 1395.896390] mdt01_006 R running task 0 3034 2 0x00000088 [ 1395.896391] Call Trace: [ 1395.896394] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 1395.896397] [] ? queued_spin_lock_slowpath+0xb/0xf [ 1395.896399] [] ? _raw_spin_lock+0x30/0x40 [ 1395.896409] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1395.896418] [] ? lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1395.896421] [] ? wake_up_atomic_t+0x30/0x30 [ 1395.896426] [] ? LNetPrimaryNID+0xd5/0x220 [lnet] [ 1395.896463] [] ? ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1395.896481] [] ? ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1395.896499] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1395.896517] [] ? ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1395.896534] [] ? ptlrpc_error+0x10/0x20 [ptlrpc] [ 1395.896562] [] ? tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1395.896583] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1395.896584] [] ? __getnstimeofday64+0x3f/0xd0 [ 1395.896604] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1395.896622] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1395.896623] [] ? __wake_up_common_lock+0x91/0xc0 [ 1395.896624] [] ? sched_feat_set+0xf0/0xf0 [ 1395.896643] [] ? ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1395.896644] [] ? __switch_to+0xce/0x5a0 [ 1395.896662] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1395.896664] [] ? kthread+0xd1/0xe0 [ 1395.896666] [] ? insert_kthread_work+0x40/0x40 [ 1395.896667] [] ? ret_from_fork_nospec_begin+0x21/0x21 [ 1395.896668] [] ? insert_kthread_work+0x40/0x40 [ 1395.896669] Task dump for CPU 29: [ 1395.896670] mdt01_013 R running task 0 4292 2 0x00000088 [ 1395.896671] Call Trace: [ 1395.896674] [] sched_show_task+0xbf/0x120 [ 1395.896676] [] dump_cpu_task+0x39/0x70 [ 1395.896680] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 1395.896683] [] rcu_check_callbacks+0x482/0x770 [ 1395.896686] [] update_process_times+0x46/0x80 [ 1395.896689] [] tick_sched_handle+0x30/0x70 [ 1395.896691] [] tick_sched_timer+0x39/0x80 [ 1395.896692] [] __hrtimer_run_queues+0x13e/0x2f0 [ 1395.896694] [] ? tick_sched_do_timer+0x50/0x50 [ 1395.896695] [] hrtimer_interrupt+0xb9/0x1f0 [ 1395.896698] [] local_apic_timer_interrupt+0x3b/0x60 [ 1395.896700] [] smp_apic_timer_interrupt+0x43/0x60 [ 1395.896701] [] apic_timer_interrupt+0x16a/0x170 [ 1395.896703] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 1395.896705] [] queued_spin_lock_slowpath+0xb/0xf [ 1395.896706] [] _raw_spin_lock+0x30/0x40 [ 1395.896712] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1395.896718] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1395.896720] [] ? wake_up_atomic_t+0x30/0x30 [ 1395.896725] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1395.896743] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1395.896760] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1395.896778] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1395.896795] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1395.896812] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1395.896834] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1395.896859] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1395.896860] [] ? __getnstimeofday64+0x3f/0xd0 [ 1395.896886] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1395.896911] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1395.896912] [] ? __wake_up_common_lock+0x91/0xc0 [ 1395.896913] [] ? sched_feat_set+0xf0/0xf0 [ 1395.896939] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1395.896940] [] ? __switch_to+0xce/0x5a0 [ 1395.896966] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1395.896967] [] kthread+0xd1/0xe0 [ 1395.896969] [] ? insert_kthread_work+0x40/0x40 [ 1395.896970] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1395.896971] [] ? insert_kthread_work+0x40/0x40 [ 1396.442123] { 11} (t=240553 jiffies g=25118 c=25117 q=98103) [ 1396.449149] Task dump for CPU 11: [ 1396.453328] mdt01_006 R running task 0 3034 2 0x00000088 [ 1396.461697] Call Trace: [ 1396.464909] [] sched_show_task+0xbf/0x120 [ 1396.472400] [] dump_cpu_task+0x39/0x70 [ 1396.478908] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 1396.485997] [] rcu_check_callbacks+0x482/0x770 [ 1396.493281] [] update_process_times+0x46/0x80 [ 1396.500470] [] tick_sched_handle+0x30/0x70 [ 1396.507364] [] tick_sched_timer+0x39/0x80 [ 1396.514164] [] __hrtimer_run_queues+0x13e/0x2f0 [ 1396.521544] [] ? tick_sched_do_timer+0x50/0x50 [ 1396.528827] [] hrtimer_interrupt+0xb9/0x1f0 [ 1396.535820] [] local_apic_timer_interrupt+0x3b/0x60 [ 1396.543589] [] smp_apic_timer_interrupt+0x43/0x60 [ 1396.551164] [] apic_timer_interrupt+0x16a/0x170 [ 1396.558543] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 1396.567973] [] queued_spin_lock_slowpath+0xb/0xf [ 1396.575451] [] _raw_spin_lock+0x30/0x40 [ 1396.582062] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1396.589740] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1396.598189] [] ? wake_up_atomic_t+0x30/0x30 [ 1396.605188] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1396.612594] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1396.620866] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1396.628847] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1396.637504] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1396.645388] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1396.652695] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1396.660871] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1396.669803] [] ? __getnstimeofday64+0x3f/0xd0 [ 1396.677009] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1396.686057] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1396.694117] [] ? __wake_up_common_lock+0x91/0xc0 [ 1396.701594] [] ? sched_feat_set+0xf0/0xf0 [ 1396.708412] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1396.715891] [] ? __switch_to+0xce/0x5a0 [ 1396.722515] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1396.731252] [] kthread+0xd1/0xe0 [ 1396.737178] [] ? insert_kthread_work+0x40/0x40 [ 1396.744460] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1396.752230] [] ? insert_kthread_work+0x40/0x40 [ 1396.759513] Task dump for CPU 29: [ 1396.763694] mdt01_013 R running task 0 4292 2 0x00000088 [ 1396.772065] Call Trace: [ 1396.775275] [] ? native_queued_spin_lock_slowpath+0x120/0x200 [ 1396.784014] [] ? queued_spin_lock_slowpath+0xb/0xf [ 1396.791687] [] ? _raw_spin_lock+0x30/0x40 [ 1396.798489] [] ? cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1396.806352] [] ? lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1396.814995] [] ? wake_up_atomic_t+0x30/0x30 [ 1396.821992] [] ? LNetPrimaryNID+0xd5/0x220 [lnet] [ 1396.829585] [] ? ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1396.838052] [] ? ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1396.846225] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1396.854884] [] ? ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1396.862961] [] ? ptlrpc_error+0x10/0x20 [ptlrpc] [ 1396.870462] [] ? tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1396.876360] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt01_011:4290] [ 1396.876394] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1396.876422] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1396.876424] CPU: 9 PID: 4290 Comm: mdt01_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1396.876425] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1396.876426] task: ffff8892ca848000 ti: ffff8892f6434000 task.ti: ffff8892f6434000 [ 1396.876429] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1396.876430] RSP: 0018:ffff8892f6437ab0 EFLAGS: 00000246 [ 1396.876431] RAX: 0000000000000000 RBX: ffff8892ff240000 RCX: 0000000000490000 [ 1396.876432] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1396.876433] RBP: ffff8892f6437ab0 R08: ffff8892ff25b8c0 R09: 0000000000000000 [ 1396.876434] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f6437a78 [ 1396.876434] R13: ffffffff930ec9b1 R14: ffff8892f6437a58 R15: ffff8892f6437a88 [ 1396.876436] FS: 0000000000000000(0000) GS:ffff8892ff240000(0000) knlGS:0000000000000000 [ 1396.876437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1396.876438] CR2: 00007ffff7ff8000 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1396.876439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1396.876439] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1396.876440] Call Trace: [ 1396.876442] [] queued_spin_lock_slowpath+0xb/0xf [ 1396.876444] [] _raw_spin_lock+0x30/0x40 [ 1396.876451] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1396.876459] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1396.876462] [] ? wake_up_atomic_t+0x30/0x30 [ 1396.876469] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1396.876493] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1396.876518] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1396.876543] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1396.876566] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1396.876589] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1396.876619] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1396.876647] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1396.876649] [] ? __getnstimeofday64+0x3f/0xd0 [ 1396.876675] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1396.876699] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1396.876701] [] ? __wake_up_common_lock+0x91/0xc0 [ 1396.876702] [] ? sched_feat_set+0xf0/0xf0 [ 1396.876727] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1396.876729] [] ? __switch_to+0xce/0x5a0 [ 1396.876752] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1396.876754] [] kthread+0xd1/0xe0 [ 1396.876756] [] ? insert_kthread_work+0x40/0x40 [ 1396.876758] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1396.876760] [] ? insert_kthread_work+0x40/0x40 [ 1396.876779] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 1397.379678] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1397.388609] [] ? __getnstimeofday64+0x3f/0xd0 [ 1397.395814] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1397.405054] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1397.413114] [] ? __wake_up_common_lock+0x91/0xc0 [ 1397.420591] [] ? sched_feat_set+0xf0/0xf0 [ 1397.427409] [] ? ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1397.435080] [] ? __switch_to+0xce/0x5a0 [ 1397.441703] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1397.450440] [] ? kthread+0xd1/0xe0 [ 1397.456560] [] ? insert_kthread_work+0x40/0x40 [ 1397.463842] [] ? ret_from_fork_nospec_begin+0x21/0x21 [ 1397.471804] [] ? insert_kthread_work+0x40/0x40 [ 1400.791265] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mdt00_009:4257] [ 1400.800188] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1400.933262] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mdt01_002:2849] [ 1400.879955] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log [ 1400.933263] Modules linked in: [ 1400.933264] osp(OE) [ 1400.933264] mdd(OE) [ 1400.933265] lod(OE) [ 1400.933265] mdt(OE) [ 1400.933265] lfsck(OE) [ 1400.933266] mgc(OE) [ 1400.933266] osd_zfs(OE) [ 1400.933266] lquota(OE) [ 1400.933267] fid(OE) [ 1400.933267] fld(OE) [ 1400.933267] ptlrpc(OE) [ 1400.933268] obdclass(OE) [ 1400.933268] ko2iblnd(OE) [ 1400.933268] lnet(OE) [ 1400.933269] libcfs(OE) [ 1400.933269] iTCO_wdt [ 1400.933269] iTCO_vendor_support [ 1400.933270] rpcrdma [ 1400.933270] ib_iser [ 1400.933270] sb_edac [ 1400.933271] intel_powerclamp [ 1400.933271] coretemp [ 1400.933271] intel_rapl [ 1400.933272] iosf_mbi [ 1400.933272] kvm [ 1400.933272] irqbypass [ 1400.933273] pcspkr [ 1400.933273] joydev [ 1400.933273] lpc_ich [ 1400.933274] i2c_i801 [ 1400.933274] ioatdma [ 1400.933274] ses [ 1400.933275] enclosure [ 1400.933275] sg [ 1400.933275] ipmi_si [ 1400.933275] ipmi_devintf [ 1400.933276] ipmi_msghandler [ 1400.933276] dm_round_robin [ 1400.933276] acpi_cpufreq [ 1400.933277] acpi_power_meter [ 1400.933277] ib_ipoib [ 1400.933277] rdma_ucm [ 1400.933278] ib_umad [ 1400.933278] sch_fq_codel [ 1400.933278] iw_cxgb4 [ 1400.933278] rdma_cm [ 1400.933279] iw_cm [ 1400.933279] ib_cm [ 1400.933279] iw_cxgb3 [ 1400.933280] zfs(POE) [ 1400.933280] binfmt_misc [ 1400.933281] zunicode(POE) [ 1400.933281] zavl(POE) [ 1400.933282] icp(POE) [ 1400.933282] zcommon(POE) [ 1400.933282] znvpair(POE) [ 1400.933283] spl(OE) [ 1400.933283] msr_safe(OE) [ 1400.933284] ip_tables [ 1400.933284] nfsv3 [ 1400.933285] nfs_acl [ 1400.933285] rpcsec_gss_krb5 [ 1400.933285] auth_rpcgss [ 1400.933286] nfsv4 [ 1400.933286] dns_resolver [ 1400.933286] nfs [ 1400.933286] lockd [ 1400.933287] grace [ 1400.933287] fscache [ 1400.933288] overlay(T) [ 1400.933288] ext4 [ 1400.933288] mbcache [ 1400.933289] jbd2 [ 1400.933289] dm_service_time [ 1400.933290] sd_mod [ 1400.933290] crc_t10dif [ 1400.933290] crct10dif_generic [ 1400.933290] mlx5_ib [ 1400.933291] ib_uverbs [ 1400.933291] ib_core [ 1400.933291] be2iscsi [ 1400.933292] bnx2i [ 1400.933292] cnic [ 1400.933293] uio [ 1400.933293] cxgb4i [ 1400.933293] cxgb4 [ 1400.933294] cxgb3i [ 1400.933294] cxgb3 [ 1400.933294] mdio [ 1400.933295] libcxgbi [ 1400.933295] libcxgb [ 1400.933295] qla4xxx [ 1400.933296] iscsi_boot_sysfs [ 1400.933296] 8021q [ 1400.933296] garp [ 1400.933297] mrp [ 1400.933297] stp [ 1400.933298] llc [ 1400.933298] mgag200 [ 1400.933298] i2c_algo_bit [ 1400.933298] drm_kms_helper [ 1400.933299] syscopyarea [ 1400.933299] sysfillrect [ 1400.933300] sysimgblt [ 1400.933300] crct10dif_pclmul [ 1400.933301] fb_sys_fops [ 1400.933301] crct10dif_common [ 1400.933301] ttm [ 1400.933302] crc32_pclmul [ 1400.933302] crc32c_intel [ 1400.933302] mlx5_core [ 1400.933303] ahci [ 1400.933303] ghash_clmulni_intel [ 1400.933303] mxm_wmi [ 1400.933303] aesni_intel [ 1400.933304] mlxfw [ 1400.933304] drm [ 1400.933304] ixgbe(OE) [ 1400.933305] mpt3sas [ 1400.933305] lrw [ 1400.933305] devlink [ 1400.933306] libahci [ 1400.933306] gf128mul [ 1400.933306] dca [ 1400.933307] glue_helper [ 1400.933307] ablk_helper [ 1400.933307] ptp [ 1400.933308] raid_class [ 1400.933308] libata [ 1400.933308] drm_panel_orientation_quirks [ 1400.933308] cryptd [ 1400.933309] scsi_transport_sas [ 1400.933309] pps_core [ 1400.933310] dm_multipath [ 1400.933310] wmi [ 1400.933310] sunrpc [ 1400.933311] dm_mirror [ 1400.933311] dm_region_hash [ 1400.933311] dm_log [ 1400.933311] dm_mod [ 1400.933312] iscsi_tcp [ 1400.933312] libiscsi_tcp [ 1400.933312] libiscsi [ 1400.933313] scsi_transport_iscsi [ 1400.933313] fuse [ 1400.933313] [ 1400.933315] CPU: 26 PID: 2849 Comm: mdt01_002 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1400.933316] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1400.933317] task: ffff8852a8968000 ti: ffff8852ebf00000 task.ti: ffff8852ebf00000 [ 1400.933318] RIP: 0010:[] [ 1400.933320] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 1400.933321] RSP: 0018:ffff8852ebf03ab0 EFLAGS: 00000246 [ 1400.933322] RAX: 0000000000000000 RBX: ffff8892ff480000 RCX: 0000000000d10000 [ 1400.933322] RDX: ffff8892ff2db8c0 RSI: 0000000000590000 RDI: ffff8852f5420480 [ 1400.933323] RBP: ffff8852ebf03ab0 R08: ffff8892ff49b8c0 R09: 0000000000000000 [ 1400.933324] R10: 0000000000000001 R11: fffff2ffbc2b2500 R12: ffff8852ebf03a78 [ 1400.933324] R13: ffffffff930ec9b1 R14: ffff8852ebf03a58 R15: ffff8852ebf03a88 [ 1400.933325] FS: 0000000000000000(0000) GS:ffff8892ff480000(0000) knlGS:0000000000000000 [ 1400.933326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1400.933327] CR2: 00007f37450e7504 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1400.933327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1400.933328] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1400.933328] Call Trace: [ 1400.933331] [] queued_spin_lock_slowpath+0xb/0xf [ 1400.933333] [] _raw_spin_lock+0x30/0x40 [ 1400.933341] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1400.933351] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1400.933353] [] ? wake_up_atomic_t+0x30/0x30 [ 1400.933358] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1400.933391] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1400.933409] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1400.933428] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1400.933445] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1400.933468] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1400.933498] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1400.933524] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1400.933526] [] ? __getnstimeofday64+0x3f/0xd0 [ 1400.933551] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1400.933576] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1400.933578] [] ? __wake_up_common_lock+0x91/0xc0 [ 1400.933579] [] ? sched_feat_set+0xf0/0xf0 [ 1400.933604] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1400.933606] [] ? __switch_to+0xce/0x5a0 [ 1400.933631] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1400.933633] [] kthread+0xd1/0xe0 [ 1400.933635] [] ? insert_kthread_work+0x40/0x40 [ 1400.933636] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1400.933637] [] ? insert_kthread_work+0x40/0x40 [ 1400.933638] Code: [ 1400.933638] 13 [ 1400.933639] 48 [ 1400.933639] c1 [ 1400.933639] ea [ 1400.933639] 0d [ 1400.933640] 48 [ 1400.933640] 98 [ 1400.933640] 83 [ 1400.933640] e2 [ 1400.933641] 30 [ 1400.933641] 48 [ 1400.933641] 81 [ 1400.933641] c2 [ 1400.933642] c0 [ 1400.933642] b8 [ 1400.933642] 01 [ 1400.933643] 00 [ 1400.933643] 48 [ 1400.933644] 03 [ 1400.933645] 14 [ 1400.933645] c5 [ 1400.933645] e0 [ 1400.933646] 17 [ 1400.933646] d5 [ 1400.933646] 93 [ 1400.933646] 4c [ 1400.933647] 89 [ 1400.933647] 02 [ 1400.933647] 41 [ 1400.933647] 8b [ 1400.933648] 40 [ 1400.933648] 08 [ 1400.933648] 85 [ 1400.933648] c0 [ 1400.933649] 75 [ 1400.933649] 0f [ 1400.933649] 0f [ 1400.933649] 1f [ 1400.933650] 44 [ 1400.933650] 00 [ 1400.933650] 00 [ 1400.933651] f3 [ 1400.933651] 90 [ 1400.933651] <41> [ 1400.933651] 8b [ 1400.933652] 40 [ 1400.933652] 08 [ 1400.933652] 85 [ 1400.933652] c0 [ 1400.933653] 74 [ 1400.933653] f6 [ 1400.933653] 4d [ 1400.933653] 8b [ 1400.933654] 08 [ 1400.933654] 4d [ 1400.933654] 85 [ 1400.933654] c9 [ 1400.933655] 74 [ 1400.933655] 04 [ 1400.933655] 41 [ 1400.933655] 0f [ 1400.933655] 18 [ 1400.933656] 09 [ 1400.933656] 8b [ 1400.933656] [ 1401.260264] LNet: Service thread pid 4290 was inactive for 200.43s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 1401.260266] LustreError: dumping log to /tmp/lustre-log.1644346615.4290 [ 1401.459216] dm_mod [ 1401.462048] iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1401.467921] CPU: 6 PID: 4257 Comm: mdt00_009 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1401.482088] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1401.494412] task: ffff8892c975c200 ti: ffff8892ac49c000 task.ti: ffff8892ac49c000 [ 1401.503244] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1401.514807] RSP: 0018:ffff8892ac49fab0 EFLAGS: 00000246 [ 1401.521216] RAX: 0000000000000000 RBX: ffff8852ffb80000 RCX: 0000000000310000 [ 1401.529661] RDX: ffff8892ff39b8c0 RSI: 0000000000710000 RDI: ffff8852f5420480 [ 1401.538106] RBP: ffff8892ac49fab0 R08: ffff8852ffb9b8c0 R09: 0000000000000000 [ 1401.546551] R10: 0000000000000001 R11: 00000000000024c0 R12: ffff8892ac49fa78 [ 1401.554996] R13: ffffffff930ec9b1 R14: ffff8892ac49fa58 R15: ffff8892ac49fa88 [ 1401.563442] FS: 0000000000000000(0000) GS:ffff8852ffb80000(0000) knlGS:0000000000000000 [ 1401.572954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1401.579849] CR2: 00007ffff1096160 CR3: 0000003f25442000 CR4: 00000000003607e0 [ 1401.588295] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1401.596741] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1401.605186] Call Trace: [ 1401.608400] [] queued_spin_lock_slowpath+0xb/0xf [ 1401.615879] [] _raw_spin_lock+0x30/0x40 [ 1401.622493] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1401.630175] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1401.638624] [] ? wake_up_atomic_t+0x30/0x30 [ 1401.645622] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1401.653034] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1401.661303] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1401.669290] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1401.677948] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1401.685829] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1401.693138] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1401.701320] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1401.710252] [] ? __getnstimeofday64+0x3f/0xd0 [ 1401.717463] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1401.726516] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1401.734575] [] ? __wake_up_common_lock+0x91/0xc0 [ 1401.742051] [] ? sched_feat_set+0xf0/0xf0 [ 1401.748869] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1401.753255] LNet: 30707:0:(o2iblnd_cb.c:3432:kiblnd_check_conns()) Timed out tx for 172.19.3.102@o2ib600: 1 seconds [ 1401.768474] [] ? __switch_to+0xce/0x5a0 [ 1401.775103] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1401.783843] [] kthread+0xd1/0xe0 [ 1401.789769] [] ? insert_kthread_work+0x40/0x40 [ 1401.797053] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1401.804821] [] ? insert_kthread_work+0x40/0x40 [ 1401.812103] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1404.938166] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [mdt01_012:4291] [ 1404.947197] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1405.027058] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1405.087366] CPU: 28 PID: 4291 Comm: mdt01_012 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1405.101631] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1405.113956] task: ffff8892ca84e300 ti: ffff8892ada10000 task.ti: ffff8892ada10000 [ 1405.122790] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1405.134351] RSP: 0018:ffff8892ada13ab0 EFLAGS: 00000246 [ 1405.140761] RAX: 0000000000000000 RBX: ffff8892ff500000 RCX: 0000000000e10000 [ 1405.149207] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1405.157653] RBP: ffff8892ada13ab0 R08: ffff8892ff51b8c0 R09: 0000000000000000 [ 1405.166098] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892ada13a78 [ 1405.174546] R13: ffffffff930ec9b1 R14: ffff8892ada13a58 R15: ffff8892ada13a88 [ 1405.182992] FS: 0000000000000000(0000) GS:ffff8892ff500000(0000) knlGS:0000000000000000 [ 1405.192504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1405.199401] CR2: 00007ffff69e4e10 CR3: 0000007f79b8a000 CR4: 00000000003607e0 [ 1405.207847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1405.216293] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1405.224740] Call Trace: [ 1405.227952] [] queued_spin_lock_slowpath+0xb/0xf [ 1405.235431] [] _raw_spin_lock+0x30/0x40 [ 1405.242045] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1405.249727] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1405.258175] [] ? wake_up_atomic_t+0x30/0x30 [ 1405.265175] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1405.272594] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1405.280863] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1405.288849] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1405.297513] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1405.305400] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1405.312718] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1405.320901] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1405.329833] [] ? __getnstimeofday64+0x3f/0xd0 [ 1405.337045] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1405.346097] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1405.354156] [] ? __wake_up_common_lock+0x91/0xc0 [ 1405.361634] [] ? sched_feat_set+0xf0/0xf0 [ 1405.368456] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1405.375935] [] ? __switch_to+0xce/0x5a0 [ 1405.382563] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1405.391301] [] kthread+0xd1/0xe0 [ 1405.397229] [] ? insert_kthread_work+0x40/0x40 [ 1405.404514] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1405.412282] [] ? insert_kthread_work+0x40/0x40 [ 1405.419566] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1412.898971] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mdt01_007:3041] [ 1412.942970] NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [mdt01_005:2899] [ 1412.908003] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si [ 1412.942970] Modules linked in: [ 1412.942971] osp(OE) [ 1412.942971] mdd(OE) [ 1412.942971] lod(OE) [ 1412.942972] mdt(OE) [ 1412.942972] lfsck(OE) [ 1412.942972] mgc(OE) [ 1412.942973] osd_zfs(OE) [ 1412.942973] lquota(OE) [ 1412.942974] fid(OE) [ 1412.942974] fld(OE) [ 1412.942974] ptlrpc(OE) [ 1412.942975] obdclass(OE) [ 1412.942975] ko2iblnd(OE) [ 1412.942975] lnet(OE) [ 1412.942976] libcfs(OE) [ 1412.942976] iTCO_wdt [ 1412.942976] iTCO_vendor_support [ 1412.942977] rpcrdma [ 1412.942977] ib_iser [ 1412.942977] sb_edac [ 1412.942978] intel_powerclamp [ 1412.942978] coretemp [ 1412.942978] intel_rapl [ 1412.942979] iosf_mbi [ 1412.942979] kvm [ 1412.942979] irqbypass [ 1412.942980] pcspkr [ 1412.942980] joydev [ 1412.942980] lpc_ich [ 1412.942980] i2c_i801 [ 1412.942981] ioatdma [ 1412.942981] ses [ 1412.942981] enclosure [ 1412.942982] sg [ 1412.942982] ipmi_si [ 1412.942982] ipmi_devintf [ 1412.942983] ipmi_msghandler [ 1412.942983] dm_round_robin [ 1412.942984] acpi_cpufreq [ 1412.942984] acpi_power_meter [ 1412.942984] ib_ipoib [ 1412.942985] rdma_ucm [ 1412.942985] ib_umad [ 1412.942985] sch_fq_codel [ 1412.942986] iw_cxgb4 [ 1412.942986] rdma_cm [ 1412.942986] iw_cm [ 1412.942987] ib_cm [ 1412.942987] iw_cxgb3 [ 1412.942987] zfs(POE) [ 1412.942988] binfmt_misc [ 1412.942988] zunicode(POE) [ 1412.942988] zavl(POE) [ 1412.942989] icp(POE) [ 1412.942989] zcommon(POE) [ 1412.942990] znvpair(POE) [ 1412.942990] spl(OE) [ 1412.942991] msr_safe(OE) [ 1412.942991] ip_tables [ 1412.942991] nfsv3 [ 1412.942992] nfs_acl [ 1412.942992] rpcsec_gss_krb5 [ 1412.942993] auth_rpcgss [ 1412.942993] nfsv4 [ 1412.942993] dns_resolver [ 1412.942993] nfs [ 1412.942994] lockd [ 1412.942994] grace [ 1412.942995] fscache [ 1412.942995] overlay(T) [ 1412.942995] ext4 [ 1412.942996] mbcache [ 1412.942996] jbd2 [ 1412.942997] dm_service_time [ 1412.942997] sd_mod [ 1412.942997] crc_t10dif [ 1412.942998] crct10dif_generic [ 1412.942998] mlx5_ib [ 1412.942998] ib_uverbs [ 1412.942999] ib_core [ 1412.942999] be2iscsi [ 1412.943000] bnx2i [ 1412.943000] cnic [ 1412.943000] uio [ 1412.943001] cxgb4i [ 1412.943001] cxgb4 [ 1412.943001] cxgb3i [ 1412.943002] cxgb3 [ 1412.943002] mdio [ 1412.943002] libcxgbi [ 1412.943003] libcxgb [ 1412.943003] qla4xxx [ 1412.943004] iscsi_boot_sysfs [ 1412.943004] 8021q [ 1412.943004] garp [ 1412.943005] mrp [ 1412.943005] stp [ 1412.943006] llc [ 1412.943006] mgag200 [ 1412.943007] i2c_algo_bit [ 1412.943007] drm_kms_helper [ 1412.943007] syscopyarea [ 1412.943007] sysfillrect [ 1412.943008] sysimgblt [ 1412.943008] crct10dif_pclmul [ 1412.943009] fb_sys_fops [ 1412.943009] crct10dif_common [ 1412.943009] ttm [ 1412.943010] crc32_pclmul [ 1412.943010] crc32c_intel [ 1412.943011] mlx5_core [ 1412.943011] ahci [ 1412.943011] ghash_clmulni_intel [ 1412.943012] mxm_wmi [ 1412.943012] aesni_intel [ 1412.943013] mlxfw [ 1412.943013] drm [ 1412.943014] ixgbe(OE) [ 1412.943014] mpt3sas [ 1412.943014] lrw [ 1412.943015] devlink [ 1412.943015] libahci [ 1412.943016] gf128mul [ 1412.943016] dca [ 1412.943016] glue_helper [ 1412.943017] ablk_helper [ 1412.943017] ptp [ 1412.943017] raid_class [ 1412.943018] libata [ 1412.943018] drm_panel_orientation_quirks [ 1412.943019] cryptd [ 1412.943019] scsi_transport_sas [ 1412.943019] pps_core [ 1412.943020] dm_multipath [ 1412.943020] wmi [ 1412.943020] sunrpc [ 1412.943021] dm_mirror [ 1412.943021] dm_region_hash [ 1412.943021] dm_log [ 1412.943022] dm_mod [ 1412.943022] iscsi_tcp [ 1412.943023] libiscsi_tcp [ 1412.943023] libiscsi [ 1412.943024] scsi_transport_iscsi [ 1412.943024] fuse [ 1412.943024] [ 1412.943026] CPU: 30 PID: 2899 Comm: mdt01_005 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1412.943027] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1412.943028] task: ffff88151ecc3180 ti: ffff88528ac90000 task.ti: ffff88528ac90000 [ 1412.943029] RIP: 0010:[] [ 1412.943031] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1412.943032] RSP: 0018:ffff88528ac93ab0 EFLAGS: 00000246 [ 1412.943033] RAX: 0000000000000000 RBX: ffff8892ff580000 RCX: 0000000000f10000 [ 1412.943034] RDX: ffff8892ff49b8c0 RSI: 0000000000d10000 RDI: ffff8852f5420480 [ 1412.943035] RBP: ffff88528ac93ab0 R08: ffff8892ff59b8c0 R09: 0000000000000000 [ 1412.943035] R10: 0000000000000001 R11: fffff300bca63500 R12: ffff88528ac93a78 [ 1412.943036] R13: ffffffff930ec9b1 R14: ffff88528ac93a58 R15: ffff88528ac93a88 [ 1412.943038] FS: 0000000000000000(0000) GS:ffff8892ff580000(0000) knlGS:0000000000000000 [ 1412.943038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1412.943039] CR2: 00007ffff67b4dd0 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1412.943040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1412.943041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1412.943041] Call Trace: [ 1412.943044] [] queued_spin_lock_slowpath+0xb/0xf [ 1412.943046] [] _raw_spin_lock+0x30/0x40 [ 1412.943056] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1412.943067] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1412.943069] [] ? wake_up_atomic_t+0x30/0x30 [ 1412.943076] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1412.943114] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1412.943137] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1412.943161] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1412.943182] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1412.943204] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1412.943236] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1412.943263] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1412.943264] [] ? __getnstimeofday64+0x3f/0xd0 [ 1412.943289] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1412.943312] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1412.943313] [] ? __wake_up_common_lock+0x91/0xc0 [ 1412.943314] [] ? sched_feat_set+0xf0/0xf0 [ 1412.943338] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1412.943340] [] ? __switch_to+0xce/0x5a0 [ 1412.943362] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1412.943364] [] kthread+0xd1/0xe0 [ 1412.943366] [] ? insert_kthread_work+0x40/0x40 [ 1412.943368] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1412.943369] [] ? insert_kthread_work+0x40/0x40 [ 1412.943370] Code: [ 1412.943371] 0d [ 1412.943371] 48 [ 1412.943372] 98 [ 1412.943372] 83 [ 1412.943372] e2 [ 1412.943373] 30 [ 1412.943373] 48 [ 1412.943373] 81 [ 1412.943373] c2 [ 1412.943375] c0 [ 1412.943375] b8 [ 1412.943375] 01 [ 1412.943376] 00 [ 1412.943376] 48 [ 1412.943376] 03 [ 1412.943377] 14 [ 1412.943377] c5 [ 1412.943377] e0 [ 1412.943378] 17 [ 1412.943378] d5 [ 1412.943378] 93 [ 1412.943378] 4c [ 1412.943379] 89 [ 1412.943379] 02 [ 1412.943379] 41 [ 1412.943380] 8b [ 1412.943380] 40 [ 1412.943380] 08 [ 1412.943381] 85 [ 1412.943381] c0 [ 1412.943381] 75 [ 1412.943382] 0f [ 1412.943382] 0f [ 1412.943382] 1f [ 1412.943383] 44 [ 1412.943383] 00 [ 1412.943383] 00 [ 1412.943384] f3 [ 1412.943384] 90 [ 1412.943384] 41 [ 1412.943384] 8b [ 1412.943385] 40 [ 1412.943385] 08 [ 1412.943385] <85> [ 1412.943386] c0 [ 1412.943386] 74 [ 1412.943386] f6 [ 1412.943387] 4d [ 1412.943387] 8b [ 1412.943387] 08 [ 1412.943388] 4d [ 1412.943388] 85 [ 1412.943388] c9 [ 1412.943388] 74 [ 1412.943389] 04 [ 1412.943389] 41 [ 1412.943389] 0f [ 1412.943390] 18 [ 1412.943390] 09 [ 1412.943390] 8b [ 1412.943391] 17 [ 1412.943391] 0f [ 1412.943391] b7 [ 1412.943391] c2 [ 1412.943392] [ 1413.446571] ipmi_devintf [ 1413.449985] ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw [ 1413.527812] devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1413.553480] CPU: 14 PID: 3041 Comm: mdt01_007 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1413.567743] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1413.580067] task: ffff8852a6e84200 ti: ffff88528ad4c000 task.ti: ffff88528ad4c000 [ 1413.588902] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1413.600464] RSP: 0018:ffff88528ad4fab0 EFLAGS: 00000246 [ 1413.606873] RAX: 0000000000000000 RBX: ffff8892ff380000 RCX: 0000000000710000 [ 1413.615319] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1413.623764] RBP: ffff88528ad4fab0 R08: ffff8892ff39b8c0 R09: 0000000000000000 [ 1413.632212] R10: 0000000000000001 R11: ffff8852a386f800 R12: ffff88528ad4fa78 [ 1413.640657] R13: ffffffff930ec9b1 R14: ffff88528ad4fa58 R15: ffff88528ad4fa88 [ 1413.649104] FS: 0000000000000000(0000) GS:ffff8892ff380000(0000) knlGS:0000000000000000 [ 1413.658617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1413.665513] CR2: 00007fb699688008 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1413.673958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1413.682404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1413.690850] Call Trace: [ 1413.694064] [] queued_spin_lock_slowpath+0xb/0xf [ 1413.701541] [] _raw_spin_lock+0x30/0x40 [ 1413.708153] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1413.715829] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1413.724278] [] ? wake_up_atomic_t+0x30/0x30 [ 1413.731278] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1413.738682] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1413.746957] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1413.754952] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1413.763616] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1413.771504] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1413.778816] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1413.787000] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1413.795933] [] ? __getnstimeofday64+0x3f/0xd0 [ 1413.803144] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1413.812195] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1413.820255] [] ? __wake_up_common_lock+0x91/0xc0 [ 1413.827733] [] ? sched_feat_set+0xf0/0xf0 [ 1413.834554] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1413.842033] [] ? __switch_to+0xce/0x5a0 [ 1413.848661] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1413.857399] [] kthread+0xd1/0xe0 [ 1413.863325] [] ? insert_kthread_work+0x40/0x40 [ 1413.870609] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1413.878378] [] ? insert_kthread_work+0x40/0x40 [ 1413.885662] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1416.778876] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [mdt00_011:4259] [ 1416.787802] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1416.867641] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1416.927922] CPU: 3 PID: 4259 Comm: mdt00_011 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1416.942089] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1416.954414] task: ffff8892c975b180 ti: ffff8892f75f0000 task.ti: ffff8892f75f0000 [ 1416.963248] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1416.974811] RSP: 0018:ffff8892f75f3ab0 EFLAGS: 00000246 [ 1416.981219] RAX: 0000000000000000 RBX: ffff8852ffac0000 RCX: 0000000000190000 [ 1416.989664] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1416.998110] RBP: ffff8892f75f3ab0 R08: ffff8852ffadb8c0 R09: 0000000000000000 [ 1417.006555] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8892f75f3a78 [ 1417.015001] R13: ffffffff930ec9b1 R14: ffff8892f75f3a58 R15: ffff8892f75f3a88 [ 1417.023448] FS: 0000000000000000(0000) GS:ffff8852ffac0000(0000) knlGS:0000000000000000 [ 1417.032962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1417.039857] CR2: 00007ffff1096160 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1417.048303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1417.056747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1417.065194] Call Trace: [ 1417.068408] [] queued_spin_lock_slowpath+0xb/0xf [ 1417.075886] [] _raw_spin_lock+0x30/0x40 [ 1417.082499] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1417.090173] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1417.098147] [] lnet_discover_peer_locked+0x252/0x450 [lnet] [ 1417.106693] [] ? wake_up_atomic_t+0x30/0x30 [ 1417.113692] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1417.121108] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1417.129379] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1417.137359] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1417.146016] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1417.153896] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1417.161206] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1417.169383] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1417.178315] [] ? __getnstimeofday64+0x3f/0xd0 [ 1417.185520] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1417.194567] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1417.202626] [] ? __wake_up_common_lock+0x91/0xc0 [ 1417.210102] [] ? sched_feat_set+0xf0/0xf0 [ 1417.216919] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1417.224398] [] ? __switch_to+0xce/0x5a0 [ 1417.231018] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1417.239757] [] kthread+0xd1/0xe0 [ 1417.245683] [] ? insert_kthread_work+0x40/0x40 [ 1417.252968] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1417.260734] [] ? insert_kthread_work+0x40/0x40 [ 1417.268017] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 d5 93 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 1419.636329] SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(V) show-blocked-tasks(w) dump-ftrace-buffer(z) [ 1420.884773] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mdt01_006:3034] [ 1420.940771] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mdt01_013:4292] [ 1420.893793] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 [ 1420.940772] Modules linked in: [ 1420.940773] osp(OE) [ 1420.940773] mdd(OE) [ 1420.940774] lod(OE) [ 1420.940774] mdt(OE) [ 1420.940775] lfsck(OE) [ 1420.940775] mgc(OE) [ 1420.940775] osd_zfs(OE) [ 1420.940776] lquota(OE) [ 1420.940776] fid(OE) [ 1420.940776] fld(OE) [ 1420.940777] ptlrpc(OE) [ 1420.940777] obdclass(OE) [ 1420.940778] ko2iblnd(OE) [ 1420.940778] lnet(OE) [ 1420.940778] libcfs(OE) [ 1420.940779] iTCO_wdt [ 1420.940779] iTCO_vendor_support [ 1420.940779] rpcrdma [ 1420.940780] ib_iser [ 1420.940780] sb_edac [ 1420.940780] intel_powerclamp [ 1420.940781] coretemp [ 1420.940781] intel_rapl [ 1420.940781] iosf_mbi [ 1420.940782] kvm [ 1420.940782] irqbypass [ 1420.940782] pcspkr [ 1420.940783] joydev [ 1420.940783] lpc_ich [ 1420.940783] i2c_i801 [ 1420.940784] ioatdma [ 1420.940784] ses [ 1420.940784] enclosure [ 1420.940784] sg [ 1420.940785] ipmi_si [ 1420.940785] ipmi_devintf [ 1420.940786] ipmi_msghandler [ 1420.940786] dm_round_robin [ 1420.940786] acpi_cpufreq [ 1420.940787] acpi_power_meter [ 1420.940787] ib_ipoib [ 1420.940787] rdma_ucm [ 1420.940787] ib_umad [ 1420.940788] sch_fq_codel [ 1420.940788] iw_cxgb4 [ 1420.940788] rdma_cm [ 1420.940789] iw_cm [ 1420.940789] ib_cm [ 1420.940789] iw_cxgb3 [ 1420.940790] zfs(POE) [ 1420.940790] binfmt_misc [ 1420.940791] zunicode(POE) [ 1420.940791] zavl(POE) [ 1420.940792] icp(POE) [ 1420.940792] zcommon(POE) [ 1420.940792] znvpair(POE) [ 1420.940793] spl(OE) [ 1420.940793] msr_safe(OE) [ 1420.940794] ip_tables [ 1420.940794] nfsv3 [ 1420.940795] nfs_acl [ 1420.940795] rpcsec_gss_krb5 [ 1420.940796] auth_rpcgss [ 1420.940796] nfsv4 [ 1420.940796] dns_resolver [ 1420.940797] nfs [ 1420.940797] lockd [ 1420.940797] grace [ 1420.940798] fscache [ 1420.940798] overlay(T) [ 1420.940798] ext4 [ 1420.940799] mbcache [ 1420.940799] jbd2 [ 1420.940800] dm_service_time [ 1420.940800] sd_mod [ 1420.940800] crc_t10dif [ 1420.940801] crct10dif_generic [ 1420.940801] mlx5_ib [ 1420.940801] ib_uverbs [ 1420.940802] ib_core [ 1420.940802] be2iscsi [ 1420.940802] bnx2i [ 1420.940803] cnic [ 1420.940803] uio [ 1420.940803] cxgb4i [ 1420.940804] cxgb4 [ 1420.940804] cxgb3i [ 1420.940805] cxgb3 [ 1420.940805] mdio [ 1420.940805] libcxgbi [ 1420.940806] libcxgb [ 1420.940806] qla4xxx [ 1420.940807] iscsi_boot_sysfs [ 1420.940807] 8021q [ 1420.940808] garp [ 1420.940808] mrp [ 1420.940809] stp [ 1420.940809] llc [ 1420.940809] mgag200 [ 1420.940810] i2c_algo_bit [ 1420.940810] drm_kms_helper [ 1420.940810] syscopyarea [ 1420.940811] sysfillrect [ 1420.940811] sysimgblt [ 1420.940811] crct10dif_pclmul [ 1420.940812] fb_sys_fops [ 1420.940812] crct10dif_common [ 1420.940812] ttm [ 1420.940813] crc32_pclmul [ 1420.940813] crc32c_intel [ 1420.940814] mlx5_core [ 1420.940814] ahci [ 1420.940814] ghash_clmulni_intel [ 1420.940815] mxm_wmi [ 1420.940815] aesni_intel [ 1420.940815] mlxfw [ 1420.940816] drm [ 1420.940816] ixgbe(OE) [ 1420.940817] mpt3sas [ 1420.940817] lrw [ 1420.940817] devlink [ 1420.940818] libahci [ 1420.940818] gf128mul [ 1420.940819] dca [ 1420.940819] glue_helper [ 1420.940819] ablk_helper [ 1420.940819] ptp [ 1420.940820] raid_class [ 1420.940820] libata [ 1420.940820] drm_panel_orientation_quirks [ 1420.940821] cryptd [ 1420.940821] scsi_transport_sas [ 1420.940821] pps_core [ 1420.940822] dm_multipath [ 1420.940822] wmi [ 1420.940822] sunrpc [ 1420.940823] dm_mirror [ 1420.940823] dm_region_hash [ 1420.940823] dm_log [ 1420.940824] dm_mod [ 1420.940824] iscsi_tcp [ 1420.940824] libiscsi_tcp [ 1420.940825] libiscsi [ 1420.940825] scsi_transport_iscsi [ 1420.940825] fuse [ 1420.940825] [ 1420.940827] CPU: 29 PID: 4292 Comm: mdt01_013 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1420.940828] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1420.940829] task: ffff88549effe300 ti: ffff8892a6698000 task.ti: ffff8892a6698000 [ 1420.940829] RIP: 0010:[] [ 1420.940832] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1420.940832] RSP: 0018:ffff8892a669bab0 EFLAGS: 00000246 [ 1420.940833] RAX: 0000000000000000 RBX: ffff8892ff540000 RCX: 0000000000e90000 [ 1420.940834] RDX: ffff8892ff25b8c0 RSI: 0000000000490000 RDI: ffff8852f5420480 [ 1420.940834] RBP: ffff8892a669bab0 R08: ffff8892ff55b8c0 R09: 0000000000000000 [ 1420.940835] R10: 0000000000000001 R11: ffff8892af0ef500 R12: ffff8892a669ba78 [ 1420.940836] R13: ffffffff930ec9b1 R14: ffff8892a669ba58 R15: ffff8892a669ba88 [ 1420.940837] FS: 0000000000000000(0000) GS:ffff8892ff540000(0000) knlGS:0000000000000000 [ 1420.940837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1420.940838] CR2: 00007ffff7ffd978 CR3: 0000001206210000 CR4: 00000000003607e0 [ 1420.940839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1420.940839] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1420.940840] Call Trace: [ 1420.940842] [] queued_spin_lock_slowpath+0xb/0xf [ 1420.940844] [] _raw_spin_lock+0x30/0x40 [ 1420.940853] [] cfs_percpt_lock+0xc1/0x110 [libcfs] [ 1420.940863] [] lnet_discover_peer_locked+0xa0/0x450 [lnet] [ 1420.940865] [] ? wake_up_atomic_t+0x30/0x30 [ 1420.940872] [] LNetPrimaryNID+0xd5/0x220 [lnet] [ 1420.940909] [] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [ 1420.940933] [] ptlrpc_send_reply+0x394/0x840 [ptlrpc] [ 1420.940958] [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [ 1420.940982] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1420.941006] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1420.941037] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1420.941064] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1420.941066] [] ? __getnstimeofday64+0x3f/0xd0 [ 1420.941091] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1420.941117] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1420.941119] [] ? __wake_up_common_lock+0x91/0xc0 [ 1420.941120] [] ? sched_feat_set+0xf0/0xf0 [ 1420.941145] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1420.941147] [] ? __switch_to+0xce/0x5a0 [ 1420.941172] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1420.941174] [] kthread+0xd1/0xe0 [ 1420.941175] [] ? insert_kthread_work+0x40/0x40 [ 1420.941177] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1420.941178] [] ? insert_kthread_work+0x40/0x40 [ 1420.941179] Code: [ 1420.941179] 0d [ 1420.941179] 48 [ 1420.941180] 98 [ 1420.941180] 83 [ 1420.941180] e2 [ 1420.941180] 30 [ 1420.941181] 48 [ 1420.941181] 81 [ 1420.941181] c2 [ 1420.941182] c0 [ 1420.941182] b8 [ 1420.941182] 01 [ 1420.941182] 00 [ 1420.941183] 48 [ 1420.941183] 03 [ 1420.941183] 14 [ 1420.941183] c5 [ 1420.941184] e0 [ 1420.941184] 17 [ 1420.941184] d5 [ 1420.941184] 93 [ 1420.941185] 4c [ 1420.941185] 89 [ 1420.941185] 02 [ 1420.941185] 41 [ 1420.941186] 8b [ 1420.941186] 40 [ 1420.941186] 08 [ 1420.941187] 85 [ 1420.941187] c0 [ 1420.941187] 75 [ 1420.941187] 0f [ 1420.941188] 0f [ 1420.941188] 1f [ 1420.941188] 44 [ 1420.941188] 00 [ 1420.941189] 00 [ 1420.941189] f3 [ 1420.941189] 90 [ 1420.941190] 41 [ 1420.941190] 8b [ 1420.941190] 40 [ 1420.941190] 08 [ 1420.941191] <85> [ 1420.941191] c0 [ 1420.941191] 74 [ 1420.941191] f6 [ 1420.941192] 4d [ 1420.941192] 8b [ 1420.941192] 08 [ 1420.941192] 4d [ 1420.941193] 85 [ 1420.941193] c9 [ 1420.941193] 74 [ 1420.941193] 04 [ 1420.941194] 41 [ 1420.941194] 0f [ 1420.941194] 18 [ 1420.941194] 09 [ 1420.941195] 8b [ 1420.941195] 17 [ 1420.941195] 0f [ 1420.941195] b7 [ 1420.941195] c2 [ 1420.941196] [ 1421.444385] rdma_cm [ 1421.447317] iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas [ 1421.526529] pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1421.539488] CPU: 11 PID: 3034 Comm: mdt01_006 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1421.553754] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1421.566080] task: ffff8852a6e83180 ti: ffff88528ac80000 task.ti: ffff88528ac80000 [ 1421.574916] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 1421.586480] RSP: 0018:ffff88528ac83ab0 EFLAGS: 00000246 [ 1421.592890] RAX: 0000000000000000 RBX: ffff8892ff2c0000 RCX: 0000000000590000 [ 1421.592901] SysRq : Trigger a crash [ 1421.592912] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1421.592919] IP: [] sysrq_handle_crash+0x26/0x30 [ 1421.592921] PGD 7f56069067 PUD 7f560ea067 PMD 0 [ 1421.592922] Oops: 0002 [#1] SMP [ 1421.592950] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) iTCO_wdt iTCO_vendor_support rpcrdma ib_iser sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass pcspkr joydev lpc_ich i2c_i801 ioatdma ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler dm_round_robin acpi_cpufreq acpi_power_meter ib_ipoib rdma_ucm ib_umad sch_fq_codel iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 zfs(POE) binfmt_misc zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time sd_mod crc_t10dif crct10dif_generic mlx5_ib ib_uverbs ib_core [ 1421.592970] be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32_pclmul crc32c_intel mlx5_core ahci ghash_clmulni_intel mxm_wmi aesni_intel mlxfw drm ixgbe(OE) mpt3sas lrw devlink libahci gf128mul dca glue_helper ablk_helper ptp raid_class libata drm_panel_orientation_quirks cryptd scsi_transport_sas pps_core dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 1421.592973] CPU: 8 PID: 3044 Comm: mdt01_008 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 1421.592973] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0024.021320181901 02/13/2018 [ 1421.592975] task: ffff8852a6e81080 ti: ffff88528ad20000 task.ti: ffff88528ad20000 [ 1421.592977] RIP: 0010:[] [] sysrq_handle_crash+0x26/0x30 [ 1421.592978] RSP: 0000:ffff8892ff203d80 EFLAGS: 00010006 [ 1421.592979] RAX: ffff88528ad23fd8 RBX: ffffffff93ce75c0 RCX: 0000000000000000 [ 1421.592980] RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000063 [ 1421.592981] RBP: ffff8892ff203d80 R08: ffffffff94002fc0 R09: ffffffff940a0623 [ 1421.592981] R10: 0000000000063280 R11: 0000000000100000 R12: 0000000000000063 [ 1421.592982] R13: 0000000000000001 R14: 0000000000000008 R15: 0000000000000063 [ 1421.592984] FS: 0000000000000000(0000) GS:ffff8892ff200000(0000) knlGS:0000000000000000 [ 1421.592985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1421.592985] CR2: 0000000000000000 CR3: 0000007f49076000 CR4: 00000000003607e0 [ 1421.592986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1421.592987] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1421.592988] Call Trace: [ 1421.592991] [ 1421.592991] [] __handle_sysrq+0x11d/0x180 [ 1421.592993] [] handle_sysrq+0x26/0x30 [ 1421.592997] [] serial8250_rx_chars+0x107/0x210 [ 1421.592999] [] serial8250_handle_irq.part.13+0x6a/0xb0 [ 1421.593000] [] serial8250_default_handle_irq+0x2d/0x30 [ 1421.593002] [] serial8250_interrupt+0x69/0xd0 [ 1421.593007] [] __handle_irq_event_percpu+0x54/0x200 [ 1421.593009] [] handle_irq_event_percpu+0x32/0x80 [ 1421.593011] [] handle_irq_event+0x4c/0x70 [ 1421.593013] [] handle_edge_irq+0x83/0x160 [ 1421.593015] [] handle_irq+0xe4/0x1a0 [ 1421.593018] [] do_IRQ+0x4d/0xf0 [ 1421.593020] [] common_interrupt+0x16a/0x16a [ 1421.593023] [ 1421.593023] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 1421.593025] [] queued_spin_lock_slowpath+0xb/0xf [ 1421.593026] [] _raw_spin_lock+0x30/0x40 [ 1421.593033] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 1421.593039] [] ? cfs_percpt_unlock+0x1a/0xe0 [libcfs] [ 1421.593047] [] lnet_handle_send+0x8f/0x450 [lnet] [ 1421.593054] [] ? lnet_handle_find_routed_path+0x3c/0x520 [lnet] [ 1421.593062] [] lnet_select_pathway+0x56b/0x1200 [lnet] [ 1421.593064] [] ? find_busiest_group+0x144/0x990 [ 1421.593071] [] lnet_send+0x5d/0x1c0 [lnet] [ 1421.593078] [] LNetPut+0x2cc/0xb50 [lnet] [ 1421.593103] [] ptl_send_buf+0x146/0x530 [ptlrpc] [ 1421.593127] [] ptlrpc_send_reply+0x29b/0x840 [ptlrpc] [ 1421.593151] [] ptlrpc_send_error+0x9b/0x1b0 [ptlrpc] [ 1421.593174] [] ptlrpc_error+0x10/0x20 [ptlrpc] [ 1421.593204] [] tgt_request_handle+0x988/0x1570 [ptlrpc] [ 1421.593232] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 1421.593234] [] ? __getnstimeofday64+0x3f/0xd0 [ 1421.593260] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 1421.593284] [] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] [ 1421.593285] [] ? __wake_up_common_lock+0x91/0xc0 [ 1421.593287] [] ? sched_feat_set+0xf0/0xf0 [ 1421.593311] [] ptlrpc_main+0xc49/0x1c50 [ptlrpc] [ 1421.593313] [] ? __switch_to+0xce/0x5a0 [ 1421.593337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [ 1421.593339] [] kthread+0xd1/0xe0 [ 1421.593341] [] ? insert_kthread_work+0x40/0x40 [ 1421.593343] [] ret_from_fork_nospec_begin+0x21/0x21 [ 1421.593344] [] ? insert_kthread_work+0x40/0x40 [ 1421.593363] Code: f7 ff ff eb db 0f 1f 44 00 00 55 48 89 e5 65 48 8b 04 25 b8 0e 01 00 83 a8 44 c0 ff ff 01 c7 05 91 40 7b 00 01 00 00 00 0f ae f8 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 c7 05 0e [ 1421.593366] RIP [] sysrq_handle_crash+0x26/0x30 [ 1421.593366] RSP [ 1421.593367] CR2: 0000000000000000