[ 0.000000] microcode: microcode updated early to revision 0x44, date = 2020-05-27 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 (sthiell@oak-md2-s1) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Mon Dec 14 21:25:04 PST 2020 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.6.1.el7_lustre.pl1.x86_64 root=UUID=18de92bd-59b4-4fc6-a82a-ea4037cbac02 ro transparent_hugepage=madvise iommu=pt nospectre_v2 spectre_v2_user=off crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000067b09fff] usable [ 0.000000] BIOS-e820: [mem 0x0000000067b0a000-0x000000006fb11fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000006fb12000-0x000000007a288fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007a289000-0x000000007af0afff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007af0b000-0x000000007b93afff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000007b93b000-0x000000007bab7fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007bab8000-0x000000007bae8fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007bae9000-0x000000007bafefff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007baff000-0x000000007bafffff] usable [ 0.000000] BIOS-e820: [mem 0x000000007bb00000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feda8000-0x00000000fedabfff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff310000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000207fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x2080000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask 3FFF80000000 write-back [ 0.000000] 1 base 000100000000 mask 3FFF00000000 write-back [ 0.000000] 2 base 000200000000 mask 3FFE00000000 write-back [ 0.000000] 3 base 000400000000 mask 3FFC00000000 write-back [ 0.000000] 4 base 000800000000 mask 3FF800000000 write-back [ 0.000000] 5 base 001000000000 mask 3FF000000000 write-back [ 0.000000] 6 base 002000000000 mask 3FFF80000000 write-back [ 0.000000] 7 base 0000FF000000 mask 3FFFFF000000 write-protect [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] x2apic enabled by BIOS, switching to x2apic ops [ 0.000000] e820: last_pfn = 0x7bb00 max_arch_pfn = 0x400000000 [ 0.000000] Base memory trampoline at [ffff8a5540096000] 96000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x1edb874000, 0x1edb874fff] PGTABLE [ 0.000000] BRK [0x1edb875000, 0x1edb875fff] PGTABLE [ 0.000000] BRK [0x1edb876000, 0x1edb876fff] PGTABLE [ 0.000000] BRK [0x1edb877000, 0x1edb877fff] PGTABLE [ 0.000000] BRK [0x1edb878000, 0x1edb878fff] PGTABLE [ 0.000000] BRK [0x1edb879000, 0x1edb879fff] PGTABLE [ 0.000000] BRK [0x1edb87a000, 0x1edb87afff] PGTABLE [ 0.000000] BRK [0x1edb87b000, 0x1edb87bfff] PGTABLE [ 0.000000] BRK [0x1edb87c000, 0x1edb87cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0x35b53000-0x36da1fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000fe320 00024 (v02 DELL ) [ 0.000000] ACPI: XSDT 000000007bab60e8 000C4 (v01 DELL PE_SC3 00000000 01000013) [ 0.000000] ACPI: FACP 000000007bab2000 000F4 (v04 DELL PE_SC3 00000000 DELL 00000001) [ 0.000000] ACPI: DSDT 000000007ba98000 10F68 (v02 DELL PE_SC3 00000003 DELL 00000001) [ 0.000000] ACPI: FACS 000000007b8d3000 00040 [ 0.000000] ACPI: MCEJ 000000007bab5000 00130 (v01 INTEL 00000001 INTL 0100000D) [ 0.000000] ACPI: WD__ 000000007bab4000 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SLIC 000000007bab3000 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: HPET 000000007bab1000 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: APIC 000000007baaf000 01FFC (v03 DELL PE_SC3 00000000 DELL 00000001) [ 0.000000] ACPI: MCFG 000000007baae000 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: MSCT 000000007baad000 00090 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SLIT 000000007baac000 0006C (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SRAT 000000007baa9000 02330 (v03 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: SSDT 000000007b955000 1424A9 (v02 DELL PE_SC3 00004000 INTL 20121114) [ 0.000000] ACPI: SSDT 000000007b952000 0217F (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: SSDT 000000007b951000 0006E (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: PRAD 000000007b950000 00132 (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: SPCR 000000007bafe000 00050 (v01 00000000 00000000) [ 0.000000] ACPI: DMAR 000000007bafd000 00120 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: HEST 000000007baf1000 0017C (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: BERT 000000007baf0000 00030 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: ERST 000000007baef000 00230 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: EINJ 000000007baee000 00150 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] Setting APIC routing to cluster x2apic. [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x10 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x12 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x14 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x16 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x18 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x1a -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x11 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x13 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x15 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x17 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x19 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x1b -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x107fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NODE_DATA(0) allocated [mem 0x107ffd9000-0x107fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x207ffd8000-0x207fffefff] [ 0.000000] Reserving 168MB of memory at 688MB for crashkernel (System RAM: 130850MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x207fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009bfff] [ 0.000000] node 0: [mem 0x00100000-0x67b09fff] [ 0.000000] node 0: [mem 0x6fb12000-0x7a288fff] [ 0.000000] node 0: [mem 0x7bab8000-0x7bae8fff] [ 0.000000] node 0: [mem 0x7baff000-0x7bafffff] [ 0.000000] node 0: [mem 0x100000000-0x107fffffff] [ 0.000000] node 1: [mem 0x1080000000-0x207fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x107fffffff] [ 0.000000] On node 0 totalpages: 16720462 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3995 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 7243 pages used for memmap [ 0.000000] DMA32 zone: 463539 pages, LIFO batch:31 [ 0.000000] Normal zone: 253952 pages used for memmap [ 0.000000] Normal zone: 16252928 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] On node 1 totalpages: 16777216 [ 0.000000] Normal zone: 262144 pages used for memmap [ 0.000000] Normal zone: 16777216 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x18] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x1a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x19] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x1b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC_NMI (uid[0x00] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x01] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x02] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x03] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x04] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x05] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x06] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x07] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x08] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x09] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x10] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x11] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x12] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x13] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x14] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x15] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x16] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x17] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x18] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x19] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x20] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x21] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x22] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x23] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x24] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x25] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x26] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x27] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x28] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x29] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x30] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x31] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x32] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x33] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x34] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x35] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x36] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x37] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x38] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x39] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x40] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x41] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x42] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x43] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x44] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x45] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x46] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x47] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x48] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x49] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x50] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x51] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x52] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x53] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x54] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x55] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x56] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x57] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x58] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x59] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x60] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x61] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x62] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x63] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x64] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x65] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x66] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x67] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x68] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x69] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x70] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x71] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x72] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x73] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x74] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x75] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x76] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x77] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x78] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x79] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x80] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x81] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x82] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x83] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x84] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x85] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x86] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x87] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x88] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x89] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x90] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x91] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x92] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x93] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x94] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x95] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x96] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x97] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x98] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x99] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa0] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa1] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa2] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa3] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa4] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa5] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa6] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa7] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa8] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa9] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xaa] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xab] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xac] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xad] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xae] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xaf] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb0] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb1] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb2] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb3] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb4] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb5] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb6] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb7] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb8] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb9] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xba] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbb] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbc] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbd] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbe] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbf] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x90] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x91] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x92] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x93] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x94] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x95] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x96] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x97] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x98] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x99] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaa] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xab] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xac] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xad] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xae] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaf] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xba] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbb] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbc] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbd] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbe] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbf] high level lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 216 CPUs, 192 hotplug CPUs [ 0.000000] smpboot: Ignoring 168 unusable CPUs in ACPI table [ 0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x67b0a000-0x6fb11fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7a289000-0x7af0afff] [ 0.000000] PM: Registered nosave memory: [mem 0x7af0b000-0x7b93afff] [ 0.000000] PM: Registered nosave memory: [mem 0x7b93b000-0x7bab7fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7bae9000-0x7bafefff] [ 0.000000] PM: Registered nosave memory: [mem 0x7bb00000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfeda7fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeda8000-0xfedabfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfedac000-0xff30ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff310000-0xffffffff] [ 0.000000] e820: [mem 0x90000000-0xfeda7fff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:216 nr_cpu_ids:216 nr_node_ids:2 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 002 004 006 008 010 012 014 [ 0.000000] pcpu-alloc: [0] 016 018 020 022 024 026 028 030 [ 0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 [ 0.000000] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 [ 0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 [ 0.000000] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 [ 0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 [ 0.000000] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 [ 0.000000] pcpu-alloc: [0] 128 130 132 134 136 138 140 142 [ 0.000000] pcpu-alloc: [0] 144 146 148 150 152 154 156 158 [ 0.000000] pcpu-alloc: [0] 160 162 164 166 168 170 172 174 [ 0.000000] pcpu-alloc: [0] 176 178 180 182 184 186 188 190 [ 0.000000] pcpu-alloc: [0] 192 194 196 198 200 202 204 206 [ 0.000000] pcpu-alloc: [0] 208 210 212 214 --- --- --- --- [ 0.000000] pcpu-alloc: [1] 001 003 005 007 009 011 013 015 [ 0.000000] pcpu-alloc: [1] 017 019 021 023 025 027 029 031 [ 0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 [ 0.000000] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 [ 0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 [ 0.000000] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 [ 0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 [ 0.000000] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 [ 0.000000] pcpu-alloc: [1] 129 131 133 135 137 139 141 143 [ 0.000000] pcpu-alloc: [1] 145 147 149 151 153 155 157 159 [ 0.000000] pcpu-alloc: [1] 161 163 165 167 169 171 173 175 [ 0.000000] pcpu-alloc: [1] 177 179 181 183 185 187 189 191 [ 0.000000] pcpu-alloc: [1] 193 195 197 199 201 203 205 207 [ 0.000000] pcpu-alloc: [1] 209 211 213 215 --- --- --- --- [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 32974254 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.6.1.el7_lustre.pl1.x86_64 root=UUID=18de92bd-59b4-4fc6-a82a-ea4037cbac02 ro transparent_hugepage=madvise iommu=pt nospectre_v2 spectre_v2_user=off crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 5808008k/136314880k available (7792k kernel code, 2324168k absent, 2406588k reserved, 5950k data, 1984k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=216, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=216. [ 0.000000] NR_IRQS:327936 nr_irqs:2968 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 3489660928 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] spurious 8259A interrupt: IRQ7. [ 0.000000] tsc: Detected 3400.037 MHz processor [ 0.000072] Calibrating delay loop (skipped), value calculated using timer frequency.. 6800.07 BogoMIPS (lpj=3400037) [ 0.011948] pid_max: default: 221184 minimum: 1728 [ 0.017915] Security Framework initialized [ 0.022505] SELinux: Initializing. [ 0.026553] SELinux: Starting in permissive mode [ 0.026554] Yama: becoming mindful. [ 0.036553] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.066738] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) [ 0.084323] Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.092304] Mountpoint-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.102187] Initializing cgroup subsys memory [ 0.107092] Initializing cgroup subsys devices [ 0.112060] Initializing cgroup subsys freezer [ 0.117028] Initializing cgroup subsys net_cls [ 0.121996] Initializing cgroup subsys blkio [ 0.126769] Initializing cgroup subsys perf_event [ 0.132052] Initializing cgroup subsys hugetlb [ 0.137020] Initializing cgroup subsys pids [ 0.141697] Initializing cgroup subsys net_prio [ 0.146905] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.153618] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.162213] CPU0: Thermal monitoring enabled (TM1) [ 0.167606] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.173543] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 [ 0.179577] tlb_flushall_shift: 6 [ 0.183327] FEATURE SPEC_CTRL Present [ 0.187423] FEATURE IBPB_SUPPORT Present [ 0.191809] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.202594] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.209695] Spectre V2 : Vulnerable [ 0.213604] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.224305] MDS: Mitigation: Clear CPU buffers [ 0.229728] Freeing SMP alternatives: 28k freed [ 0.236262] ACPI: Core revision 20130517 [ 0.273937] ACPI: All ACPI Tables successfully acquired [ 0.281176] ftrace: allocating 29655 entries in 116 pages [ 0.297241] DMAR: Host address width 46 [ 0.301534] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.307476] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.316323] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.322261] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.331100] DMAR: RMRR base: 0x00000067b0a000 end: 0x0000006fb11fff [ 0.338103] DMAR: RMRR base: 0x0000007ae07000 end: 0x0000007af06fff [ 0.345106] DMAR: ATSR flags: 0x0 [ 0.348814] DMAR: ATSR flags: 0x0 [ 0.352523] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 0.359711] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.366813] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.373913] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 0.379947] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.389757] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode [ 0.398587] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode [ 0.407360] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 0.414097] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.430791] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (fam: 06, model: 3f, stepping: 02) [ 0.441303] TSC deadline timer enabled [ 0.441364] Performance Events: PEBS fmt2+, Haswell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 0.452675] ... version: 3 [ 0.457144] ... bit width: 48 [ 0.461710] ... generic registers: 4 [ 0.466179] ... value mask: 0000ffffffffffff [ 0.472103] ... max period: 00007fffffffffff [ 0.478027] ... fixed-purpose events: 3 [ 0.482496] ... event mask: 000000070000000f [ 0.500966] smpboot: CPU 1 Converting physical 0 to logical die 1 [ 0.590260] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.491101] smpboot: Booting Node 1, Processors #1 OK [ 0.599785] smpboot: Booting Node 0, Processors #2 OK [ 0.609264] smpboot: Booting Node 1, Processors #3 OK [ 0.618401] smpboot: Booting Node 0, Processors #4 OK [ 0.627575] smpboot: Booting Node 1, Processors #5 OK [ 0.636714] smpboot: Booting Node 0, Processors #6 OK [ 0.645886] smpboot: Booting Node 1, Processors #7 OK [ 0.655025] smpboot: Booting Node 0, Processors #8 OK [ 0.664206] smpboot: Booting Node 1, Processors #9 OK [ 0.673345] smpboot: Booting Node 0, Processors #10 OK [ 0.682609] smpboot: Booting Node 1, Processors #11 OK [ 0.691847] smpboot: Booting Node 0, Processors #12 [ 0.699087] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 0.714731] OK [ 0.716682] smpboot: Booting Node 1, Processors #13 OK [ 0.724217] smpboot: Booting Node 0, Processors #14 OK [ 0.731791] smpboot: Booting Node 1, Processors #15 OK [ 0.739327] smpboot: Booting Node 0, Processors #16 OK [ 0.746904] smpboot: Booting Node 1, Processors #17 OK [ 0.754438] smpboot: Booting Node 0, Processors #18 OK [ 0.762010] smpboot: Booting Node 1, Processors #19 OK [ 0.769541] smpboot: Booting Node 0, Processors #20 OK [ 0.777112] smpboot: Booting Node 1, Processors #21 OK [ 0.784645] smpboot: Booting Node 0, Processors #22 OK [ 0.792219] smpboot: Booting Node 1, Processors #23 [ 0.799175] Brought up 24 CPUs [ 0.802786] smpboot: Max logical packages: 18 [ 0.807646] smpboot: Total of 24 processors activated (163252.70 BogoMIPS) [ 1.044534] node 0 initialised, 15429500 pages in 195ms [ 1.051333] node 1 initialised, 15948989 pages in 196ms [ 1.057728] devtmpfs: initialized [ 1.061485] x86/mm: Memory block size: 2048MB [ 1.067710] EVM: security.selinux [ 1.071404] EVM: security.ima [ 1.074711] EVM: security.capability [ 1.078831] PM: Registering ACPI NVS region [mem 0x7af0b000-0x7b93afff] (10682368 bytes) [ 1.088968] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.096639] pinctrl core: initialized pinctrl subsystem [ 1.102609] RTC time: 20:02:42, date: 02/16/21 [ 1.107817] NET: Registered protocol family 16 [ 1.112952] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.121399] ACPI: bus type PCI registered [ 1.125869] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.133385] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.143772] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.151351] PCI: Using configuration type 1 for base access [ 1.157577] PCI: Dell System detected, enabling pci=bfsort. [ 1.163814] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 1.181053] ACPI: Added _OSI(Module Device) [ 1.185720] ACPI: Added _OSI(Processor Device) [ 1.190675] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.195920] ACPI: Added _OSI(Processor Aggregator Device) [ 1.201941] ACPI: Added _OSI(Linux-Dell-Video) [ 1.212301] ACPI: EC: Look up EC in DSDT [ 1.313483] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 1.322774] ACPI: Interpreter enabled [ 1.326866] ACPI: (supports S0 S5) [ 1.330657] ACPI: Using IOAPIC for interrupt routing [ 1.336231] HEST: Table parsing has been initialized. [ 1.341866] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.352075] ACPI: Enabled 2 GPEs in block 00 to 3F [ 1.365782] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff]) [ 1.372389] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.381560] acpi PNP0A03:02: PCIe AER handled by firmware [ 1.387663] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.397369] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.406522] PCI host bridge to bus 0000:ff [ 1.411090] pci_bus 0000:ff: root bus resource [bus ff] [ 1.416926] pci 0000:ff:08.0: [8086:2f80] type 00 class 0x088000 [ 1.416970] pci 0000:ff:08.2: [8086:2f32] type 00 class 0x110100 [ 1.417004] pci 0000:ff:08.3: [8086:2f83] type 00 class 0x088000 [ 1.417038] pci 0000:ff:09.0: [8086:2f90] type 00 class 0x088000 [ 1.417071] pci 0000:ff:09.2: [8086:2f33] type 00 class 0x110100 [ 1.417101] pci 0000:ff:09.3: [8086:2f93] type 00 class 0x088000 [ 1.417137] pci 0000:ff:0b.0: [8086:2f81] type 00 class 0x088000 [ 1.417165] pci 0000:ff:0b.1: [8086:2f36] type 00 class 0x110100 [ 1.417195] pci 0000:ff:0b.2: [8086:2f37] type 00 class 0x110100 [ 1.417225] pci 0000:ff:0c.0: [8086:2fe0] type 00 class 0x088000 [ 1.417255] pci 0000:ff:0c.1: [8086:2fe1] type 00 class 0x088000 [ 1.417283] pci 0000:ff:0c.2: [8086:2fe2] type 00 class 0x088000 [ 1.417313] pci 0000:ff:0c.3: [8086:2fe3] type 00 class 0x088000 [ 1.417341] pci 0000:ff:0c.4: [8086:2fe4] type 00 class 0x088000 [ 1.417370] pci 0000:ff:0c.5: [8086:2fe5] type 00 class 0x088000 [ 1.417398] pci 0000:ff:0c.6: [8086:2fe6] type 00 class 0x088000 [ 1.417427] pci 0000:ff:0c.7: [8086:2fe7] type 00 class 0x088000 [ 1.417456] pci 0000:ff:0f.0: [8086:2ff8] type 00 class 0x088000 [ 1.417485] pci 0000:ff:0f.1: [8086:2ff9] type 00 class 0x088000 [ 1.417514] pci 0000:ff:0f.4: [8086:2ffc] type 00 class 0x088000 [ 1.417545] pci 0000:ff:0f.5: [8086:2ffd] type 00 class 0x088000 [ 1.417574] pci 0000:ff:0f.6: [8086:2ffe] type 00 class 0x088000 [ 1.417606] pci 0000:ff:10.0: [8086:2f1d] type 00 class 0x088000 [ 1.417634] pci 0000:ff:10.1: [8086:2f34] type 00 class 0x110100 [ 1.417666] pci 0000:ff:10.5: [8086:2f1e] type 00 class 0x088000 [ 1.417693] pci 0000:ff:10.6: [8086:2f7d] type 00 class 0x110100 [ 1.417723] pci 0000:ff:10.7: [8086:2f1f] type 00 class 0x088000 [ 1.417751] pci 0000:ff:12.0: [8086:2fa0] type 00 class 0x088000 [ 1.417784] pci 0000:ff:12.1: [8086:2f30] type 00 class 0x110100 [ 1.417816] pci 0000:ff:12.2: [8086:2f70] type 00 class 0x088000 [ 1.417851] pci 0000:ff:13.0: [8086:2fa8] type 00 class 0x088000 [ 1.417883] pci 0000:ff:13.1: [8086:2f71] type 00 class 0x088000 [ 1.417918] pci 0000:ff:13.2: [8086:2faa] type 00 class 0x088000 [ 1.417954] pci 0000:ff:13.3: [8086:2fab] type 00 class 0x088000 [ 1.417988] pci 0000:ff:13.4: [8086:2fac] type 00 class 0x088000 [ 1.418021] pci 0000:ff:13.5: [8086:2fad] type 00 class 0x088000 [ 1.418055] pci 0000:ff:13.6: [8086:2fae] type 00 class 0x088000 [ 1.418086] pci 0000:ff:13.7: [8086:2faf] type 00 class 0x088000 [ 1.418118] pci 0000:ff:14.0: [8086:2fb0] type 00 class 0x088000 [ 1.418151] pci 0000:ff:14.1: [8086:2fb1] type 00 class 0x088000 [ 1.418186] pci 0000:ff:14.2: [8086:2fb2] type 00 class 0x088000 [ 1.418219] pci 0000:ff:14.3: [8086:2fb3] type 00 class 0x088000 [ 1.418253] pci 0000:ff:14.4: [8086:2fbc] type 00 class 0x088000 [ 1.418284] pci 0000:ff:14.5: [8086:2fbd] type 00 class 0x088000 [ 1.418317] pci 0000:ff:14.6: [8086:2fbe] type 00 class 0x088000 [ 1.418348] pci 0000:ff:14.7: [8086:2fbf] type 00 class 0x088000 [ 1.418380] pci 0000:ff:15.0: [8086:2fb4] type 00 class 0x088000 [ 1.418414] pci 0000:ff:15.1: [8086:2fb5] type 00 class 0x088000 [ 1.418449] pci 0000:ff:15.2: [8086:2fb6] type 00 class 0x088000 [ 1.418483] pci 0000:ff:15.3: [8086:2fb7] type 00 class 0x088000 [ 1.418521] pci 0000:ff:16.0: [8086:2f68] type 00 class 0x088000 [ 1.418556] pci 0000:ff:16.6: [8086:2f6e] type 00 class 0x088000 [ 1.418590] pci 0000:ff:16.7: [8086:2f6f] type 00 class 0x088000 [ 1.418621] pci 0000:ff:17.0: [8086:2fd0] type 00 class 0x088000 [ 1.418657] pci 0000:ff:17.4: [8086:2fb8] type 00 class 0x088000 [ 1.418688] pci 0000:ff:17.5: [8086:2fb9] type 00 class 0x088000 [ 1.418720] pci 0000:ff:17.6: [8086:2fba] type 00 class 0x088000 [ 1.418752] pci 0000:ff:17.7: [8086:2fbb] type 00 class 0x088000 [ 1.418789] pci 0000:ff:1e.0: [8086:2f98] type 00 class 0x088000 [ 1.418819] pci 0000:ff:1e.1: [8086:2f99] type 00 class 0x088000 [ 1.418851] pci 0000:ff:1e.2: [8086:2f9a] type 00 class 0x088000 [ 1.418881] pci 0000:ff:1e.3: [8086:2fc0] type 00 class 0x088000 [ 1.418904] pci 0000:ff:1e.4: [8086:2f9c] type 00 class 0x088000 [ 1.418938] pci 0000:ff:1f.0: [8086:2f88] type 00 class 0x088000 [ 1.418971] pci 0000:ff:1f.2: [8086:2f8a] type 00 class 0x088000 [ 1.419036] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f]) [ 1.425640] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.434816] acpi PNP0A03:03: PCIe AER handled by firmware [ 1.440928] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.450635] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.459790] PCI host bridge to bus 0000:7f [ 1.464357] pci_bus 0000:7f: root bus resource [bus 7f] [ 1.470190] pci 0000:7f:08.0: [8086:2f80] type 00 class 0x088000 [ 1.470222] pci 0000:7f:08.2: [8086:2f32] type 00 class 0x110100 [ 1.470254] pci 0000:7f:08.3: [8086:2f83] type 00 class 0x088000 [ 1.470286] pci 0000:7f:09.0: [8086:2f90] type 00 class 0x088000 [ 1.470318] pci 0000:7f:09.2: [8086:2f33] type 00 class 0x110100 [ 1.470347] pci 0000:7f:09.3: [8086:2f93] type 00 class 0x088000 [ 1.470380] pci 0000:7f:0b.0: [8086:2f81] type 00 class 0x088000 [ 1.470406] pci 0000:7f:0b.1: [8086:2f36] type 00 class 0x110100 [ 1.470434] pci 0000:7f:0b.2: [8086:2f37] type 00 class 0x110100 [ 1.470462] pci 0000:7f:0c.0: [8086:2fe0] type 00 class 0x088000 [ 1.470489] pci 0000:7f:0c.1: [8086:2fe1] type 00 class 0x088000 [ 1.470516] pci 0000:7f:0c.2: [8086:2fe2] type 00 class 0x088000 [ 1.470544] pci 0000:7f:0c.3: [8086:2fe3] type 00 class 0x088000 [ 1.470569] pci 0000:7f:0c.4: [8086:2fe4] type 00 class 0x088000 [ 1.470598] pci 0000:7f:0c.5: [8086:2fe5] type 00 class 0x088000 [ 1.470624] pci 0000:7f:0c.6: [8086:2fe6] type 00 class 0x088000 [ 1.470652] pci 0000:7f:0c.7: [8086:2fe7] type 00 class 0x088000 [ 1.470679] pci 0000:7f:0f.0: [8086:2ff8] type 00 class 0x088000 [ 1.470707] pci 0000:7f:0f.1: [8086:2ff9] type 00 class 0x088000 [ 1.470735] pci 0000:7f:0f.4: [8086:2ffc] type 00 class 0x088000 [ 1.470763] pci 0000:7f:0f.5: [8086:2ffd] type 00 class 0x088000 [ 1.470789] pci 0000:7f:0f.6: [8086:2ffe] type 00 class 0x088000 [ 1.470818] pci 0000:7f:10.0: [8086:2f1d] type 00 class 0x088000 [ 1.470844] pci 0000:7f:10.1: [8086:2f34] type 00 class 0x110100 [ 1.470873] pci 0000:7f:10.5: [8086:2f1e] type 00 class 0x088000 [ 1.470898] pci 0000:7f:10.6: [8086:2f7d] type 00 class 0x110100 [ 1.470927] pci 0000:7f:10.7: [8086:2f1f] type 00 class 0x088000 [ 1.470958] pci 0000:7f:12.0: [8086:2fa0] type 00 class 0x088000 [ 1.470987] pci 0000:7f:12.1: [8086:2f30] type 00 class 0x110100 [ 1.471016] pci 0000:7f:12.2: [8086:2f70] type 00 class 0x088000 [ 1.471048] pci 0000:7f:13.0: [8086:2fa8] type 00 class 0x088000 [ 1.471078] pci 0000:7f:13.1: [8086:2f71] type 00 class 0x088000 [ 1.471110] pci 0000:7f:13.2: [8086:2faa] type 00 class 0x088000 [ 1.471140] pci 0000:7f:13.3: [8086:2fab] type 00 class 0x088000 [ 1.471171] pci 0000:7f:13.4: [8086:2fac] type 00 class 0x088000 [ 1.471204] pci 0000:7f:13.5: [8086:2fad] type 00 class 0x088000 [ 1.471236] pci 0000:7f:13.6: [8086:2fae] type 00 class 0x088000 [ 1.471265] pci 0000:7f:13.7: [8086:2faf] type 00 class 0x088000 [ 1.471295] pci 0000:7f:14.0: [8086:2fb0] type 00 class 0x088000 [ 1.471325] pci 0000:7f:14.1: [8086:2fb1] type 00 class 0x088000 [ 1.471357] pci 0000:7f:14.2: [8086:2fb2] type 00 class 0x088000 [ 1.471388] pci 0000:7f:14.3: [8086:2fb3] type 00 class 0x088000 [ 1.471419] pci 0000:7f:14.4: [8086:2fbc] type 00 class 0x088000 [ 1.471449] pci 0000:7f:14.5: [8086:2fbd] type 00 class 0x088000 [ 1.471479] pci 0000:7f:14.6: [8086:2fbe] type 00 class 0x088000 [ 1.471508] pci 0000:7f:14.7: [8086:2fbf] type 00 class 0x088000 [ 1.471541] pci 0000:7f:15.0: [8086:2fb4] type 00 class 0x088000 [ 1.471573] pci 0000:7f:15.1: [8086:2fb5] type 00 class 0x088000 [ 1.471605] pci 0000:7f:15.2: [8086:2fb6] type 00 class 0x088000 [ 1.471637] pci 0000:7f:15.3: [8086:2fb7] type 00 class 0x088000 [ 1.471672] pci 0000:7f:16.0: [8086:2f68] type 00 class 0x088000 [ 1.471705] pci 0000:7f:16.6: [8086:2f6e] type 00 class 0x088000 [ 1.471735] pci 0000:7f:16.7: [8086:2f6f] type 00 class 0x088000 [ 1.471764] pci 0000:7f:17.0: [8086:2fd0] type 00 class 0x088000 [ 1.471796] pci 0000:7f:17.4: [8086:2fb8] type 00 class 0x088000 [ 1.471825] pci 0000:7f:17.5: [8086:2fb9] type 00 class 0x088000 [ 1.471856] pci 0000:7f:17.6: [8086:2fba] type 00 class 0x088000 [ 1.471885] pci 0000:7f:17.7: [8086:2fbb] type 00 class 0x088000 [ 1.471919] pci 0000:7f:1e.0: [8086:2f98] type 00 class 0x088000 [ 1.471947] pci 0000:7f:1e.1: [8086:2f99] type 00 class 0x088000 [ 1.471977] pci 0000:7f:1e.2: [8086:2f9a] type 00 class 0x088000 [ 1.472005] pci 0000:7f:1e.3: [8086:2fc0] type 00 class 0x088000 [ 1.472027] pci 0000:7f:1e.4: [8086:2f9c] type 00 class 0x088000 [ 1.472057] pci 0000:7f:1f.0: [8086:2f88] type 00 class 0x088000 [ 1.472087] pci 0000:7f:1f.2: [8086:2f8a] type 00 class 0x088000 [ 1.473162] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 1.480060] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.489225] acpi PNP0A08:00: PCIe AER handled by firmware [ 1.495291] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] [ 1.503041] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.511584] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.520824] PCI host bridge to bus 0000:00 [ 1.525392] pci_bus 0000:00: root bus resource [io 0x0000-0x03bb window] [ 1.532966] pci_bus 0000:00: root bus resource [io 0x03bc-0x03df window] [ 1.540539] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] [ 1.548113] pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window] [ 1.555687] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.564038] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window] [ 1.572386] pci_bus 0000:00: root bus resource [mem 0x38000000000-0x3bfffffffff window] [ 1.581317] pci_bus 0000:00: root bus resource [bus 00-7e] [ 1.587440] pci 0000:00:00.0: [8086:2f00] type 00 class 0x060000 [ 1.587519] pci 0000:00:01.0: [8086:2f02] type 01 class 0x060400 [ 1.587558] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 1.587672] pci 0000:00:01.0: System wakeup disabled by ACPI [ 1.594011] pci 0000:00:02.0: [8086:2f04] type 01 class 0x060400 [ 1.594050] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 1.594101] pci 0000:00:02.0: System wakeup disabled by ACPI [ 1.600436] pci 0000:00:03.0: [8086:2f08] type 01 class 0x060400 [ 1.600475] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 1.600521] pci 0000:00:03.0: System wakeup disabled by ACPI [ 1.606855] pci 0000:00:03.1: [8086:2f09] type 01 class 0x060400 [ 1.606893] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold [ 1.606922] pci 0000:00:03.1: System wakeup disabled by ACPI [ 1.613256] pci 0000:00:03.2: [8086:2f0a] type 01 class 0x060400 [ 1.613294] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold [ 1.613346] pci 0000:00:03.2: System wakeup disabled by ACPI [ 1.619684] pci 0000:00:05.0: [8086:2f28] type 00 class 0x088000 [ 1.619744] pci 0000:00:05.1: [8086:2f29] type 00 class 0x088000 [ 1.619816] pci 0000:00:05.2: [8086:2f2a] type 00 class 0x088000 [ 1.619875] pci 0000:00:05.4: [8086:2f2c] type 00 class 0x080020 [ 1.619880] pci 0000:00:05.4: reg 0x10: [mem 0x91f04000-0x91f04fff] [ 1.619953] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 [ 1.620007] pci 0000:00:11.0: reg 0x30: [mem 0xffff0000-0xffffffff pref] [ 1.620088] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601 [ 1.620101] pci 0000:00:11.4: reg 0x10: [io 0x4078-0x407f] [ 1.620107] pci 0000:00:11.4: reg 0x14: [io 0x408c-0x408f] [ 1.620114] pci 0000:00:11.4: reg 0x18: [io 0x4070-0x4077] [ 1.620120] pci 0000:00:11.4: reg 0x1c: [io 0x4088-0x408b] [ 1.620127] pci 0000:00:11.4: reg 0x20: [io 0x4040-0x405f] [ 1.620134] pci 0000:00:11.4: reg 0x24: [mem 0x91f01000-0x91f017ff] [ 1.620168] pci 0000:00:11.4: PME# supported from D3hot [ 1.620219] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 [ 1.620233] pci 0000:00:16.0: reg 0x10: [mem 0x3bffff03000-0x3bffff0300f 64bit] [ 1.620284] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 1.620330] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 [ 1.620344] pci 0000:00:16.1: reg 0x10: [mem 0x3bffff02000-0x3bffff0200f 64bit] [ 1.620394] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 1.620449] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 [ 1.620463] pci 0000:00:1a.0: reg 0x10: [mem 0x91f03000-0x91f033ff] [ 1.620534] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 1.620586] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 [ 1.620644] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 1.620702] pci 0000:00:1c.7: [8086:8d1e] type 01 class 0x060400 [ 1.620761] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 1.620796] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 1.627135] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 [ 1.627150] pci 0000:00:1d.0: reg 0x10: [mem 0x91f02000-0x91f023ff] [ 1.627221] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 1.627274] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 [ 1.627404] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601 [ 1.627414] pci 0000:00:1f.2: reg 0x10: [io 0x4068-0x406f] [ 1.627420] pci 0000:00:1f.2: reg 0x14: [io 0x4084-0x4087] [ 1.627425] pci 0000:00:1f.2: reg 0x18: [io 0x4060-0x4067] [ 1.627431] pci 0000:00:1f.2: reg 0x1c: [io 0x4080-0x4083] [ 1.627436] pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f] [ 1.627442] pci 0000:00:1f.2: reg 0x24: [mem 0x91f00000-0x91f007ff] [ 1.627471] pci 0000:00:1f.2: PME# supported from D3hot [ 1.627539] pci 0000:03:00.0: [1000:005f] type 00 class 0x010400 [ 1.627547] pci 0000:03:00.0: reg 0x10: [io 0x3000-0x30ff] [ 1.627553] pci 0000:03:00.0: reg 0x14: [mem 0x91e00000-0x91e0ffff 64bit] [ 1.627558] pci 0000:03:00.0: reg 0x1c: [mem 0x91d00000-0x91dfffff 64bit] [ 1.627565] pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.627624] pci 0000:03:00.0: supports D1 D2 [ 1.627661] pci 0000:00:01.0: PCI bridge to [bus 03] [ 1.633199] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] [ 1.633200] pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] [ 1.633232] pci 0000:04:00.0: [1000:0097] type 00 class 0x010700 [ 1.633241] pci 0000:04:00.0: reg 0x10: [io 0x2000-0x20ff] [ 1.633246] pci 0000:04:00.0: reg 0x14: [mem 0x91c40000-0x91c4ffff 64bit] [ 1.633252] pci 0000:04:00.0: reg 0x1c: [mem 0x91c00000-0x91c3ffff 64bit] [ 1.633259] pci 0000:04:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.633364] pci 0000:04:00.0: supports D1 D2 [ 1.633382] pci 0000:04:00.0: reg 0x174: [mem 0x00000000-0x0000ffff 64bit] [ 1.633383] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 16 VFs) [ 1.644355] pci 0000:04:00.0: reg 0x17c: [mem 0x00000000-0x0003ffff 64bit] [ 1.644356] pci 0000:04:00.0: VF(n) BAR2 space: [mem 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs) [ 1.655374] pci 0000:00:02.0: PCI bridge to [bus 04] [ 1.660911] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] [ 1.660913] pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] [ 1.660984] pci 0000:02:00.0: [14e4:165f] type 00 class 0x020000 [ 1.661000] pci 0000:02:00.0: reg 0x10: [mem 0x91a30000-0x91a3ffff 64bit pref] [ 1.661009] pci 0000:02:00.0: reg 0x18: [mem 0x91a40000-0x91a4ffff 64bit pref] [ 1.661019] pci 0000:02:00.0: reg 0x20: [mem 0x91a50000-0x91a5ffff 64bit pref] [ 1.661025] pci 0000:02:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.661101] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 1.661124] pci 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:03.0 (capable of 15.752 Gb/s with 8 GT/s x2 link) [ 1.676516] pci 0000:02:00.1: [14e4:165f] type 00 class 0x020000 [ 1.676533] pci 0000:02:00.1: reg 0x10: [mem 0x91a00000-0x91a0ffff 64bit pref] [ 1.676542] pci 0000:02:00.1: reg 0x18: [mem 0x91a10000-0x91a1ffff 64bit pref] [ 1.676551] pci 0000:02:00.1: reg 0x20: [mem 0x91a20000-0x91a2ffff 64bit pref] [ 1.676558] pci 0000:02:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.676633] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold [ 1.678465] pci 0000:00:03.0: PCI bridge to [bus 02] [ 1.684005] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] [ 1.684045] pci 0000:01:00.0: [14e4:165f] type 00 class 0x020000 [ 1.684061] pci 0000:01:00.0: reg 0x10: [mem 0x91b30000-0x91b3ffff 64bit pref] [ 1.684070] pci 0000:01:00.0: reg 0x18: [mem 0x91b40000-0x91b4ffff 64bit pref] [ 1.684080] pci 0000:01:00.0: reg 0x20: [mem 0x91b50000-0x91b5ffff 64bit pref] [ 1.684086] pci 0000:01:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.684163] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 1.684186] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:03.1 (capable of 15.752 Gb/s with 8 GT/s x2 link) [ 1.699567] pci 0000:01:00.1: [14e4:165f] type 00 class 0x020000 [ 1.699584] pci 0000:01:00.1: reg 0x10: [mem 0x91b00000-0x91b0ffff 64bit pref] [ 1.699593] pci 0000:01:00.1: reg 0x18: [mem 0x91b10000-0x91b1ffff 64bit pref] [ 1.699603] pci 0000:01:00.1: reg 0x20: [mem 0x91b20000-0x91b2ffff 64bit pref] [ 1.699609] pci 0000:01:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.699684] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold [ 1.701528] pci 0000:00:03.1: PCI bridge to [bus 01] [ 1.707068] pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] [ 1.707090] pci 0000:00:03.2: PCI bridge to [bus 05] [ 1.712656] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 1.718550] pci 0000:07:00.0: [1912:001d] type 01 class 0x060400 [ 1.721336] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold [ 1.722258] pci 0000:07:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:1c.7 (capable of 7.876 Gb/s with 8 GT/s x1 link) [ 1.739784] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] [ 1.745616] pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] [ 1.745619] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.746831] pci 0000:08:00.0: [1912:001d] type 01 class 0x060400 [ 1.750125] pci 0000:08:00.0: PME# supported from D0 D3hot D3cold [ 1.752946] pci 0000:07:00.0: PCI bridge to [bus 08-0b] [ 1.758966] pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] [ 1.759098] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.760407] pci 0000:09:00.0: [1912:001a] type 01 class 0x060400 [ 1.764807] pci 0000:08:00.0: PCI bridge to [bus 09-0a] [ 1.770865] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] [ 1.771021] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.772181] pci 0000:0a:00.0: [102b:0534] type 00 class 0x030000 [ 1.772663] pci 0000:0a:00.0: reg 0x10: [mem 0x90000000-0x90ffffff pref] [ 1.772944] pci 0000:0a:00.0: reg 0x14: [mem 0x91800000-0x91803fff] [ 1.773224] pci 0000:0a:00.0: reg 0x18: [mem 0x91000000-0x917fffff] [ 1.777656] pci 0000:09:00.0: PCI bridge to [bus 0a] [ 1.783426] pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] [ 1.783583] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.784423] pci_bus 0000:00: on NUMA node 0 [ 1.784528] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 *15) [ 1.792718] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 *14 15) [ 1.800914] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) [ 1.809090] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 *6 7 9 10 11 12 14 15) [ 1.817273] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.826732] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.836194] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.845656] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.855270] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 1.862166] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.871335] acpi PNP0A08:01: PCIe AER handled by firmware [ 1.877437] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.887144] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.896358] PCI host bridge to bus 0000:80 [ 1.900926] pci_bus 0000:80: root bus resource [io 0x8000-0xffff window] [ 1.908500] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window] [ 1.916850] pci_bus 0000:80: root bus resource [mem 0x3c000000000-0x3ffffffffff window] [ 1.925781] pci_bus 0000:80: root bus resource [bus 80-fe] [ 1.931905] pci 0000:80:01.0: [8086:2f02] type 01 class 0x060400 [ 1.931950] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold [ 1.932006] pci 0000:80:01.0: System wakeup disabled by ACPI [ 1.938343] pci 0000:80:03.0: [8086:2f08] type 01 class 0x060400 [ 1.938387] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold [ 1.938442] pci 0000:80:03.0: System wakeup disabled by ACPI [ 1.944779] pci 0000:80:05.0: [8086:2f28] type 00 class 0x088000 [ 1.944839] pci 0000:80:05.1: [8086:2f29] type 00 class 0x088000 [ 1.944909] pci 0000:80:05.2: [8086:2f2a] type 00 class 0x088000 [ 1.944968] pci 0000:80:05.4: [8086:2f2c] type 00 class 0x080020 [ 1.944974] pci 0000:80:05.4: reg 0x10: [mem 0xca000000-0xca000fff] [ 1.945063] pci 0000:80:01.0: PCI bridge to [bus 81] [ 1.950681] pci 0000:82:00.0: [15b3:101b] type 00 class 0x020700 [ 1.950828] pci 0000:82:00.0: reg 0x10: [mem 0xc8000000-0xc9ffffff 64bit pref] [ 1.951072] pci 0000:82:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.951624] pci 0000:82:00.0: PME# supported from D3cold [ 1.951826] pci 0000:82:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:80:03.0 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 1.967799] pci 0000:80:03.0: PCI bridge to [bus 82] [ 1.973338] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc9ffffff 64bit pref] [ 1.973346] pci_bus 0000:80: on NUMA node 1 [ 1.973641] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none [ 1.982669] vgaarb: loaded [ 1.985685] vgaarb: bridge control possible 0000:0a:00.0 [ 1.991824] SCSI subsystem initialized [ 1.996021] ACPI: bus type USB registered [ 2.000503] usbcore: registered new interface driver usbfs [ 2.006625] usbcore: registered new interface driver hub [ 2.012723] usbcore: registered new device driver usb [ 2.018668] EDAC MC: Ver: 3.0.0 [ 2.022772] PCI: Using ACPI for IRQ routing [ 2.032822] PCI: pci_cache_line_size set to 64 bytes [ 2.033246] e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff] [ 2.033247] e820: reserve RAM buffer [mem 0x67b0a000-0x67ffffff] [ 2.033248] e820: reserve RAM buffer [mem 0x7a289000-0x7bffffff] [ 2.033249] e820: reserve RAM buffer [mem 0x7bae9000-0x7bffffff] [ 2.033249] e820: reserve RAM buffer [mem 0x7bb00000-0x7bffffff] [ 2.033465] NetLabel: Initializing [ 2.037249] NetLabel: domain hash size = 128 [ 2.042106] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.047653] NetLabel: unlabeled traffic allowed by default [ 2.054054] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.061025] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.069559] amd_nb: Cannot enumerate AMD northbridges [ 2.075283] Switched to clocksource hpet [ 2.084461] pnp: PnP ACPI init [ 2.087875] ACPI: bus type PNP registered [ 2.092610] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.092680] system 00:01: [io 0x0500-0x053f] has been reserved [ 2.099278] system 00:01: [io 0x0400-0x047f] could not be reserved [ 2.106272] system 00:01: [io 0x0540-0x057f] has been reserved [ 2.112879] system 00:01: [io 0x0600-0x061f] has been reserved [ 2.119483] system 00:01: [io 0x0ca0-0x0ca5] has been reserved [ 2.126089] system 00:01: [io 0x0880-0x0883] has been reserved [ 2.132695] system 00:01: [io 0x0800-0x081f] has been reserved [ 2.139302] system 00:01: [mem 0xfeda8000-0xfedcbfff] could not be reserved [ 2.147072] system 00:01: [mem 0xff000000-0xffffffff] could not be reserved [ 2.154842] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved [ 2.162223] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved [ 2.169604] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved [ 2.176985] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved [ 2.184368] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.184497] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.184614] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.184666] system 00:04: [io 0x0ca8] has been reserved [ 2.190597] system 00:04: [io 0x0cac] has been reserved [ 2.196526] system 00:04: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active) [ 2.196577] pnp: PnP ACPI: found 5 devices [ 2.201146] ACPI: bus type PNP unregistered [ 2.211927] pci 0000:03:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.222993] pci 0000:04:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.234058] pci 0000:02:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.245124] pci 0000:02:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.256191] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.267260] pci 0000:01:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.278326] pci 0000:00:11.0: can't claim BAR 6 [mem 0xffff0000-0xffffffff pref]: no compatible bridge window [ 2.289392] pci 0000:82:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.300477] pci 0000:00:03.0: BAR 14: assigned [mem 0x92000000-0x920fffff] [ 2.308151] pci 0000:00:03.1: BAR 14: assigned [mem 0x92100000-0x921fffff] [ 2.315824] pci 0000:00:11.0: BAR 6: assigned [mem 0x91f10000-0x91f1ffff pref] [ 2.323886] pci 0000:03:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.331752] pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.340006] pci 0000:00:01.0: PCI bridge to [bus 03] [ 2.345547] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] [ 2.352347] pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] [ 2.359924] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.367790] pci 0000:04:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.376045] pci 0000:04:00.0: BAR 9: no space for [mem size 0x00400000 64bit] [ 2.384007] pci 0000:04:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit] [ 2.392359] pci 0000:04:00.0: BAR 7: no space for [mem size 0x00100000 64bit] [ 2.400323] pci 0000:04:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit] [ 2.408675] pci 0000:00:02.0: PCI bridge to [bus 04] [ 2.414215] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] [ 2.421015] pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] [ 2.428593] pci 0000:02:00.0: BAR 6: assigned [mem 0x92000000-0x9203ffff pref] [ 2.436654] pci 0000:02:00.1: BAR 6: assigned [mem 0x92040000-0x9207ffff pref] [ 2.444715] pci 0000:00:03.0: PCI bridge to [bus 02] [ 2.450254] pci 0000:00:03.0: bridge window [mem 0x92000000-0x920fffff] [ 2.457829] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] [ 2.466472] pci 0000:01:00.0: BAR 6: assigned [mem 0x92100000-0x9213ffff pref] [ 2.474533] pci 0000:01:00.1: BAR 6: assigned [mem 0x92140000-0x9217ffff pref] [ 2.482594] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.488134] pci 0000:00:03.1: bridge window [mem 0x92100000-0x921fffff] [ 2.495710] pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] [ 2.504354] pci 0000:00:03.2: PCI bridge to [bus 05] [ 2.509898] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 2.515445] pci 0000:09:00.0: PCI bridge to [bus 0a] [ 2.521132] pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] [ 2.528802] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.537647] pci 0000:08:00.0: PCI bridge to [bus 09-0a] [ 2.543626] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] [ 2.551299] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.560140] pci 0000:07:00.0: PCI bridge to [bus 08-0b] [ 2.566098] pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] [ 2.573755] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.582615] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] [ 2.588448] pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] [ 2.596024] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.604668] pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc [ 2.614764] pci_bus 0000:00: resource 4 [io 0x0000-0x03bb window] [ 2.614765] pci_bus 0000:00: resource 5 [io 0x03bc-0x03df window] [ 2.614766] pci_bus 0000:00: resource 6 [io 0x03e0-0x0cf7 window] [ 2.614767] pci_bus 0000:00: resource 7 [io 0x1000-0x7fff window] [ 2.614768] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff window] [ 2.614769] pci_bus 0000:00: resource 9 [mem 0x90000000-0xc7ffbfff window] [ 2.614770] pci_bus 0000:00: resource 10 [mem 0x38000000000-0x3bfffffffff window] [ 2.614771] pci_bus 0000:03: resource 0 [io 0x3000-0x3fff] [ 2.614771] pci_bus 0000:03: resource 1 [mem 0x91d00000-0x91efffff] [ 2.614772] pci_bus 0000:04: resource 0 [io 0x2000-0x2fff] [ 2.614773] pci_bus 0000:04: resource 1 [mem 0x91c00000-0x91cfffff] [ 2.614774] pci_bus 0000:02: resource 1 [mem 0x92000000-0x920fffff] [ 2.614775] pci_bus 0000:02: resource 2 [mem 0x91a00000-0x91afffff 64bit pref] [ 2.614776] pci_bus 0000:01: resource 1 [mem 0x92100000-0x921fffff] [ 2.614777] pci_bus 0000:01: resource 2 [mem 0x91b00000-0x91bfffff 64bit pref] [ 2.614778] pci_bus 0000:07: resource 1 [mem 0x91000000-0x919fffff] [ 2.614779] pci_bus 0000:07: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.614780] pci_bus 0000:08: resource 1 [mem 0x91000000-0x919fffff] [ 2.614780] pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.614781] pci_bus 0000:09: resource 1 [mem 0x91000000-0x918fffff] [ 2.614782] pci_bus 0000:09: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.614783] pci_bus 0000:0a: resource 1 [mem 0x91000000-0x918fffff] [ 2.614784] pci_bus 0000:0a: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.614787] pci 0000:80:03.0: BAR 14: assigned [mem 0xca100000-0xca1fffff] [ 2.622460] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.628002] pci 0000:82:00.0: BAR 6: assigned [mem 0xca100000-0xca1fffff pref] [ 2.636061] pci 0000:80:03.0: PCI bridge to [bus 82] [ 2.641599] pci 0000:80:03.0: bridge window [mem 0xca100000-0xca1fffff] [ 2.649176] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc9ffffff 64bit pref] [ 2.657819] pci_bus 0000:80: resource 4 [io 0x8000-0xffff window] [ 2.657820] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window] [ 2.657821] pci_bus 0000:80: resource 6 [mem 0x3c000000000-0x3ffffffffff window] [ 2.657822] pci_bus 0000:82: resource 1 [mem 0xca100000-0xca1fffff] [ 2.657823] pci_bus 0000:82: resource 2 [mem 0xc8000000-0xc9ffffff 64bit pref] [ 2.657991] NET: Registered protocol family 2 [ 2.663895] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 2.672842] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 2.680452] TCP: Hash tables configured (established 524288 bind 65536) [ 2.687875] TCP: reno registered [ 2.691621] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 2.698968] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 2.707280] NET: Registered protocol family 1 [ 2.734715] pci 0000:0a:00.0: Boot video device [ 2.734883] PCI: CLS 32 bytes, default 64 [ 2.734919] Unpacking initramfs... [ 2.913404] Freeing initrd memory: 18748k freed [ 2.921957] DMAR: [Firmware Bug]: RMRR entry for device 03:00.0 is broken - applying workaround [ 2.931736] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.938924] software IO TLB [mem 0x76289000-0x7a289000] (64MB) mapped at [ffff8a55b6289000-ffff8a55ba288fff] [ 2.950183] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer [ 2.959020] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules [ 2.965430] RAPL PMU: hw unit of domain package 2^-14 Joules [ 2.971744] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 2.983715] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 2.990562] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 2.999543] futex hash table entries: 65536 (order: 10, 4194304 bytes) [ 3.007334] Initialise system trusted keyring [ 3.012258] audit: initializing netlink socket (disabled) [ 3.018290] type=2000 audit(1613505756.497:1): initialized [ 3.042133] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.049227] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.057267] zpool: loaded [ 3.060190] zbud: loaded [ 3.063481] VFS: Disk quotas dquot_6.6.0 [ 3.068098] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.076167] Key type big_key registered [ 3.080449] SELinux: Registering netfilter hooks [ 3.081293] NET: Registered protocol family 38 [ 3.086257] Key type asymmetric registered [ 3.090827] Asymmetric key parser 'x509' registered [ 3.096339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 3.104718] io scheduler noop registered [ 3.109097] io scheduler deadline registered (default) [ 3.114890] io scheduler cfq registered [ 3.119170] io scheduler mq-deadline registered [ 3.124225] io scheduler kyber registered [ 3.129060] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 3.129190] pcieport 0000:00:02.0: irq 26 for MSI/MSI-X [ 3.129302] pcieport 0000:00:03.0: irq 27 for MSI/MSI-X [ 3.129389] pcieport 0000:00:03.1: irq 28 for MSI/MSI-X [ 3.129473] pcieport 0000:00:03.2: irq 29 for MSI/MSI-X [ 3.129583] pcieport 0000:00:1c.0: irq 30 for MSI/MSI-X [ 3.129695] pcieport 0000:00:1c.7: irq 31 for MSI/MSI-X [ 3.132473] pcieport 0000:80:01.0: irq 33 for MSI/MSI-X [ 3.132589] pcieport 0000:80:03.0: irq 34 for MSI/MSI-X [ 3.132650] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 3.140423] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 3.147700] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded [ 3.147710] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 3.155477] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 3.162762] pcie_pme 0000:00:02.0:pcie001: service driver pcie_pme loaded [ 3.162771] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 3.170539] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 3.177823] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt [ 3.185108] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded [ 3.185118] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt [ 3.192886] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 3.200171] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt [ 3.207456] pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded [ 3.207465] pcieport 0000:00:03.2: Signaling PME through PCIe PME interrupt [ 3.215235] pcie_pme 0000:00:03.2:pcie001: service driver pcie_pme loaded [ 3.215253] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 3.223024] pcie_pme 0000:00:1c.0:pcie001: service driver pcie_pme loaded [ 3.223035] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt [ 3.230839] pcieport 0000:07:00.0: Signaling PME through PCIe PME interrupt [ 3.238611] pcieport 0000:08:00.0: Signaling PME through PCIe PME interrupt [ 3.246383] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt [ 3.253668] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt [ 3.260953] pcie_pme 0000:00:1c.7:pcie001: service driver pcie_pme loaded [ 3.271309] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt [ 3.279081] pcie_pme 0000:80:01.0:pcie001: service driver pcie_pme loaded [ 3.279115] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt [ 3.286885] pci 0000:82:00.0: Signaling PME through PCIe PME interrupt [ 3.294170] pcie_pme 0000:80:03.0:pcie001: service driver pcie_pme loaded [ 3.294215] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 3.300438] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 3.308070] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.315605] intel_idle: MWAIT substates: 0x2120 [ 3.315606] intel_idle: v0.4.1 model 0x3F [ 3.315749] intel_idle: lapic_timer_reliable_states 0xffffffff [ 3.315870] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.324129] ACPI: Power Button [PWRF] [ 3.328730] ERST: Error Record Serialization Table (ERST) support is initialized. [ 3.337082] pstore: Registered erst as persistent store backend [ 3.343947] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 3.352270] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.379786] 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.406543] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.413090] Non-volatile memory driver v1.3 [ 3.417782] Linux agpgart interface v0.103 [ 3.422550] crash memory driver: version 1.1 [ 3.427469] rdac: device handler registered [ 3.432162] hp_sw: device handler registered [ 3.436927] emc: device handler registered [ 3.441622] alua: device handler registered [ 3.446342] libphy: Fixed MDIO Bus: probed [ 3.450944] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.458232] ehci-pci: EHCI PCI platform driver [ 3.463330] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 3.469196] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 3.477461] ehci-pci 0000:00:1a.0: debug port 2 [ 3.486415] ehci-pci 0000:00:1a.0: cache line size of 32 is not supported [ 3.486424] ehci-pci 0000:00:1a.0: irq 18, io mem 0x91f03000 [ 3.498252] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 3.504691] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 3.513914] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.521974] usb usb1: Product: EHCI Host Controller [ 3.527416] usb usb1: Manufacturer: Linux 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 ehci_hcd [ 3.536446] usb usb1: SerialNumber: 0000:00:1a.0 [ 3.541676] hub 1-0:1.0: USB hub found [ 3.545863] hub 1-0:1.0: 2 ports detected [ 3.550495] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 3.556356] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 3.564619] ehci-pci 0000:00:1d.0: debug port 2 [ 3.573572] ehci-pci 0000:00:1d.0: cache line size of 32 is not supported [ 3.573575] ehci-pci 0000:00:1d.0: irq 18, io mem 0x91f02000 [ 3.585251] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 3.591679] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 3.600903] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.608963] usb usb2: Product: EHCI Host Controller [ 3.614405] usb usb2: Manufacturer: Linux 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 ehci_hcd [ 3.623435] usb usb2: SerialNumber: 0000:00:1d.0 [ 3.628650] hub 2-0:1.0: USB hub found [ 3.632834] hub 2-0:1.0: 2 ports detected [ 3.637401] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.644305] ohci-pci: OHCI PCI platform driver [ 3.649278] uhci_hcd: USB Universal Host Controller Interface driver [ 3.656434] usbcore: registered new interface driver usbserial_generic [ 3.663725] usbserial: USB Serial support registered for generic [ 3.670456] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 3.983262] tsc: Refined TSC clocksource calibration: 3399.996 MHz [ 4.711325] i8042: No controller found [ 4.715562] Switched to clocksource tsc [ 4.715588] mousedev: PS/2 mouse device common for all mice [ 4.715683] rtc_cmos 00:00: RTC can wake from S4 [ 4.715833] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.715869] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 4.715904] intel_pstate: Intel P-state driver initializing [ 4.720832] cpuidle: using governor menu [ 4.721447] hidraw: raw HID events driver (C) Jiri Kosina [ 4.721669] usbcore: registered new interface driver usbhid [ 4.721669] usbhid: USB HID core driver [ 4.721914] drop_monitor: Initializing network drop monitor service [ 4.722270] TCP: cubic registered [ 4.722279] Initializing XFRM netlink socket [ 4.722652] NET: Registered protocol family 10 [ 4.723794] NET: Registered protocol family 17 [ 4.723807] mpls_gso: MPLS GSO support [ 4.726142] intel_rdt: Intel RDT L3 monitoring detected [ 4.726147] mce: Using 22 MCE banks [ 4.726234] microcode: sig=0x306f2, pf=0x1, revision=0x44 [ 4.731389] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.731567] PM: Hibernation image not present or could not be loaded. [ 4.731572] Loading compiled-in X.509 certificates [ 4.731604] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.731621] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.732534] Loaded X.509 cert 'CentOS Linux kernel signing key: ea54dd7032b69f8db25b962eaa6a9a92fa53e7ed' [ 4.732566] registered taskstats version 1 [ 4.732602] page_owner is disabled [ 4.736926] Key type trusted registered [ 4.740625] Key type encrypted registered [ 4.740691] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 4.742722] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 4.742802] Magic number: 9:261:43 [ 4.746562] rtc_cmos 00:00: setting system clock to 2021-02-16 20:02:45 UTC (1613505765) [ 4.827237] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 4.924502] Freeing unused kernel memory: 1984k freed [ 4.927236] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 4.937965] Write protecting the kernel read-only data: 12288k [ 4.946373] Freeing unused kernel memory: 388k freed [ 4.954048] Freeing unused kernel memory: 536k freed [ 4.956645] usb 1-1: New USB device found, idVendor=8087, idProduct=800a, bcdDevice= 0.05 [ 4.956648] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 4.956937] hub 1-1:1.0: USB hub found [ 4.957014] hub 1-1:1.0: 6 ports detected [ 4.991809] random: systemd: uninitialized urandom read (16 bytes read) [ 5.001926] random: systemd: uninitialized urandom read (16 bytes read) [ 5.009344] random: systemd: uninitialized urandom read (16 bytes read) [ 5.018535] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.038836] systemd[1]: Detected architecture x86-64. [ 5.044481] systemd[1]: Running in initial RAM disk. [ 5.053646] usb 2-1: New USB device found, idVendor=8087, idProduct=8002, bcdDevice= 0.05 [ 5.063957] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.072186] hub 2-1:1.0: USB hub found [ 5.077772] hub 2-1:1.0: 8 ports detected [ 5.085357] systemd[1]: Set hostname to . [ 5.151057] random: systemd: uninitialized urandom read (16 bytes read) [ 5.158480] random: systemd: uninitialized urandom read (16 bytes read) [ 5.165919] random: systemd: uninitialized urandom read (16 bytes read) [ 5.173402] random: systemd: uninitialized urandom read (16 bytes read) [ 5.181872] random: systemd: uninitialized urandom read (16 bytes read) [ 5.189475] random: systemd: uninitialized urandom read (16 bytes read) [ 5.197381] random: systemd: uninitialized urandom read (16 bytes read) [ 5.211038] systemd[1]: Reached target Swap. [ 5.220684] systemd[1]: Created slice Root Slice. [ 5.232452] systemd[1]: Listening on Journal Socket. [ 5.244407] systemd[1]: Listening on udev Control Socket. [ 5.256274] usb 1-1.6: new high-speed USB device number 3 using ehci-pci [ 5.263827] systemd[1]: Reached target Timers. [ 5.273339] systemd[1]: Reached target Local File Systems. [ 5.286369] systemd[1]: Listening on udev Kernel Socket. [ 5.298346] systemd[1]: Reached target Sockets. [ 5.308469] systemd[1]: Created slice System Slice. [ 5.321087] systemd[1]: Starting Journal Service... [ 5.332128] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.353031] systemd[1]: Starting Setup Virtual Console... [ 5.360012] usb 1-1.6: New USB device found, idVendor=413c, idProduct=a001, bcdDevice= 0.00 [ 5.370119] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 5.370120] usb 1-1.6: Product: Gadget USB HUB [ 5.370122] usb 1-1.6: Manufacturer: no manufacturer [ 5.370124] usb 1-1.6: SerialNumber: 0123456789 [ 5.370599] hub 1-1.6:1.0: USB hub found [ 5.370771] hub 1-1.6:1.0: 6 ports detected [ 5.374204] systemd[1]: Starting dracut cmdline hook... [ 5.424286] systemd[1]: Starting Load Kernel Modules... [ 5.435350] systemd[1]: Reached target Slices. [ 5.445802] systemd[1]: Started Journal Service. [ 5.626916] random: fast init done [ 5.672513] pps_core: LinuxPPS API ver. 1 registered [ 5.678494] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 5.692851] PTP clock support registered [ 5.700412] megasas: 07.714.04.00-rh1 [ 5.704893] megaraid_sas 0000:03:00.0: BAR:0x1 BAR's base_addr(phys):0x0000000091e00000 mapped virt_addr:0xffff9d74cc5fc000 [ 5.704898] megaraid_sas 0000:03:00.0: FW now in Ready state [ 5.712638] megaraid_sas 0000:03:00.0: 63 bit DMA mask and 32 bit consistent mask [ 5.712919] mpt3sas: loading out-of-tree module taints kernel. [ 5.713093] megaraid_sas 0000:03:00.0: firmware supports msix : (96) [ 5.713170] megaraid_sas 0000:03:00.0: irq 36 for MSI/MSI-X [ 5.713190] megaraid_sas 0000:03:00.0: irq 37 for MSI/MSI-X [ 5.713211] megaraid_sas 0000:03:00.0: irq 38 for MSI/MSI-X [ 5.713239] megaraid_sas 0000:03:00.0: irq 39 for MSI/MSI-X [ 5.713264] megaraid_sas 0000:03:00.0: irq 40 for MSI/MSI-X [ 5.713291] megaraid_sas 0000:03:00.0: irq 41 for MSI/MSI-X [ 5.713338] megaraid_sas 0000:03:00.0: irq 42 for MSI/MSI-X [ 5.713364] megaraid_sas 0000:03:00.0: irq 43 for MSI/MSI-X [ 5.713389] megaraid_sas 0000:03:00.0: irq 44 for MSI/MSI-X [ 5.713415] megaraid_sas 0000:03:00.0: irq 45 for MSI/MSI-X [ 5.713439] megaraid_sas 0000:03:00.0: irq 46 for MSI/MSI-X [ 5.713464] megaraid_sas 0000:03:00.0: irq 47 for MSI/MSI-X [ 5.713489] megaraid_sas 0000:03:00.0: irq 48 for MSI/MSI-X [ 5.713522] megaraid_sas 0000:03:00.0: irq 49 for MSI/MSI-X [ 5.713550] megaraid_sas 0000:03:00.0: irq 50 for MSI/MSI-X [ 5.713580] megaraid_sas 0000:03:00.0: irq 51 for MSI/MSI-X [ 5.713607] megaraid_sas 0000:03:00.0: irq 52 for MSI/MSI-X [ 5.713634] megaraid_sas 0000:03:00.0: irq 53 for MSI/MSI-X [ 5.713663] megaraid_sas 0000:03:00.0: irq 54 for MSI/MSI-X [ 5.713699] megaraid_sas 0000:03:00.0: irq 55 for MSI/MSI-X [ 5.713727] megaraid_sas 0000:03:00.0: irq 56 for MSI/MSI-X [ 5.713754] megaraid_sas 0000:03:00.0: irq 57 for MSI/MSI-X [ 5.713804] megaraid_sas 0000:03:00.0: irq 58 for MSI/MSI-X [ 5.713831] megaraid_sas 0000:03:00.0: irq 59 for MSI/MSI-X [ 5.713857] megaraid_sas 0000:03:00.0: irq 60 for MSI/MSI-X [ 5.713939] megaraid_sas 0000:03:00.0: requested/available msix 25/25 [ 5.713944] megaraid_sas 0000:03:00.0: current msix/online cpus : (25/24) [ 5.713946] megaraid_sas 0000:03:00.0: RDPQ mode : (disabled) [ 5.713949] megaraid_sas 0000:03:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 [ 5.714285] megaraid_sas 0000:03:00.0: Configured max firmware commands: 927 [ 5.717777] megaraid_sas 0000:03:00.0: Performance mode :Latency [ 5.717778] megaraid_sas 0000:03:00.0: FW supports sync cache : No [ 5.717783] megaraid_sas 0000:03:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 [ 5.803120] mpt3sas: module verification failed: signature and/or required key missing - tainting kernel [ 5.814716] Compat-mlnx-ofed backport release: 354818c [ 5.814717] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 354818c [ 5.814717] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 5.814948] libata version 3.00 loaded. [ 5.821224] tg3.c:v3.137 (May 11, 2014) [ 5.837756] tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:ee:94 [ 5.837759] tg3 0000:01:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.837760] tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.837762] tg3 0000:01:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.847597] tg3 0000:01:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:ee:95 [ 5.847598] tg3 0000:01:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.847599] tg3 0000:01:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.847600] tg3 0000:01:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.857564] tg3 0000:02:00.0 eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:ee:96 [ 5.857565] tg3 0000:02:00.0 eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.857566] tg3 0000:02:00.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.857567] tg3 0000:02:00.0 eth2: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.876369] tg3 0000:02:00.1 eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:ee:97 [ 5.876370] tg3 0000:02:00.1 eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.876372] tg3 0000:02:00.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.876373] tg3 0000:02:00.1 eth3: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.905253] mpt3sas version 26.00.00.00 loaded [ 5.905415] mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131605808 kB) [ 5.982243] mpt3sas_cm0: IOC Number : 0 [ 5.982244] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 5.982327] mpt3sas 0000:04:00.0: irq 66 for MSI/MSI-X [ 5.982339] mpt3sas 0000:04:00.0: irq 67 for MSI/MSI-X [ 5.982351] mpt3sas 0000:04:00.0: irq 68 for MSI/MSI-X [ 5.982363] mpt3sas 0000:04:00.0: irq 69 for MSI/MSI-X [ 5.982395] mpt3sas 0000:04:00.0: irq 70 for MSI/MSI-X [ 5.982408] mpt3sas 0000:04:00.0: irq 71 for MSI/MSI-X [ 5.982420] mpt3sas 0000:04:00.0: irq 74 for MSI/MSI-X [ 5.982431] mpt3sas 0000:04:00.0: irq 75 for MSI/MSI-X [ 5.982442] mpt3sas 0000:04:00.0: irq 76 for MSI/MSI-X [ 5.982454] mpt3sas 0000:04:00.0: irq 77 for MSI/MSI-X [ 5.982465] mpt3sas 0000:04:00.0: irq 78 for MSI/MSI-X [ 5.982476] mpt3sas 0000:04:00.0: irq 79 for MSI/MSI-X [ 5.982487] mpt3sas 0000:04:00.0: irq 80 for MSI/MSI-X [ 5.982499] mpt3sas 0000:04:00.0: irq 81 for MSI/MSI-X [ 5.982510] mpt3sas 0000:04:00.0: irq 82 for MSI/MSI-X [ 5.982522] mpt3sas 0000:04:00.0: irq 83 for MSI/MSI-X [ 5.982533] mpt3sas 0000:04:00.0: irq 84 for MSI/MSI-X [ 5.982544] mpt3sas 0000:04:00.0: irq 85 for MSI/MSI-X [ 5.982555] mpt3sas 0000:04:00.0: irq 86 for MSI/MSI-X [ 5.982567] mpt3sas 0000:04:00.0: irq 87 for MSI/MSI-X [ 5.982578] mpt3sas 0000:04:00.0: irq 88 for MSI/MSI-X [ 5.982590] mpt3sas 0000:04:00.0: irq 89 for MSI/MSI-X [ 5.982602] mpt3sas 0000:04:00.0: irq 90 for MSI/MSI-X [ 5.982613] mpt3sas 0000:04:00.0: irq 91 for MSI/MSI-X [ 5.982862] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 66 [ 5.982863] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 67 [ 5.982864] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 68 [ 5.982864] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 69 [ 5.982865] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 70 [ 5.982865] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 71 [ 5.982865] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 74 [ 5.982866] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 75 [ 5.982866] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 76 [ 5.982866] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 77 [ 5.982867] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 78 [ 5.982867] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 79 [ 5.982867] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 80 [ 5.982867] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 81 [ 5.982868] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 82 [ 5.982868] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 83 [ 5.982868] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 84 [ 5.982868] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 85 [ 5.982869] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 86 [ 5.982869] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 87 [ 5.982869] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 88 [ 5.982869] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 89 [ 5.982870] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 90 [ 5.982870] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 91 [ 5.982871] mpt3sas_cm0: iomem(0x0000000091c40000), mapped(0xffff9d74cd4e0000), size(65536) [ 5.982871] mpt3sas_cm0: ioport(0x0000000000002000), size(256) [ 6.062239] mpt3sas_cm0: IOC Number : 0 [ 6.062240] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 6.102262] megaraid_sas 0000:03:00.0: FW provided supportMaxExtLDs: 0 max_lds: 32 [ 6.102265] megaraid_sas 0000:03:00.0: controller type : iMR(0MB) [ 6.102266] megaraid_sas 0000:03:00.0: Online Controller Reset(OCR) : Enabled [ 6.102268] megaraid_sas 0000:03:00.0: Secure JBOD support : No [ 6.102269] megaraid_sas 0000:03:00.0: NVMe passthru support : No [ 6.102271] megaraid_sas 0000:03:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs [ 6.102272] megaraid_sas 0000:03:00.0: JBOD sequence map support : No [ 6.102274] megaraid_sas 0000:03:00.0: PCI Lane Margining support : No [ 6.125849] megaraid_sas 0000:03:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000 [ 6.125850] megaraid_sas 0000:03:00.0: INIT adapter done [ 6.125851] megaraid_sas 0000:03:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5641 [ 6.152367] megaraid_sas 0000:03:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) [ 6.152368] megaraid_sas 0000:03:00.0: unevenspan support : yes [ 6.152369] megaraid_sas 0000:03:00.0: firmware crash dump : no [ 6.152370] megaraid_sas 0000:03:00.0: JBOD sequence map : disabled [ 6.152419] scsi host0: Avago SAS based MegaRAID driver [ 6.178434] scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.30 PQ: 0 ANSI: 5 [ 6.191622] mpt3sas_cm0: Allocated physical memory: size(29776 kB) [ 6.191623] mpt3sas_cm0: Current Controller Queue Depth(9564), Max Controller Queue Depth(9664) [ 6.191623] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 6.193151] ahci 0000:00:11.4: version 3.0 [ 6.193349] ahci 0000:00:11.4: irq 92 for MSI/MSI-X [ 6.193361] ahci 0000:00:11.4: SSS flag set, parallel bus scan disabled [ 6.205279] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 6.205280] ahci 0000:00:11.4: flags: 64bit ncq stag led clo pio slum part ems apst [ 6.214579] scsi host2: ahci [ 6.214663] scsi host3: ahci [ 6.214747] scsi host4: ahci [ 6.214814] scsi host5: ahci [ 6.214842] ata1: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01100 irq 92 [ 6.214844] ata2: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01180 irq 92 [ 6.214847] ata3: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01200 irq 92 [ 6.214848] ata4: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01280 irq 92 [ 6.215029] ahci 0000:00:1f.2: irq 93 for MSI/MSI-X [ 6.215038] ahci 0000:00:1f.2: SSS flag set, parallel bus scan disabled [ 6.225285] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 6.225286] ahci 0000:00:1f.2: flags: 64bit ncq stag led clo pio slum part ems apst [ 6.240654] scsi host6: ahci [ 6.240735] scsi host7: ahci [ 6.240802] scsi host8: ahci [ 6.240867] scsi host9: ahci [ 6.240938] scsi host10: ahci [ 6.241004] scsi host11: ahci [ 6.241029] ata5: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00100 irq 93 [ 6.241032] ata6: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00180 irq 93 [ 6.241033] ata7: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00200 irq 93 [ 6.241034] ata8: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00280 irq 93 [ 6.241035] ata9: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00300 irq 93 [ 6.241037] ata10: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00380 irq 93 [ 6.324600] mlx5_core 0000:82:00.0: firmware version: 20.29.2002 [ 6.324625] mlx5_core 0000:82:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:80:03.0 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 6.381925] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.01.00), ChipRevision(0x02), BiosVersion(04.00.00.00) [ 6.381928] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 6.381995] mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 [ 6.382019] scsi host1: Fusion MPT SAS Host [ 6.382297] mpt3sas_cm0: sending port enable !! [ 6.522255] ata1: SATA link down (SStatus 0 SControl 300) [ 6.547255] ata5: SATA link down (SStatus 0 SControl 300) [ 6.551290] mlx5_core 0000:82:00.0: irq 95 for MSI/MSI-X [ 6.551331] mlx5_core 0000:82:00.0: irq 96 for MSI/MSI-X [ 6.551358] mlx5_core 0000:82:00.0: irq 97 for MSI/MSI-X [ 6.551398] mlx5_core 0000:82:00.0: irq 98 for MSI/MSI-X [ 6.551424] mlx5_core 0000:82:00.0: irq 99 for MSI/MSI-X [ 6.551452] mlx5_core 0000:82:00.0: irq 100 for MSI/MSI-X [ 6.551478] mlx5_core 0000:82:00.0: irq 101 for MSI/MSI-X [ 6.551505] mlx5_core 0000:82:00.0: irq 102 for MSI/MSI-X [ 6.551530] mlx5_core 0000:82:00.0: irq 103 for MSI/MSI-X [ 6.551557] mlx5_core 0000:82:00.0: irq 104 for MSI/MSI-X [ 6.551583] mlx5_core 0000:82:00.0: irq 105 for MSI/MSI-X [ 6.551609] mlx5_core 0000:82:00.0: irq 106 for MSI/MSI-X [ 6.551635] mlx5_core 0000:82:00.0: irq 107 for MSI/MSI-X [ 6.551660] mlx5_core 0000:82:00.0: irq 108 for MSI/MSI-X [ 6.551686] mlx5_core 0000:82:00.0: irq 109 for MSI/MSI-X [ 6.551734] mlx5_core 0000:82:00.0: irq 110 for MSI/MSI-X [ 6.551760] mlx5_core 0000:82:00.0: irq 111 for MSI/MSI-X [ 6.551785] mlx5_core 0000:82:00.0: irq 112 for MSI/MSI-X [ 6.551814] mlx5_core 0000:82:00.0: irq 113 for MSI/MSI-X [ 6.551841] mlx5_core 0000:82:00.0: irq 114 for MSI/MSI-X [ 6.551869] mlx5_core 0000:82:00.0: irq 115 for MSI/MSI-X [ 6.551894] mlx5_core 0000:82:00.0: irq 116 for MSI/MSI-X [ 6.551919] mlx5_core 0000:82:00.0: irq 117 for MSI/MSI-X [ 6.551945] mlx5_core 0000:82:00.0: irq 118 for MSI/MSI-X [ 6.551971] mlx5_core 0000:82:00.0: irq 119 for MSI/MSI-X [ 6.553179] mlx5_core 0000:82:00.0: Port module event: module 0, Cable plugged [ 6.553441] mlx5_core 0000:82:00.0: mlx5_pcie_event:303:(pid 189): PCIe slot advertised sufficient power (27W). [ 6.558378] mlx5_core 0000:82:00.0: mlx5_fw_tracer_start:810:(pid 162): FWTracer: Ownership granted and active [ 6.677922] sd 0:2:0:0: [sda] 487325696 512-byte logical blocks: (249 GB/232 GiB) [ 6.686467] sd 0:2:0:0: [sda] Write Protect is off [ 6.691816] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 [ 6.691847] sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA [ 6.729236] sda: sda1 sda2 [ 6.732735] sd 0:2:0:0: [sda] Attached SCSI disk [ 6.969226] ata2: SATA link down (SStatus 0 SControl 300) [ 7.280213] ata3: SATA link down (SStatus 0 SControl 300) [ 7.591209] ata4: SATA link down (SStatus 0 SControl 300) [ 7.902204] ata6: SATA link down (SStatus 0 SControl 300) [ 8.213199] ata7: SATA link down (SStatus 0 SControl 300) [ 8.238201] mpt3sas_cm0: hba_port entry: ffff8a564ebf8f80, port: 255 is added to hba_port list [ 8.249170] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aafc0d0), phys(8) [ 8.258678] mpt3sas_cm0: detecting: handle(0x0009), sas_address(0x500a09849de54f14), phy(0) [ 8.268005] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(0) [ 8.274853] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(1) [ 8.282680] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0009), lun(0) [ 8.290253] mpt3sas_cm0: detecting: handle(0x0009), sas_address(0x500a09849de54f14), phy(0) [ 8.299577] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(0) [ 8.307033] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0009), lun(0) [ 8.314356] scsi 1:0:0:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.323461] scsi 1:0:0:0: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.335203] scsi 1:0:0:0: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.343067] scsi 1:0:0:0: enclosure level(0x0000), connector name( ) [ 8.350550] scsi 1:0:0:0: serial_number(021546008063 ) [ 8.356577] scsi 1:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.378312] scsi 1:0:0:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.387426] scsi 1:0:0:1: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.399171] scsi 1:0:0:1: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.407035] scsi 1:0:0:1: enclosure level(0x0000), connector name( ) [ 8.414521] scsi 1:0:0:1: serial_number(021546008063 ) [ 8.420549] scsi 1:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.431171] scsi 1:0:0:1: Mode parameters changed [ 8.450560] scsi 1:0:0:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.459909] scsi 1:0:0:2: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.471653] scsi 1:0:0:2: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.479517] scsi 1:0:0:2: enclosure level(0x0000), connector name( ) [ 8.487001] scsi 1:0:0:2: serial_number(021546008063 ) [ 8.493030] scsi 1:0:0:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.503200] scsi 1:0:0:2: Mode parameters changed [ 8.519431] scsi 1:0:0:3: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.524203] ata8: SATA link down (SStatus 0 SControl 300) [ 8.534564] scsi 1:0:0:3: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.546307] scsi 1:0:0:3: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.554170] scsi 1:0:0:3: enclosure level(0x0000), connector name( ) [ 8.561652] scsi 1:0:0:3: serial_number(021546008063 ) [ 8.567680] scsi 1:0:0:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.577817] scsi 1:0:0:3: Mode parameters changed [ 8.597497] scsi 1:0:0:4: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.606609] scsi 1:0:0:4: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.618354] scsi 1:0:0:4: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.626218] scsi 1:0:0:4: enclosure level(0x0000), connector name( ) [ 8.633702] scsi 1:0:0:4: serial_number(021546008063 ) [ 8.639729] scsi 1:0:0:4: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.649846] scsi 1:0:0:4: Mode parameters changed [ 8.671443] scsi 1:0:0:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 8.680652] scsi 1:0:0:31: SSP: handle(0x0009), sas_addr(0x500a09849de54f14), phy(0), device_name(0x500a09849de54f14) [ 8.692493] scsi 1:0:0:31: enclosure logical id(0x500605b00aafc0d0), slot(3) [ 8.700454] scsi 1:0:0:31: enclosure level(0x0000), connector name( ) [ 8.708034] scsi 1:0:0:31: serial_number(021546008063 ) [ 8.714157] scsi 1:0:0:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.724382] scsi 1:0:0:31: Power-on or device reset occurred [ 8.744762] mpt3sas_cm0: detecting: handle(0x000a), sas_address(0x500a09849dcf7b14), phy(4) [ 8.754090] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(0) [ 8.760944] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(1) [ 8.768572] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000a), lun(0) [ 8.775577] mpt3sas_cm0: detecting: handle(0x000a), sas_address(0x500a09849dcf7b14), phy(4) [ 8.784901] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(0) [ 8.793250] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000a), lun(0) [ 8.800514] scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.809613] scsi 1:0:1:0: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 8.821356] scsi 1:0:1:0: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 8.829201] ata9: SATA link down (SStatus 0 SControl 300) [ 8.835240] scsi 1:0:1:0: enclosure level(0x0000), connector name( ) [ 8.842721] scsi 1:0:1:0: serial_number(021546007824 ) [ 8.848746] scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.870174] scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.879292] scsi 1:0:1:1: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 8.891039] scsi 1:0:1:1: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 8.898905] scsi 1:0:1:1: enclosure level(0x0000), connector name( ) [ 8.906392] scsi 1:0:1:1: serial_number(021546007824 ) [ 8.912421] scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.936486] scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.945599] scsi 1:0:1:2: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 8.957343] scsi 1:0:1:2: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 8.965207] scsi 1:0:1:2: enclosure level(0x0000), connector name( ) [ 8.972693] scsi 1:0:1:2: serial_number(021546007824 ) [ 8.978721] scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.005516] scsi 1:0:1:3: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 9.014651] scsi 1:0:1:3: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 9.026395] scsi 1:0:1:3: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 9.034261] scsi 1:0:1:3: enclosure level(0x0000), connector name( ) [ 9.041748] scsi 1:0:1:3: serial_number(021546007824 ) [ 9.047777] scsi 1:0:1:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.072501] scsi 1:0:1:4: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 9.081613] scsi 1:0:1:4: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 9.093357] scsi 1:0:1:4: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 9.101224] scsi 1:0:1:4: enclosure level(0x0000), connector name( ) [ 9.108709] scsi 1:0:1:4: serial_number(021546007824 ) [ 9.114736] scsi 1:0:1:4: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.134192] ata10: SATA link down (SStatus 0 SControl 300) [ 9.141493] scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 9.150704] scsi 1:0:1:31: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b14), phy(4), device_name(0x500a09849dcf7b14) [ 9.162547] scsi 1:0:1:31: enclosure logical id(0x500605b00aafc0d0), slot(7) [ 9.170509] scsi 1:0:1:31: enclosure level(0x0000), connector name( ) [ 9.178095] scsi 1:0:1:31: serial_number(021546007824 ) [ 9.184220] scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.194451] scsi 1:0:1:31: Power-on or device reset occurred [ 13.369173] mpt3sas_cm0: port enable: SUCCESS [ 13.374585] scsi 1:0:0:0: rdac: LUN 0 (IOSHIP) (owned) [ 13.380584] sd 1:0:0:0: [sdb] 574402560 512-byte logical blocks: (294 GB/273 GiB) [ 13.389313] scsi 1:0:0:1: rdac: LUN 1 (IOSHIP) (owned) [ 13.395202] sd 1:0:0:0: [sdb] Write Protect is off [ 13.400568] sd 1:0:0:0: [sdb] Mode Sense: 83 00 10 08 [ 13.400779] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 13.410348] sd 1:0:0:1: [sdc] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.410671] scsi 1:0:0:2: rdac: LUN 2 (IOSHIP) (unowned) [ 13.410993] sd 1:0:0:2: [sdd] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.410994] sd 1:0:0:2: [sdd] 4096-byte physical blocks [ 13.411521] scsi 1:0:0:3: rdac: LUN 3 (IOSHIP) (owned) [ 13.411767] sd 1:0:0:2: [sdd] Write Protect is off [ 13.411769] sd 1:0:0:2: [sdd] Mode Sense: 83 00 10 08 [ 13.412026] sd 1:0:0:3: [sde] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.412027] sd 1:0:0:3: [sde] 4096-byte physical blocks [ 13.412097] sd 1:0:0:2: [sdd] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.412498] scsi 1:0:0:4: rdac: LUN 4 (IOSHIP) (unowned) [ 13.412861] sd 1:0:0:3: [sde] Write Protect is off [ 13.412864] sd 1:0:0:3: [sde] Mode Sense: 83 00 10 08 [ 13.412964] sd 1:0:0:4: [sdf] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.412966] sd 1:0:0:4: [sdf] 4096-byte physical blocks [ 13.413156] sd 1:0:0:3: [sde] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.413229] scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (unowned) [ 13.413451] sd 1:0:1:0: [sdg] 574402560 512-byte logical blocks: (294 GB/273 GiB) [ 13.413819] sd 1:0:0:4: [sdf] Write Protect is off [ 13.413821] sd 1:0:0:4: [sdf] Mode Sense: 83 00 10 08 [ 13.413853] sd 1:0:0:0: [sdb] Attached SCSI disk [ 13.413891] scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) [ 13.413990] sd 1:0:1:0: [sdg] Write Protect is off [ 13.413992] sd 1:0:1:0: [sdg] Mode Sense: 83 00 10 08 [ 13.414102] sd 1:0:0:4: [sdf] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.414159] sd 1:0:1:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 13.414226] sd 1:0:1:1: [sdh] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.414228] sd 1:0:1:1: [sdh] 4096-byte physical blocks [ 13.414955] scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (owned) [ 13.415351] sd 1:0:1:1: [sdh] Write Protect is off [ 13.415353] sd 1:0:1:1: [sdh] Mode Sense: 83 00 10 08 [ 13.415705] sd 1:0:1:2: [sdi] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.415706] sd 1:0:1:2: [sdi] 4096-byte physical blocks [ 13.415858] sd 1:0:1:1: [sdh] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.416167] scsi 1:0:1:3: rdac: LUN 3 (IOSHIP) (unowned) [ 13.416772] sd 1:0:0:3: [sde] Attached SCSI disk [ 13.417045] sd 1:0:1:2: [sdi] Write Protect is off [ 13.417047] sd 1:0:1:2: [sdi] Mode Sense: 83 00 10 08 [ 13.417099] sd 1:0:1:3: [sdj] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.417101] sd 1:0:1:3: [sdj] 4096-byte physical blocks [ 13.417358] scsi 1:0:1:4: rdac: LUN 4 (IOSHIP) (owned) [ 13.417685] sd 1:0:0:2: [sdd] Attached SCSI disk [ 13.417701] sd 1:0:1:2: [sdi] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.418587] sd 1:0:1:4: [sdk] 7478427648 512-byte logical blocks: (3.82 TB/3.48 TiB) [ 13.418589] sd 1:0:1:4: [sdk] 4096-byte physical blocks [ 13.418684] sd 1:0:1:3: [sdj] Write Protect is off [ 13.418686] sd 1:0:1:3: [sdj] Mode Sense: 83 00 10 08 [ 13.419080] sd 1:0:1:3: [sdj] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.419176] sd 1:0:0:4: [sdf] Attached SCSI disk [ 13.419732] sd 1:0:1:4: [sdk] Write Protect is off [ 13.419734] sd 1:0:1:4: [sdk] Mode Sense: 83 00 10 08 [ 13.420195] sd 1:0:1:4: [sdk] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.421006] sd 1:0:1:0: [sdg] Attached SCSI disk [ 13.423291] sd 1:0:1:1: [sdh] Attached SCSI disk [ 13.423724] sd 1:0:1:2: [sdi] Attached SCSI disk [ 13.424685] sd 1:0:1:4: [sdk] Attached SCSI disk [ 13.424701] sd 1:0:1:3: [sdj] Attached SCSI disk [ 13.741385] sd 1:0:0:1: [sdc] 4096-byte physical blocks [ 13.747639] sd 1:0:0:1: [sdc] Write Protect is off [ 13.752995] sd 1:0:0:1: [sdc] Mode Sense: 83 00 10 08 [ 13.753179] sd 1:0:0:1: [sdc] Write cache: enabled, read cache: disabled, supports DPO and FUA [ 13.765553] sd 1:0:0:1: [sdc] Attached SCSI disk [ 14.108664] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 14.418341] random: crng init done [ 14.497685] systemd-journald[221]: Received SIGTERM from PID 1 (systemd). [ 14.888571] SELinux: Disabled at runtime. [ 14.893184] SELinux: Unregistering netfilter hooks [ 14.934119] type=1404 audit(1613505775.690:2): selinux=0 auid=4294967295 ses=4294967295 [ 15.141953] ip_tables: (C) 2000-2006 Netfilter Core Team [ 15.148648] systemd[1]: Inserted module 'ip_tables' [ 16.103338] EXT4-fs (sda1): re-mounted. Opts: (null) [ 16.156984] systemd-journald[618]: Received request to flush runtime journal from PID 1 [ 16.259799] device-mapper: uevent: version 1.0.3 [ 16.265707] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 16.503867] RPC: Registered named UNIX socket transport module. [ 16.510485] RPC: Registered udp transport module. [ 16.515738] RPC: Registered tcp transport module. [ 16.520988] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 16.575611] type=1305 audit(1613505777.331:3): audit_pid=751 old=0 auid=4294967295 ses=4294967295 res=1 [ 17.540986] ACPI Error: No handler for Region [SYSI] (ffff8a56a1f74af8) [IPMI] (20130517/evregion-162) [ 17.551395] ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) [ 17.559672] ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8a56a1f43600), AE_NOT_EXIST (20130517/psparse-536) [ 17.573007] ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8a56a1f43540), AE_NOT_EXIST (20130517/psparse-536) [ 17.586341] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) [ 17.674238] IPMI message handler: version 39.2 [ 17.680415] ipmi device interface [ 17.689792] ipmi_si: IPMI System Interface driver [ 17.695060] ipmi_si dmi-ipmi-si.0: probing via SMBIOS [ 17.700698] ipmi_platform: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 [ 17.708857] ipmi_si: Adding SMBIOS-specified kcs state machine [ 17.715424] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 [ 17.730498] sd 0:2:0:0: Attached scsi generic sg0 type 0 [ 17.736622] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 17.742698] sd 1:0:0:1: Attached scsi generic sg2 type 0 [ 17.748778] sd 1:0:0:2: Attached scsi generic sg3 type 0 [ 17.755035] sd 1:0:0:3: Attached scsi generic sg4 type 0 [ 17.761348] sd 1:0:0:4: Attached scsi generic sg5 type 0 [ 17.767335] scsi 1:0:0:31: Attached scsi generic sg6 type 0 [ 17.773622] sd 1:0:1:0: Attached scsi generic sg7 type 0 [ 17.779622] sd 1:0:1:1: Attached scsi generic sg8 type 0 [ 17.785694] sd 1:0:1:2: Attached scsi generic sg9 type 0 [ 17.791685] sd 1:0:1:3: Attached scsi generic sg10 type 0 [ 17.793086] ipmi_si dmi-ipmi-si.0: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. [ 17.801237] ipmi_si dmi-ipmi-si.0: Using irq 10 [ 17.810151] ipmi_si dmi-ipmi-si.0: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) [ 17.826109] sd 1:0:1:4: Attached scsi generic sg11 type 0 [ 17.832087] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized [ 17.839014] scsi 1:0:1:31: Attached scsi generic sg12 type 0 [ 17.871600] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 17.948808] device-mapper: multipath round-robin: version 1.2.0 loaded [ 17.976976] input: PC Speaker as /devices/platform/pcspkr/input/input1 [ 18.000621] cryptd: max_cpu_qlen set to 1000 [ 18.016957] AVX2 version of gcm_enc/dec engaged. [ 18.022105] AES CTR mode by8 optimization enabled [ 18.029509] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 18.036749] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 18.084785] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) [ 18.108789] intel_rapl: Found RAPL domain package [ 18.114045] intel_rapl: Found RAPL domain dram [ 18.119006] intel_rapl: DRAM domain energy unit 15300pj [ 18.124861] intel_rapl: RAPL package 0 domain package locked by BIOS [ 18.124886] intel_rapl: Found RAPL domain package [ 18.124893] intel_rapl: Found RAPL domain dram [ 18.124894] intel_rapl: DRAM domain energy unit 15300pj [ 18.124896] intel_rapl: RAPL package 1 domain package locked by BIOS [ 18.164747] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.164763] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.164775] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.164781] EDAC sbridge: Seeking for: PCI ID 8086:2f60 [ 18.164793] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.164799] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.164807] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.164810] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.164816] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.164823] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.164827] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.164833] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.164840] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.164843] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.164849] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.164856] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.164859] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.164865] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.164872] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.164875] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.164882] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.164889] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.164892] EDAC sbridge: Seeking for: PCI ID 8086:2f68 [ 18.164898] EDAC sbridge: Seeking for: PCI ID 8086:2f79 [ 18.164910] EDAC sbridge: Seeking for: PCI ID 8086:2f6a [ 18.164922] EDAC sbridge: Seeking for: PCI ID 8086:2f6b [ 18.164936] EDAC sbridge: Seeking for: PCI ID 8086:2f6c [ 18.164951] EDAC sbridge: Seeking for: PCI ID 8086:2f6d [ 18.164966] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.164973] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.164983] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.164988] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.164995] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.165005] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.165010] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.165019] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.165029] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.165032] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.165041] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.165051] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.165054] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.165064] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.165079] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.165082] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.165092] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.165104] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.165272] EDAC MC0: Giving out device to 'sb_edac.c' 'Haswell SrcID#0_Ha#0': DEV 0000:7f:12.0 [ 18.175093] EDAC MC1: Giving out device to 'sb_edac.c' 'Haswell SrcID#1_Ha#0': DEV 0000:ff:12.0 [ 18.184803] EDAC sbridge: Ver: 1.1.2 [ 18.189098] iTCO_vendor_support: vendor-support=0 [ 18.196521] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 18.202793] iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460) [ 18.210961] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) [ 18.820051] device-mapper: multipath service-time: version 0.3.0 loaded [ 18.827811] device-mapper: table: 253:5: multipath: error getting device [ 18.835300] device-mapper: ioctl: error adding target to table [ 19.114210] Adding 4194300k swap on /dev/sda2. Priority:-2 extents:1 across:4194300k FS [ 20.305344] mlx5_ib: Mellanox Connect-IB Infiniband driver v4.9-2.2.4 [ 20.521869] mlx5_core 0000:82:00.0: slow_pci_heuristic:5720:(pid 2583): Max link speed = 100000, PCI BW = 126016 [ 20.534412] mlx5_core 0000:82:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 20.543684] mlx5_core 0000:82:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 21.092903] tg3 0000:01:00.0: irq 120 for MSI/MSI-X [ 21.092918] tg3 0000:01:00.0: irq 121 for MSI/MSI-X [ 21.092931] tg3 0000:01:00.0: irq 122 for MSI/MSI-X [ 21.092946] tg3 0000:01:00.0: irq 123 for MSI/MSI-X [ 21.092959] tg3 0000:01:00.0: irq 124 for MSI/MSI-X [ 21.217431] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [ 24.772464] tg3 0000:01:00.0 em1: Link is up at 1000 Mbps, full duplex [ 24.779757] tg3 0000:01:00.0 em1: Flow control is off for TX and off for RX [ 24.787529] tg3 0000:01:00.0 em1: EEE is enabled [ 24.792700] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 25.592941] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 29.853051] LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 [ 29.863577] alg: No test for adler32 (adler32-zlib) [ 29.870258] FS-Cache: Loaded [ 29.915414] FS-Cache: Netfs 'nfs' registered for caching [ 29.926075] Key type dns_resolver registered [ 29.961501] NFS: Registering the id_resolver key type [ 29.967148] Key type id_resolver registered [ 29.971818] Key type id_legacy registered [ 30.666284] LNet: 3202:0:(config.c:1641:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down [ 30.678088] LNet: Using FastReg for registration [ 30.691934] LNet: Added LNI 10.0.2.51@o2ib5 [8/256/0/180] [ 34.338874] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 36.735848] Fusion MPT base driver 3.04.20 [ 36.740643] Copyright (c) 1999-2008 LSI Corporation [ 36.748325] Fusion MPT misc device (ioctl) driver 3.04.20 [ 36.754381] mptctl: Registered with Fusion MPT base driver [ 36.762040] mptctl: /dev/mptctl @ (major,minor=10,220) [ 36.795545] mpt2sas version 20.103.01.00 loaded [ 449.891025] Lustre: Lustre: Build Version: 2.12.6_1_g68bbfcf [ 450.285211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.431995] Lustre: MGS: Connection restored to e375495f-5865-6ffc-36ba-6c8897e0044d (at 0@lo) [ 466.081369] Lustre: MGS: Connection restored to 255558b7-a095-4 (at 10.51.1.47@o2ib3) [ 467.097834] Lustre: MGS: Connection restored to 498658bb-cf6b-4 (at 10.49.23.33@o2ib1) [ 467.106677] Lustre: Skipped 64 previous similar messages [ 469.112745] Lustre: MGS: Connection restored to 7dbc8929-de16-4 (at 10.51.4.3@o2ib3) [ 469.121395] Lustre: Skipped 193 previous similar messages [ 473.128535] Lustre: MGS: Connection restored to fa468938-a0b6-4 (at 10.50.10.61@o2ib2) [ 473.137366] Lustre: Skipped 358 previous similar messages [ 481.152244] Lustre: MGS: Connection restored to 7bbdea2f-5233-4 (at 10.49.24.21@o2ib1) [ 481.161084] Lustre: Skipped 448 previous similar messages [ 497.430977] Lustre: MGS: Connection restored to fac224b4-233b-0a80-ef18-9ae5056dd14e (at 10.0.2.226@o2ib5) [ 497.441756] Lustre: Skipped 770 previous similar messages [ 531.079717] Lustre: MGS: Connection restored to 602c9f1a-dfa8-1f71-a1c0-58af7745d5be (at 10.12.4.48@o2ib) [ 531.090409] Lustre: Skipped 20 previous similar messages [ 595.370831] Lustre: MGS: Connection restored to ce9b189a-3eb4-3ad8-85df-636933fa82dd (at 10.12.4.56@o2ib) [ 595.381517] Lustre: Skipped 52 previous similar messages [ 733.245407] Lustre: MGS: Connection restored to 01cb8f38-fc19-4 (at 10.50.1.21@o2ib2) [ 733.254153] Lustre: Skipped 7 previous similar messages [ 1006.311898] Lustre: MGS: Connection restored to 330ef120-b053-4 (at 10.50.14.10@o2ib2) [ 4968.714541] Lustre: MGS: Connection restored to f9159938-76e4-4 (at 10.51.4.34@o2ib3) [ 4968.723306] Lustre: Skipped 2 previous similar messages [ 6592.782352] Lustre: MGS: haven't heard from client c12436d3-c71c-4 (at 10.49.19.6@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7569ebd800, cur 1613512354 expire 1613512204 last 1613512127 [ 6770.205713] Lustre: MGS: Connection restored to ecab1bd6-353e-4 (at 10.50.9.2@o2ib2) [ 6939.325167] Lustre: MGS: Connection restored to 57da2a01-98d1-4 (at 10.49.18.22@o2ib1) [ 7796.161687] Lustre: MGS: Connection restored to c12436d3-c71c-4 (at 10.49.19.6@o2ib1) [ 8464.979048] Lustre: MGS: Connection restored to 3ee56a8e-8d7e-4 (at 10.51.2.40@o2ib3) [14445.649514] Lustre: MGS: haven't heard from client ee9f226c-4ba4-4 (at 10.49.21.13@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7569902000, cur 1613520207 expire 1613520057 last 1613519980 [14598.361751] Lustre: MGS: Connection restored to ee9f226c-4ba4-4 (at 10.49.21.13@o2ib1) [16308.658857] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [16356.684981] Lustre: MGS: haven't heard from client f102d413-113d-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756c912000, cur 1613522118 expire 1613521968 last 1613521891 [16788.141779] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [16836.601498] Lustre: MGS: haven't heard from client bd5129a9-10be-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a6569391400, cur 1613522598 expire 1613522448 last 1613522371 [17018.621936] Lustre: MGS: haven't heard from client d262bd9a-a823-4 (at 10.49.18.24@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756f874400, cur 1613522780 expire 1613522630 last 1613522553 [17731.941088] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [17767.584598] Lustre: MGS: haven't heard from client 70c988cd-a887-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef1000, cur 1613523529 expire 1613523379 last 1613523302 [18247.041218] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [18284.574113] Lustre: MGS: haven't heard from client 2fc88010-fc95-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568e97c00, cur 1613524046 expire 1613523896 last 1613523819 [19936.546753] Lustre: MGS: haven't heard from client e32a4710-0244-4 (at 10.49.20.5@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756c914400, cur 1613525698 expire 1613525548 last 1613525471 [26996.425660] Lustre: MGS: haven't heard from client 6fc873a2-e777-fc3e-4525-e031089fa129 (at 10.210.12.83@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7570566c00, cur 1613532758 expire 1613532608 last 1613532531 [27072.431473] Lustre: MGS: haven't heard from client 1656bec9-ec37-aff9-fe9c-53d9ceea27ad (at 10.210.12.121@tcp1) in 151 seconds. I think it's dead, and I am evicting it. exp ffff8a756b11bc00, cur 1613532834 expire 1613532684 last 1613532683 [27072.455164] Lustre: Skipped 3 previous similar messages [27369.339927] Lustre: MGS: Connection restored to 11b3e8d6-e38e-3710-81a4-70bb507ed70f (at 10.210.12.91@tcp1) [27401.283050] Lustre: MGS: Connection restored to 6fc873a2-e777-fc3e-4525-e031089fa129 (at 10.210.12.83@tcp1) [27436.609764] Lustre: MGS: Connection restored to c424665c-82b5-429e-d276-a7faff8a221b (at 10.210.12.92@tcp1) [27530.435777] Lustre: MGS: haven't heard from client dda3c7dd-f983-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568e91000, cur 1613533292 expire 1613533142 last 1613533065 [27548.184938] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [27603.224735] Lustre: MGS: Connection restored to 4857ca02-99df-7275-3122-21a1c537a2a7 (at 10.210.12.88@tcp1) [27728.927433] Lustre: MGS: Connection restored to 1656bec9-ec37-aff9-fe9c-53d9ceea27ad (at 10.210.12.121@tcp1) [28126.149708] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [28176.401363] Lustre: MGS: haven't heard from client 71bc29c5-44ac-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568e95000, cur 1613533938 expire 1613533788 last 1613533711 [28490.405496] Lustre: MGS: haven't heard from client 07ac598f-e373-2968-064d-f329712e5ad6 (at 10.210.12.13@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756aa4d400, cur 1613534252 expire 1613534102 last 1613534025 [28900.932177] Lustre: MGS: Connection restored to be340ff1-9eeb-2c89-bb23-b446718c8317 (at 10.210.12.15@tcp1) [28998.169123] Lustre: MGS: Connection restored to 07ac598f-e373-2968-064d-f329712e5ad6 (at 10.210.12.13@tcp1) [28998.180001] Lustre: Skipped 1 previous similar message [29096.996049] Lustre: MGS: Connection restored to 5e2a0c0d-8590-4f36-34a1-504c6a5bdfee (at 10.210.12.27@tcp1) [30872.787077] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [30912.354780] Lustre: MGS: haven't heard from client cc03fe68-eb32-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568efc400, cur 1613536674 expire 1613536524 last 1613536447 [30912.376338] Lustre: Skipped 3 previous similar messages [31474.922798] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [31501.340992] Lustre: MGS: haven't heard from client f47bad55-4ce4-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568eff800, cur 1613537263 expire 1613537113 last 1613537036 [31947.703740] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [32003.334944] Lustre: MGS: haven't heard from client b61e7590-e758-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef4800, cur 1613537765 expire 1613537615 last 1613537538 [32348.046635] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [32375.354348] Lustre: MGS: haven't heard from client b0e38118-469f-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef3c00, cur 1613538137 expire 1613537987 last 1613537910 [37504.290715] Lustre: MGS: haven't heard from client f67bae08-5526-3a58-7db6-c47601c05090 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7569010400, cur 1613543266 expire 1613543116 last 1613543039 [37547.969885] Lustre: MGS: Connection restored to 38f23d47-ade5-1ac9-4ffe-91b5ece5e6e6 (at 10.12.4.68@o2ib) [37552.686372] Lustre: MGS: Connection restored to f67bae08-5526-3a58-7db6-c47601c05090 (at 10.12.4.67@o2ib) [37805.612849] Lustre: MGS: Connection restored to 0d490c46-4924-50c6-5410-04de041ac36a (at 10.12.4.49@o2ib) [47453.454760] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [47502.082038] Lustre: MGS: haven't heard from client 39238c5b-d0ce-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a6569390400, cur 1613553264 expire 1613553114 last 1613553037 [47502.103676] Lustre: Skipped 2 previous similar messages [49143.220761] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [49185.037874] Lustre: MGS: haven't heard from client 348a03a0-e999-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a5652b94400, cur 1613554947 expire 1613554797 last 1613554720 [51281.391156] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [51327.019250] Lustre: MGS: haven't heard from client b9464469-b5e7-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a65610c1000, cur 1613557089 expire 1613556939 last 1613556862 [51912.231462] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [51934.982484] Lustre: MGS: haven't heard from client c7ea310b-f1cc-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a56b7f79800, cur 1613557697 expire 1613557547 last 1613557470 [52558.319979] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [52565.996150] Lustre: MGS: haven't heard from client d630ede3-7ba5-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a65693b1800, cur 1613558328 expire 1613558178 last 1613558101 [53439.283522] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [53487.956676] Lustre: MGS: haven't heard from client 2ef0a3cd-51ea-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef9800, cur 1613559250 expire 1613559100 last 1613559023 [54179.809187] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [54217.951460] Lustre: MGS: haven't heard from client 7932f69d-adf1-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a65693a4000, cur 1613559980 expire 1613559830 last 1613559753 [55000.926404] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [55033.932536] Lustre: MGS: haven't heard from client 6c16b1be-bb3e-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a567bbcf800, cur 1613560796 expire 1613560646 last 1613560569 [55447.431222] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [55503.963247] Lustre: MGS: haven't heard from client 06c8630d-6ff6-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a567bbcf400, cur 1613561266 expire 1613561116 last 1613561039 [69948.416301] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [69998.670102] Lustre: MGS: haven't heard from client 3ebbe192-d81b-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a5654ff2800, cur 1613575761 expire 1613575611 last 1613575534 [70307.696684] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [70350.659799] Lustre: MGS: haven't heard from client d842b86c-7158-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568efc400, cur 1613576113 expire 1613575963 last 1613575886 [70678.004291] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [70735.672920] Lustre: MGS: haven't heard from client 90c00a63-a51b-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef5800, cur 1613576498 expire 1613576348 last 1613576271 [73135.590893] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [73162.609877] Lustre: MGS: haven't heard from client af341a4b-8955-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568e97400, cur 1613578925 expire 1613578775 last 1613578698 [73674.878370] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [73713.625365] Lustre: MGS: haven't heard from client 6c756c8d-d2cf-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef4800, cur 1613579476 expire 1613579326 last 1613579249 [74041.658764] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [74102.594682] Lustre: MGS: haven't heard from client 5da4db5f-2ef1-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568efc000, cur 1613579865 expire 1613579715 last 1613579638 [74431.627829] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [74468.628353] Lustre: MGS: haven't heard from client 99e67516-ac54-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756ffcd000, cur 1613580231 expire 1613580081 last 1613580004 [74820.317962] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [74883.604878] Lustre: MGS: haven't heard from client c39cfeb7-32cc-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a75695ec800, cur 1613580646 expire 1613580496 last 1613580419 [75268.263121] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [75348.592315] Lustre: MGS: haven't heard from client 9c60eaec-bd2c-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef2800, cur 1613581111 expire 1613580961 last 1613580884 [75685.088315] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [75720.567152] Lustre: MGS: haven't heard from client e0dd62fa-9669-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef5800, cur 1613581483 expire 1613581333 last 1613581256 [75743.195918] Lustre: MGS: Connection restored to 13900445-1a74-2a16-b763-5987a0de582e (at 10.0.2.102@o2ib5) [76107.377622] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [76162.584733] Lustre: MGS: haven't heard from client 2d1b5df9-a5c9-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7568ef3c00, cur 1613581925 expire 1613581775 last 1613581698 [76556.163679] Lustre: MGS: Connection restored to f102d413-113d-4 (at 10.49.27.23@o2ib1) [76610.551067] Lustre: MGS: haven't heard from client 5df3ff03-2362-4 (at 10.49.27.23@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a5654ff0800, cur 1613582373 expire 1613582223 last 1613582146 [77119.708414] Lustre: MGS: Received new LWP connection from 10.210.12.17@tcp1, removing former export from same NID [77119.719881] Lustre: MGS: Connection restored to 9aa453fe-e7b2-faac-bf7c-a1a9e4b3965b (at 10.210.12.17@tcp1) [77146.154318] Lustre: MGS: Received new LWP connection from 10.50.8.19@o2ib2, removing former export from same NID [77147.905032] Lustre: MGS: Received new LWP connection from 10.50.12.2@o2ib2, removing former export from same NID [77150.045990] Lustre: MGS: Received new LWP connection from 10.49.24.14@o2ib1, removing former export from same NID [77150.057451] Lustre: Skipped 3 previous similar messages [77154.233740] Lustre: MGS: Received new LWP connection from 10.50.10.65@o2ib2, removing former export from same NID [77154.245198] Lustre: Skipped 14 previous similar messages [77166.391706] Lustre: MGS: Received new LWP connection from 10.49.24.7@o2ib1, removing former export from same NID [77166.403057] Lustre: Skipped 15 previous similar messages [77184.259293] Lustre: MGS: Received new LWP connection from 10.50.1.2@o2ib2, removing former export from same NID [77184.270557] Lustre: Skipped 23 previous similar messages [77188.961199] LNetError: 3229:0:(o2iblnd_cb.c:3359:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [77188.972460] LNetError: 3229:0:(o2iblnd_cb.c:3434:kiblnd_check_conns()) Timed out RDMA with 10.0.2.101@o2ib5 (113): c: 7, oc: 0, rc: 8 [77216.958856] Lustre: MGS: Received new LWP connection from 10.50.4.25@o2ib2, removing former export from same NID [77216.970218] Lustre: Skipped 74 previous similar messages [77277.823103] Lustre: MGS: Export ffff8a757050ec00 already connecting from 10.49.26.16@o2ib1 [77281.361411] Lustre: MGS: Received new LWP connection from 10.50.17.6@o2ib2, removing former export from same NID [77281.372776] Lustre: Skipped 562 previous similar messages [77303.553509] Lustre: MGS: haven't heard from client 2d7fead2-0d8d-22d4-9415-4c869b5ad93f (at 10.0.2.101@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a7569954000, cur 1613583066 expire 1613582916 last 1613582839 [77412.709340] LustreError: 3858:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1613582875, 300s ago); not entering recovery in server code, just going back to sleep ns: MGS lock: ffff8a756a714ec0/0xec507abe3a9b477f lrc: 3/0,1 mode: --/EX res: [0x6b616f:0x2:0x0].0x0 rrc: 2877 type: PLN flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 3858 timeout: 0 lvb_type: 0 [77412.750044] LustreError: dumping log to /tmp/lustre-log.1613583175.3858 [77417.700695] Lustre: MGS: Received new LWP connection from 10.210.12.77@tcp1, removing former export from same NID [77417.712161] Lustre: Skipped 797 previous similar messages [77631.785364] Lustre: MGS: Connection restored to 357ef629-6e99-4 (at 10.49.20.6@o2ib1) [77631.794108] Lustre: Skipped 3355 previous similar messages [77720.761958] Lustre: MGS: Received new LWP connection from 10.0.2.52@o2ib5, removing former export from same NID [77720.773224] Lustre: Skipped 1916 previous similar messages [78231.799468] Lustre: MGS: Connection restored to dce72246-d706-4 (at 10.50.6.19@o2ib2) [78231.808211] Lustre: Skipped 3535 previous similar messages [78232.770746] Lustre: MGS: Received new LWP connection from 10.50.16.8@o2ib2, removing former export from same NID [78232.782106] Lustre: Skipped 3456 previous similar messages [78369.523138] Lustre: MGS: haven't heard from client 1ba86e31-94a4-4 (at 10.51.2.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8a756fc05800, cur 1613584132 expire 1613583982 last 1613583905 [78369.544603] Lustre: Skipped 41 previous similar messages [78831.803356] Lustre: MGS: Connection restored to 286297f6-90b9-4 (at 10.50.9.27@o2ib2) [78831.812095] Lustre: Skipped 3501 previous similar messages [78832.858725] Lustre: MGS: Received new LWP connection from 10.49.26.16@o2ib1, removing former export from same NID [78832.870198] Lustre: Skipped 3496 previous similar messages [79431.813815] Lustre: MGS: Connection restored to 9452686f-c724-4 (at 10.51.1.54@o2ib3) [79431.822561] Lustre: Skipped 3698 previous similar messages [79432.902188] Lustre: MGS: Received new LWP connection from 10.50.9.46@o2ib2, removing former export from same NID [79432.913551] Lustre: Skipped 3692 previous similar messages [80032.393308] Lustre: MGS: Connection restored to 7dbc8929-de16-4 (at 10.51.4.3@o2ib3) [80032.401957] Lustre: Skipped 3788 previous similar messages [80033.123510] Lustre: MGS: Received new LWP connection from 10.49.23.29@o2ib1, removing former export from same NID [80033.134969] Lustre: Skipped 3779 previous similar messages [80636.262031] Lustre: MGS: Received new LWP connection from 10.50.6.14@o2ib2, removing former export from same NID [80636.273392] Lustre: Skipped 3823 previous similar messages [80636.279532] Lustre: MGS: Connection restored to 0bbf5d30-33d6-4 (at 10.50.6.14@o2ib2) [80636.288272] Lustre: Skipped 3824 previous similar messages [81236.868225] Lustre: MGS: Received new LWP connection from 10.50.15.4@o2ib2, removing former export from same NID [81236.879591] Lustre: Skipped 3825 previous similar messages [81236.885718] Lustre: MGS: Connection restored to 6e4f243b-fd5c-4 (at 10.50.15.4@o2ib2) [81236.894459] Lustre: Skipped 3825 previous similar messages [81839.005986] Lustre: MGS: Received new LWP connection from 10.50.8.65@o2ib2, removing former export from same NID [81839.017350] Lustre: Skipped 3822 previous similar messages [81839.023487] Lustre: MGS: Connection restored to 0574cc4e-3e2b-4 (at 10.50.8.65@o2ib2) [81839.032229] Lustre: Skipped 3822 previous similar messages [82439.020928] Lustre: MGS: Received new LWP connection from 10.51.6.13@o2ib3, removing former export from same NID [82439.032282] Lustre: Skipped 3826 previous similar messages [82439.038420] Lustre: MGS: Connection restored to 7eb11c4d-aa71-4 (at 10.51.6.13@o2ib3) [82439.047158] Lustre: Skipped 3826 previous similar messages [83040.118471] Lustre: MGS: Received new LWP connection from 10.51.4.46@o2ib3, removing former export from same NID [83040.129833] Lustre: Skipped 3824 previous similar messages [83040.135963] Lustre: MGS: Connection restored to c46ec619-0069-4 (at 10.51.4.46@o2ib3) [83040.144705] Lustre: Skipped 3824 previous similar messages [83643.323957] Lustre: MGS: Received new LWP connection from 10.51.1.57@o2ib3, removing former export from same NID [83643.335323] Lustre: Skipped 3827 previous similar messages [83643.341452] Lustre: MGS: Connection restored to 63f6655e-1d73-4 (at 10.51.1.57@o2ib3) [83643.350191] Lustre: Skipped 3827 previous similar messages [84244.542519] Lustre: MGS: Received new LWP connection from 10.50.9.16@o2ib2, removing former export from same NID [84244.553883] Lustre: Skipped 3829 previous similar messages [84244.560011] Lustre: MGS: Connection restored to 3123d0fe-0bec-4 (at 10.50.9.16@o2ib2) [84244.568751] Lustre: Skipped 3828 previous similar messages [84846.773376] Lustre: MGS: Received new LWP connection from 10.50.9.35@o2ib2, removing former export from same NID [84846.784748] Lustre: Skipped 3825 previous similar messages [84846.790873] Lustre: MGS: Connection restored to ef42cbc4-775d-4 (at 10.50.9.35@o2ib2) [84846.799613] Lustre: Skipped 3825 previous similar messages [85453.499050] Lustre: MGS: Received new LWP connection from 10.50.8.19@o2ib2, removing former export from same NID [85453.510429] Lustre: Skipped 3829 previous similar messages [85453.516562] Lustre: MGS: Connection restored to 18aae849-c038-4 (at 10.50.8.19@o2ib2) [85453.525303] Lustre: Skipped 3829 previous similar messages [86053.662573] Lustre: MGS: Received new LWP connection from 10.51.2.27@o2ib3, removing former export from same NID [86053.673939] Lustre: Skipped 3830 previous similar messages [86053.680066] Lustre: MGS: Connection restored to ea2eddcc-744a-4 (at 10.51.2.27@o2ib3) [86053.688805] Lustre: Skipped 3829 previous similar messages [86653.662514] Lustre: MGS: Received new LWP connection from 10.50.0.64@o2ib2, removing former export from same NID [86653.673878] Lustre: Skipped 3640 previous similar messages [86653.680006] Lustre: MGS: Connection restored to 0706bac4-9be8-4 (at 10.50.0.64@o2ib2) [86653.688749] Lustre: Skipped 3639 previous similar messages [87253.705481] Lustre: MGS: Received new LWP connection from 10.51.2.57@o2ib3, removing former export from same NID [87253.716844] Lustre: Skipped 3235 previous similar messages [87253.722977] Lustre: MGS: Connection restored to 865d5eb6-08de-4 (at 10.51.2.57@o2ib3) [87253.731708] Lustre: Skipped 3236 previous similar messages [87854.188982] Lustre: MGS: Received new LWP connection from 10.210.13.37@tcp1, removing former export from same NID [87854.200435] Lustre: Skipped 3683 previous similar messages [87854.206559] Lustre: MGS: Connection restored to 828286e5-8360-3848-ee75-0f6a2a8bd930 (at 10.210.13.37@tcp1) [87854.217433] Lustre: Skipped 3684 previous similar messages [88464.670729] Lustre: MGS: Received new LWP connection from 10.50.12.11@o2ib2, removing former export from same NID [88464.682186] Lustre: Skipped 3830 previous similar messages [88464.688316] Lustre: MGS: Connection restored to 19ca1617-7766-4 (at 10.50.12.11@o2ib2) [88464.697152] Lustre: Skipped 3829 previous similar messages [89075.958006] Lustre: MGS: Received new LWP connection from 10.50.12.11@o2ib2, removing former export from same NID [89075.969466] Lustre: Skipped 3833 previous similar messages [89075.975598] Lustre: MGS: Connection restored to 19ca1617-7766-4 (at 10.50.12.11@o2ib2) [89075.984426] Lustre: Skipped 3833 previous similar messages [89691.516433] Lustre: MGS: Received new LWP connection from 10.50.12.11@o2ib2, removing former export from same NID [89691.527887] Lustre: Skipped 3832 previous similar messages [89691.534006] Lustre: MGS: Connection restored to 19ca1617-7766-4 (at 10.50.12.11@o2ib2) [89691.542843] Lustre: Skipped 3833 previous similar messages [90304.965641] Lustre: MGS: Received new LWP connection from 10.50.12.11@o2ib2, removing former export from same NID [90304.977100] Lustre: Skipped 3825 previous similar messages [90304.983230] Lustre: MGS: Connection restored to 19ca1617-7766-4 (at 10.50.12.11@o2ib2) [90304.992064] Lustre: Skipped 3832 previous similar messages [90342.831390] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_15:28509] [90342.832392] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ldlm_bl_18:28512] [90342.832430] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.832445] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.832448] CPU: 0 PID: 28512 Comm: ldlm_bl_18 Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.832449] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.832450] task: ffff8a65cab9b180 ti: ffff8a65582b8000 task.ti: ffff8a65582b8000 [90342.832457] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.832458] RSP: 0018:ffff8a65582bbc10 EFLAGS: 00000246 [90342.832459] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000010000 [90342.832460] RDX: ffff8a657e35b8c0 RSI: 0000000000510001 RDI: ffff8a56673df99c [90342.832460] RBP: ffff8a65582bbc10 R08: ffff8a657e21b8c0 R09: 0000000000000000 [90342.832461] R10: ffff8a756ee02258 R11: ffff8a756ee02248 R12: ffff8a65582bbbc0 [90342.832462] R13: ffff8a756f3138c0 R14: ffff8a65582bbb80 R15: ffffffffc0d2a5f8 [90342.832463] FS: 0000000000000000(0000) GS:ffff8a657e200000(0000) knlGS:0000000000000000 [90342.832463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.832464] CR2: 00007ff9de83a0a0 CR3: 0000001edb210000 CR4: 00000000001607f0 [90342.832465] Call Trace: [90342.832471] [] queued_spin_lock_slowpath+0xb/0xf [90342.832475] [] _raw_spin_lock+0x20/0x30 [90342.832514] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.832534] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.832555] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.832575] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.832590] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.832610] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.832631] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.832638] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.832657] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.832682] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.832704] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.832707] [] kthread+0xd1/0xe0 [90342.832709] [] ? insert_kthread_work+0x40/0x40 [90342.832712] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.832713] [] ? insert_kthread_work+0x40/0x40 [90342.832728] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.836391] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ldlm_bl_07:27292] [90342.836420] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.836431] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.836433] CPU: 2 PID: 27292 Comm: ldlm_bl_07 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.836433] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.836435] task: ffff8a5642ba2100 ti: ffff8a6560504000 task.ti: ffff8a6560504000 [90342.836439] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.836440] RSP: 0018:ffff8a6560507c10 EFLAGS: 00000246 [90342.836441] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000110000 [90342.836442] RDX: ffff8a657e41b8c0 RSI: 0000000000810001 RDI: ffff8a56673df99c [90342.836443] RBP: ffff8a6560507c10 R08: ffff8a657e25b8c0 R09: 0000000000000000 [90342.836443] R10: ffff8a56497f4458 R11: ffff8a56497f4448 R12: ffff8a6560507bc0 [90342.836444] R13: ffff8a756f2d2e40 R14: ffff8a6560507b80 R15: ffffffffc0d2a5f8 [90342.836445] FS: 0000000000000000(0000) GS:ffff8a657e240000(0000) knlGS:0000000000000000 [90342.836446] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.836446] CR2: 00007fef15016140 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.836447] Call Trace: [90342.836450] [] queued_spin_lock_slowpath+0xb/0xf [90342.836453] [] _raw_spin_lock+0x20/0x30 [90342.836484] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.836505] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.836526] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.836547] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.836558] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.836576] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.836595] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.836602] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.836620] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.836645] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.836647] [] ? wake_up_state+0x20/0x20 [90342.836671] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.836673] [] kthread+0xd1/0xe0 [90342.836674] [] ? insert_kthread_work+0x40/0x40 [90342.836677] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.836678] [] ? insert_kthread_work+0x40/0x40 [90342.836692] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.841389] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_17:28511] [90342.841420] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.841433] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.841436] CPU: 3 PID: 28511 Comm: ldlm_bl_17 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.841436] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.841437] task: ffff8a56517ae300 ti: ffff8a65582b4000 task.ti: ffff8a65582b4000 [90342.841442] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.841443] RSP: 0018:ffff8a65582b7c10 EFLAGS: 00000246 [90342.841444] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000190000 [90342.841445] RDX: ffff8a657e25b8c0 RSI: 0000000000110001 RDI: ffff8a56673df99c [90342.841445] RBP: ffff8a65582b7c10 R08: ffff8a757da5b8c0 R09: 0000000000000000 [90342.841446] R10: ffff8a756cdedc58 R11: ffff8a756cdedc48 R12: ffff8a65582b7bc0 [90342.841446] R13: ffff8a56673de000 R14: ffff8a65582b7b80 R15: ffffffffc0d2a5f8 [90342.841447] FS: 0000000000000000(0000) GS:ffff8a757da40000(0000) knlGS:0000000000000000 [90342.841448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.841448] CR2: 00000000006dee20 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.841449] Call Trace: [90342.841453] [] queued_spin_lock_slowpath+0xb/0xf [90342.841456] [] _raw_spin_lock+0x20/0x30 [90342.841515] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.841537] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.841558] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.841579] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.841590] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.841608] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.841627] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.841634] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.841654] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.841681] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.841705] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.841708] [] kthread+0xd1/0xe0 [90342.841709] [] ? insert_kthread_work+0x40/0x40 [90342.841711] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.841713] [] ? insert_kthread_work+0x40/0x40 [90342.841727] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.845391] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ldlm_bl_13:28507] [90342.845417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.845427] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.845429] CPU: 4 PID: 28507 Comm: ldlm_bl_13 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.845430] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.845431] task: ffff8a65c47a2100 ti: ffff8a6558294000 task.ti: ffff8a6558294000 [90342.845435] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.845436] RSP: 0018:ffff8a6558297c10 EFLAGS: 00000246 [90342.845436] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000210000 [90342.845437] RDX: ffff8a757dc5b8c0 RSI: 0000000000990001 RDI: ffff8a56673df99c [90342.845438] RBP: ffff8a6558297c10 R08: ffff8a657e29b8c0 R09: 0000000000000000 [90342.845438] R10: ffff8a6633765258 R11: ffff8a6633765248 R12: ffff8a6558297bc0 [90342.845439] R13: ffff8a756f313b00 R14: ffff8a6558297b80 R15: ffffffffc0d2a5f8 [90342.845440] FS: 0000000000000000(0000) GS:ffff8a657e280000(0000) knlGS:0000000000000000 [90342.845440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.845441] CR2: 00007fdec83450a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.845442] Call Trace: [90342.845444] [] queued_spin_lock_slowpath+0xb/0xf [90342.845447] [] _raw_spin_lock+0x20/0x30 [90342.845477] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.845496] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.845517] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.845538] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.845547] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.845566] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.845585] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.845592] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.845610] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.845633] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.845666] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.845668] [] kthread+0xd1/0xe0 [90342.845670] [] ? insert_kthread_work+0x40/0x40 [90342.845672] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.845673] [] ? insert_kthread_work+0x40/0x40 [90342.845687] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.850389] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [ldlm_bl_02:3841] [90342.850410] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.850418] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.850420] CPU: 5 PID: 3841 Comm: ldlm_bl_02 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.850421] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.850422] task: ffff8a6569fec200 ti: ffff8a6569ffc000 task.ti: ffff8a6569ffc000 [90342.850424] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.850425] RSP: 0018:ffff8a6569fffc10 EFLAGS: 00000246 [90342.850426] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000290000 [90342.850426] RDX: ffff8a657e39b8c0 RSI: 0000000000610001 RDI: ffff8a56673df99c [90342.850427] RBP: ffff8a6569fffc10 R08: ffff8a757da9b8c0 R09: 0000000000000000 [90342.850428] R10: ffff8a7568932058 R11: ffff8a7568932048 R12: ffff8a6569fffbc0 [90342.850428] R13: ffff8a56bd753d40 R14: ffff8a6569fffb80 R15: ffffffffc0d2a5f8 [90342.850429] FS: 0000000000000000(0000) GS:ffff8a757da80000(0000) knlGS:0000000000000000 [90342.850430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.850431] CR2: 00007f0a08ce1c70 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.850431] Call Trace: [90342.850433] [] queued_spin_lock_slowpath+0xb/0xf [90342.850435] [] _raw_spin_lock+0x20/0x30 [90342.850456] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.850474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.850495] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.850517] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.850523] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.850541] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.850561] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.850567] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.850585] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.850607] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.850610] [] ? wake_up_state+0x20/0x20 [90342.850632] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.850633] [] kthread+0xd1/0xe0 [90342.850635] [] ? insert_kthread_work+0x40/0x40 [90342.850637] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.850638] [] ? insert_kthread_work+0x40/0x40 [90342.850652] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.854391] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [ldlm_bl_04:22908] [90342.854419] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.854430] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.854432] CPU: 6 PID: 22908 Comm: ldlm_bl_04 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.854433] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.854434] task: ffff8a5644bca100 ti: ffff8a6560430000 task.ti: ffff8a6560430000 [90342.854469] RIP: 0010:[] [] ldlm_process_plain_lock+0x491/0xab0 [ptlrpc] [90342.854469] RSP: 0018:ffff8a6560433b88 EFLAGS: 00000206 [90342.854470] RAX: ffff8a655b628200 RBX: 0000000000000000 RCX: 0000000000000010 [90342.854471] RDX: 0000000000000000 RSI: ffff8a7563b9ec00 RDI: ffff8a756a714ec0 [90342.854471] RBP: ffff8a6560433bd0 R08: ffff8a6560433c28 R09: ffff8a657e39b8c0 [90342.854472] R10: ffff8a56633e4e58 R11: ffff8a7563b9ec00 R12: 0000000000000002 [90342.854472] R13: ffff8a7566cc06c0 R14: 0000000000000000 R15: 0000000000000000 [90342.854473] FS: 0000000000000000(0000) GS:ffff8a657e2c0000(0000) knlGS:0000000000000000 [90342.854474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.854475] CR2: 00007fff7e854fc8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.854475] Call Trace: [90342.854497] [] ? ldlm_errno2error+0x60/0x60 [ptlrpc] [90342.854516] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [90342.854536] [] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [90342.854555] [] ldlm_reprocess_res+0x28/0x30 [ptlrpc] [90342.854565] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.854584] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90342.854604] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90342.854611] [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] [90342.854632] [] ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90342.854651] [] ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90342.854675] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.854677] [] ? wake_up_state+0x20/0x20 [90342.854701] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.854703] [] kthread+0xd1/0xe0 [90342.854705] [] ? insert_kthread_work+0x40/0x40 [90342.854707] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.854708] [] ? insert_kthread_work+0x40/0x40 [90342.854722] Code: a0 0f 84 98 02 00 00 4c 39 fb ba 01 00 00 00 0f 84 fd 04 00 00 0f 1f 44 00 00 49 8b 87 08 02 00 00 8b 4d cc 4c 8b b0 60 fe ff ff <41> 8b 87 98 00 00 00 85 0c 85 40 70 37 c1 75 64 4d 85 e4 0f 84 [90342.859389] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ldlm_bl_01:3840] [90342.859410] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.859419] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.859420] CPU: 7 PID: 3840 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.859421] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.859422] task: ffff8a6569feb180 ti: ffff8a6569ff8000 task.ti: ffff8a6569ff8000 [90342.859424] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.859425] RSP: 0018:ffff8a6569ffbc10 EFLAGS: 00000246 [90342.859426] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000390000 [90342.859426] RDX: ffff8a657e29b8c0 RSI: 0000000000210001 RDI: ffff8a56673df99c [90342.859427] RBP: ffff8a6569ffbc10 R08: ffff8a757dadb8c0 R09: 0000000000000000 [90342.859428] R10: ffff8a75692b7658 R11: ffff8a75692b7648 R12: ffff8a6569ffbbc0 [90342.859428] R13: ffff8a756f2d3d40 R14: ffff8a6569ffbb80 R15: ffffffffc0d2a5f8 [90342.859429] FS: 0000000000000000(0000) GS:ffff8a757dac0000(0000) knlGS:0000000000000000 [90342.859430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.859430] CR2: 00007f0a09682978 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.859431] Call Trace: [90342.859433] [] queued_spin_lock_slowpath+0xb/0xf [90342.859435] [] _raw_spin_lock+0x20/0x30 [90342.859455] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.859474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.859493] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.859514] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.859520] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.859539] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.859558] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.859564] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.859584] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.859608] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.859610] [] ? wake_up_state+0x20/0x20 [90342.859633] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.859635] [] kthread+0xd1/0xe0 [90342.859636] [] ? insert_kthread_work+0x40/0x40 [90342.859638] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.859639] [] ? insert_kthread_work+0x40/0x40 [90342.859653] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.863391] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [ldlm_bl_03:22779] [90342.863419] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.863430] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.863432] CPU: 8 PID: 22779 Comm: ldlm_bl_03 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.863433] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.863434] task: ffff8a65e97a9080 ti: ffff8a6561218000 task.ti: ffff8a6561218000 [90342.863439] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.863440] RSP: 0018:ffff8a656121bc10 EFLAGS: 00000246 [90342.863440] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000410000 [90342.863441] RDX: ffff8a757dc9b8c0 RSI: 0000000000a90001 RDI: ffff8a56673df99c [90342.863442] RBP: ffff8a656121bc10 R08: ffff8a657e31b8c0 R09: 0000000000000000 [90342.863442] R10: ffff8a564c3c6c58 R11: ffff8a564c3c6c48 R12: ffff8a656121bbc0 [90342.863443] R13: ffff8a756f2d2180 R14: ffff8a656121bb80 R15: ffffffffc0d2a5f8 [90342.863444] FS: 0000000000000000(0000) GS:ffff8a657e300000(0000) knlGS:0000000000000000 [90342.863444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.863445] CR2: 0000563d758c22e8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.863446] Call Trace: [90342.863449] [] queued_spin_lock_slowpath+0xb/0xf [90342.863451] [] _raw_spin_lock+0x20/0x30 [90342.863480] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.863499] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.863520] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.863542] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.863553] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.863571] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.863590] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.863597] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.863617] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.863640] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.863643] [] ? wake_up_state+0x20/0x20 [90342.863666] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.863668] [] kthread+0xd1/0xe0 [90342.863670] [] ? insert_kthread_work+0x40/0x40 [90342.863672] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.863673] [] ? insert_kthread_work+0x40/0x40 [90342.863687] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.868388] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_21:28515] [90342.868410] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.868419] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.868420] CPU: 9 PID: 28515 Comm: ldlm_bl_21 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.868421] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.868422] task: ffff8a65cab9c200 ti: ffff8a65582c4000 task.ti: ffff8a65582c4000 [90342.868425] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.868426] RSP: 0018:ffff8a65582c7c10 EFLAGS: 00000246 [90342.868426] RAX: 0000000000000000 RBX: ffffd11580c1b340 RCX: 0000000000490000 [90342.868427] RDX: ffff8a657e45b8c0 RSI: 0000000000910001 RDI: ffff8a56673df99c [90342.868428] RBP: ffff8a65582c7c10 R08: ffff8a757db1b8c0 R09: 0000000000000000 [90342.868428] R10: ffff8a7568f2a458 R11: ffff8a7568f2a448 R12: ffff8a757db1acc0 [90342.868429] R13: ffffffffaa4e05ac R14: ffff8a65582c7b98 R15: ffffffffaa4e23f9 [90342.868430] FS: 0000000000000000(0000) GS:ffff8a757db00000(0000) knlGS:0000000000000000 [90342.868430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.868431] CR2: 00000000022f64b8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.868431] Call Trace: [90342.868433] [] queued_spin_lock_slowpath+0xb/0xf [90342.868435] [] _raw_spin_lock+0x20/0x30 [90342.868455] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.868474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.868493] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.868496] [] ? __kmalloc+0x1eb/0x230 [90342.868517] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.868523] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.868542] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.868560] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.868567] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.868587] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.868609] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.868631] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.868632] [] kthread+0xd1/0xe0 [90342.868634] [] ? insert_kthread_work+0x40/0x40 [90342.868636] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.868637] [] ? insert_kthread_work+0x40/0x40 [90342.868651] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.872390] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [ldlm_bl_12:28506] [90342.872419] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.872430] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.872432] CPU: 10 PID: 28506 Comm: ldlm_bl_12 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.872433] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.872434] task: ffff8a56bdb63180 ti: ffff8a6558290000 task.ti: ffff8a6558290000 [90342.872439] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.872440] RSP: 0018:ffff8a6558293c10 EFLAGS: 00000246 [90342.872440] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000510000 [90342.872441] RDX: ffff8a757db5b8c0 RSI: 0000000000590001 RDI: ffff8a56673df99c [90342.872442] RBP: ffff8a6558293c10 R08: ffff8a657e35b8c0 R09: 0000000000000000 [90342.872442] R10: ffff8a56943b5e58 R11: ffff8a56943b5e48 R12: ffff8a6558293bc0 [90342.872443] R13: ffff8a756f313800 R14: ffff8a6558293b80 R15: ffffffffc0d2a5f8 [90342.872444] FS: 0000000000000000(0000) GS:ffff8a657e340000(0000) knlGS:0000000000000000 [90342.872445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.872445] CR2: 00007fef13b0c1e4 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.872446] Call Trace: [90342.872449] [] queued_spin_lock_slowpath+0xb/0xf [90342.872452] [] _raw_spin_lock+0x20/0x30 [90342.872482] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.872501] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.872522] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.872543] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.872554] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.872573] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.872592] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.872599] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.872619] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.872644] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.872668] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.872670] [] kthread+0xd1/0xe0 [90342.872672] [] ? insert_kthread_work+0x40/0x40 [90342.872674] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.872676] [] ? insert_kthread_work+0x40/0x40 [90342.872690] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.877389] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [ldlm_bl_09:27294] [90342.877410] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.877418] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.877420] CPU: 11 PID: 27294 Comm: ldlm_bl_09 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.877421] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.877422] task: ffff8a5642ba4200 ti: ffff8a6561074000 task.ti: ffff8a6561074000 [90342.877424] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.877425] RSP: 0018:ffff8a6561077c10 EFLAGS: 00000246 [90342.877426] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000590000 [90342.877426] RDX: ffff8a757dc1b8c0 RSI: 0000000000890001 RDI: ffff8a56673df99c [90342.877427] RBP: ffff8a6561077c10 R08: ffff8a757db5b8c0 R09: 0000000000000000 [90342.877427] R10: ffff8a7568f4e458 R11: ffff8a7568f4e448 R12: ffff8a6561077bc0 [90342.877428] R13: ffff8a56bd7520c0 R14: ffff8a6561077b80 R15: ffffffffc0d2a5f8 [90342.877429] FS: 0000000000000000(0000) GS:ffff8a757db40000(0000) knlGS:0000000000000000 [90342.877430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.877431] CR2: 000056004b843070 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.877431] Call Trace: [90342.877433] [] queued_spin_lock_slowpath+0xb/0xf [90342.877435] [] _raw_spin_lock+0x20/0x30 [90342.877456] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.877477] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.877496] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.877517] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.877524] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.877542] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.877561] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.877567] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.877588] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.877611] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.877613] [] ? wake_up_state+0x20/0x20 [90342.877635] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.877637] [] kthread+0xd1/0xe0 [90342.877638] [] ? insert_kthread_work+0x40/0x40 [90342.877640] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.877642] [] ? insert_kthread_work+0x40/0x40 [90342.877656] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.880391] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [ldlm_bl_08:27293] [90342.880418] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.880429] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.880431] CPU: 12 PID: 27293 Comm: ldlm_bl_08 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.880432] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.880433] task: ffff8a5642ba3180 ti: ffff8a564e3c8000 task.ti: ffff8a564e3c8000 [90342.880437] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x156/0x200 [90342.880437] RSP: 0018:ffff8a564e3cbb90 EFLAGS: 00000202 [90342.880438] RAX: 0000000000000001 RBX: ffff8a564e3cbbc8 RCX: 0000000000610000 [90342.880439] RDX: 0000000000b10001 RSI: 0000000000310000 RDI: ffff8a56673df99c [90342.880439] RBP: ffff8a564e3cbb90 R08: ffff8a657e39b8c0 R09: 0000000000000000 [90342.880440] R10: ffff8a6561009a58 R11: ffff8a6561009a48 R12: 0000000000000000 [90342.880440] R13: 0000000000000000 R14: 0000000000000002 R15: ffff8a7562d77980 [90342.880441] FS: 0000000000000000(0000) GS:ffff8a657e380000(0000) knlGS:0000000000000000 [90342.880442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.880443] CR2: 00007f1c986fc9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.880443] Call Trace: [90342.880446] [] queued_spin_lock_slowpath+0xb/0xf [90342.880449] [] _raw_spin_lock+0x20/0x30 [90342.880477] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.880498] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [90342.880526] [] ? class_handle_unhash_nolock+0xd8/0x150 [obdclass] [90342.880534] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [90342.880552] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [90342.880573] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.880594] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.880599] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.880618] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.880636] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.880643] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.880661] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.880684] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.880686] [] ? wake_up_state+0x20/0x20 [90342.880710] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.880711] [] kthread+0xd1/0xe0 [90342.880713] [] ? insert_kthread_work+0x40/0x40 [90342.880715] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.880717] [] ? insert_kthread_work+0x40/0x40 [90342.880730] Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 [90342.885389] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [ldlm_bl_16:28510] [90342.885409] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.885416] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.885417] CPU: 13 PID: 28510 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.885418] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.885419] task: ffff8a56517ad280 ti: ffff8a65582b0000 task.ti: ffff8a65582b0000 [90342.885421] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.885422] RSP: 0018:ffff8a65582b3c10 EFLAGS: 00000246 [90342.885423] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000690000 [90342.885423] RDX: ffff8a757da9b8c0 RSI: 0000000000290001 RDI: ffff8a56673df99c [90342.885424] RBP: ffff8a65582b3c10 R08: ffff8a757db9b8c0 R09: 0000000000000000 [90342.885424] R10: ffff8a65c3382858 R11: ffff8a65c3382848 R12: ffff8a65582b3bc0 [90342.885425] R13: ffff8a56673dfa40 R14: ffff8a65582b3b80 R15: ffffffffc0d2a5f8 [90342.885425] FS: 0000000000000000(0000) GS:ffff8a757db80000(0000) knlGS:0000000000000000 [90342.885426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.885426] CR2: 00007f2096ed4bd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.885427] Call Trace: [90342.885429] [] queued_spin_lock_slowpath+0xb/0xf [90342.885431] [] _raw_spin_lock+0x20/0x30 [90342.885448] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.885466] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.885484] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.885502] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.885508] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.885526] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.885544] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.885550] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.885567] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.885587] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.885608] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.885610] [] kthread+0xd1/0xe0 [90342.885611] [] ? insert_kthread_work+0x40/0x40 [90342.885613] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.885614] [] ? insert_kthread_work+0x40/0x40 [90342.885625] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.888390] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [ldlm_bl_19:28513] [90342.888417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.888428] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.888430] CPU: 14 PID: 28513 Comm: ldlm_bl_19 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.888431] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.888432] task: ffff8a65cab9e300 ti: ffff8a65582bc000 task.ti: ffff8a65582bc000 [90342.888436] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.888437] RSP: 0018:ffff8a65582bfc10 EFLAGS: 00000246 [90342.888438] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000710000 [90342.888438] RDX: ffff8a757dadb8c0 RSI: 0000000000390001 RDI: ffff8a56673df99c [90342.888439] RBP: ffff8a65582bfc10 R08: ffff8a657e3db8c0 R09: 0000000000000000 [90342.888440] R10: ffff8a65cb3bca58 R11: ffff8a65cb3bca48 R12: ffff8a65582bfbc0 [90342.888440] R13: ffff8a756f313980 R14: ffff8a65582bfb80 R15: ffffffffc0d2a5f8 [90342.888441] FS: 0000000000000000(0000) GS:ffff8a657e3c0000(0000) knlGS:0000000000000000 [90342.888442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.888442] CR2: 00007fefa676b810 CR3: 000000101bf80000 CR4: 00000000001607e0 [90342.888443] Call Trace: [90342.888446] [] queued_spin_lock_slowpath+0xb/0xf [90342.888449] [] _raw_spin_lock+0x20/0x30 [90342.888477] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.888496] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.888517] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.888538] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.888549] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.888567] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.888586] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.888593] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.888614] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.888638] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.888662] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.888664] [] kthread+0xd1/0xe0 [90342.888665] [] ? insert_kthread_work+0x40/0x40 [90342.888667] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.888669] [] ? insert_kthread_work+0x40/0x40 [90342.888682] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.891388] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_10:27295] [90342.891409] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.891417] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.891419] CPU: 15 PID: 27295 Comm: ldlm_bl_10 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.891419] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.891420] task: ffff8a5664fa6300 ti: ffff8a56917dc000 task.ti: ffff8a56917dc000 [90342.891422] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.891423] RSP: 0018:ffff8a56917dfc10 EFLAGS: 00000246 [90342.891424] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000790000 [90342.891424] RDX: ffff8a757da5b8c0 RSI: 0000000000190001 RDI: ffff8a56673df99c [90342.891425] RBP: ffff8a56917dfc10 R08: ffff8a757dbdb8c0 R09: 0000000000000000 [90342.891426] R10: ffff8a7569378e58 R11: ffff8a7569378e48 R12: ffff8a56917dfbc0 [90342.891426] R13: ffff8a56bd7523c0 R14: ffff8a56917dfb80 R15: ffffffffc0d2a5f8 [90342.891427] FS: 0000000000000000(0000) GS:ffff8a757dbc0000(0000) knlGS:0000000000000000 [90342.891428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.891428] CR2: 00007f6df7e16871 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.891429] Call Trace: [90342.891431] [] queued_spin_lock_slowpath+0xb/0xf [90342.891432] [] _raw_spin_lock+0x20/0x30 [90342.891450] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.891472] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.891491] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.891511] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.891518] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.891536] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.891555] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.891561] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.891579] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.891602] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.891604] [] ? wake_up_state+0x20/0x20 [90342.891627] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.891629] [] kthread+0xd1/0xe0 [90342.891631] [] ? insert_kthread_work+0x40/0x40 [90342.891632] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.891634] [] ? insert_kthread_work+0x40/0x40 [90342.891648] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.894390] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_bl_14:28508] [90342.894417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.894428] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.894430] CPU: 16 PID: 28508 Comm: ldlm_bl_14 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.894431] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.894432] task: ffff8a65c37bb180 ti: ffff8a65582a8000 task.ti: ffff8a65582a8000 [90342.894435] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.894436] RSP: 0018:ffff8a65582abc10 EFLAGS: 00000246 [90342.894437] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000810000 [90342.894437] RDX: ffff8a657e21b8c0 RSI: 0000000000010001 RDI: ffff8a56673df99c [90342.894438] RBP: ffff8a65582abc10 R08: ffff8a657e41b8c0 R09: 0000000000000000 [90342.894439] R10: ffff8a756f2f5c58 R11: ffff8a756f2f5c48 R12: ffff8a65582abbc0 [90342.894439] R13: ffff8a756f3120c0 R14: ffff8a65582abb80 R15: ffffffffc0d2a5f8 [90342.894440] FS: 0000000000000000(0000) GS:ffff8a657e400000(0000) knlGS:0000000000000000 [90342.894441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.894441] CR2: 00007f266f2d3daa CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.894442] Call Trace: [90342.894445] [] queued_spin_lock_slowpath+0xb/0xf [90342.894447] [] _raw_spin_lock+0x20/0x30 [90342.894475] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.894494] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.894515] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.894535] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.894546] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.894564] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.894583] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.894589] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.894608] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.894631] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.894655] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.894656] [] kthread+0xd1/0xe0 [90342.894658] [] ? insert_kthread_work+0x40/0x40 [90342.894660] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.894661] [] ? insert_kthread_work+0x40/0x40 [90342.894675] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.897388] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ldlm_bl_11:28505] [90342.897408] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.897417] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.897418] CPU: 17 PID: 28505 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.897419] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.897420] task: ffff8a56bdb60000 ti: ffff8a655829c000 task.ti: ffff8a655829c000 [90342.897422] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.897423] RSP: 0018:ffff8a655829fc10 EFLAGS: 00000246 [90342.897423] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000890000 [90342.897424] RDX: ffff8a757da1b8c0 RSI: 0000000000090001 RDI: ffff8a56673df99c [90342.897425] RBP: ffff8a655829fc10 R08: ffff8a757dc1b8c0 R09: 0000000000000000 [90342.897425] R10: ffff8a661a792e58 R11: ffff8a661a792e48 R12: ffff8a655829fbc0 [90342.897426] R13: ffff8a756f312f00 R14: ffff8a655829fb80 R15: ffffffffc0d2a5f8 [90342.897426] FS: 0000000000000000(0000) GS:ffff8a757dc00000(0000) knlGS:0000000000000000 [90342.897427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.897428] CR2: 0000000000412515 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.897428] Call Trace: [90342.897430] [] queued_spin_lock_slowpath+0xb/0xf [90342.897432] [] _raw_spin_lock+0x20/0x30 [90342.897449] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.897468] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.897489] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.897510] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.897516] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.897534] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.897553] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.897559] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.897577] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.897599] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.897621] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.897622] [] kthread+0xd1/0xe0 [90342.897624] [] ? insert_kthread_work+0x40/0x40 [90342.897625] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.897627] [] ? insert_kthread_work+0x40/0x40 [90342.897641] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.900390] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [ldlm_bl_20:28514] [90342.900419] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.900430] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.900432] CPU: 18 PID: 28514 Comm: ldlm_bl_20 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.900433] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.900434] task: ffff8a65cab9d280 ti: ffff8a65582c0000 task.ti: ffff8a65582c0000 [90342.900439] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.900440] RSP: 0018:ffff8a65582c3c10 EFLAGS: 00000246 [90342.900440] RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000910000 [90342.900441] RDX: ffff8a657e3db8c0 RSI: 0000000000710001 RDI: ffff8a56673df99c [90342.900442] RBP: ffff8a65582c3c10 R08: ffff8a657e45b8c0 R09: 0000000000000000 [90342.900443] R10: ffff8a756f2bac58 R11: ffff8a756f2bac48 R12: ffff8a657e45acc0 [90342.900443] R13: ffffffffaa4e05ac R14: ffff8a65582c3b98 R15: ffffffffaa4e23f9 [90342.900445] FS: 0000000000000000(0000) GS:ffff8a657e440000(0000) knlGS:0000000000000000 [90342.900445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.900446] CR2: 0000000000ef1090 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.900447] Call Trace: [90342.900450] [] queued_spin_lock_slowpath+0xb/0xf [90342.900452] [] _raw_spin_lock+0x20/0x30 [90342.900481] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.900504] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.900527] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.900530] [] ? __kmalloc+0x1eb/0x230 [90342.900554] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.900566] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.900590] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.900614] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.900621] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.900643] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.900670] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.900697] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.900699] [] kthread+0xd1/0xe0 [90342.900701] [] ? insert_kthread_work+0x40/0x40 [90342.900703] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.900705] [] ? insert_kthread_work+0x40/0x40 [90342.900720] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.903388] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [ldlm_bl_05:27290] [90342.903408] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.903417] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.903418] CPU: 19 PID: 27290 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.903419] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.903420] task: ffff8a5642ba0000 ti: ffff8a65693d0000 task.ti: ffff8a65693d0000 [90342.903422] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.903423] RSP: 0018:ffff8a65693d3c10 EFLAGS: 00000246 [90342.903424] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000990000 [90342.903424] RDX: ffff8a757dbdb8c0 RSI: 0000000000790001 RDI: ffff8a56673df99c [90342.903425] RBP: ffff8a65693d3c10 R08: ffff8a757dc5b8c0 R09: 0000000000000000 [90342.903426] R10: ffff8a7568900c58 R11: ffff8a7568900c48 R12: ffff8a65693d3bc0 [90342.903426] R13: ffff8a756f2d3140 R14: ffff8a65693d3b80 R15: ffffffffc0d2a5f8 [90342.903427] FS: 0000000000000000(0000) GS:ffff8a757dc40000(0000) knlGS:0000000000000000 [90342.903427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.903428] CR2: 00007f99a2b321a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.903428] Call Trace: [90342.903430] [] queued_spin_lock_slowpath+0xb/0xf [90342.903432] [] _raw_spin_lock+0x20/0x30 [90342.903450] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.903469] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.903490] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.903511] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.903517] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.903535] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.903555] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.903561] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.903579] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.903602] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.903604] [] ? wake_up_state+0x20/0x20 [90342.903627] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.903629] [] kthread+0xd1/0xe0 [90342.903631] [] ? insert_kthread_work+0x40/0x40 [90342.903632] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.903634] [] ? insert_kthread_work+0x40/0x40 [90342.903648] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.906390] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ldlm_bl_23:28517] [90342.906417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.906428] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.906430] CPU: 20 PID: 28517 Comm: ldlm_bl_23 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.906431] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.906432] task: ffff8a65cab99080 ti: ffff8a65582d4000 task.ti: ffff8a65582d4000 [90342.906436] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.906437] RSP: 0018:ffff8a65582d7c10 EFLAGS: 00000246 [90342.906437] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a10000 [90342.906438] RDX: ffff8a757dcdb8c0 RSI: 0000000000b90001 RDI: ffff8a56673df99c [90342.906438] RBP: ffff8a65582d7c10 R08: ffff8a657e49b8c0 R09: 0000000000000000 [90342.906439] R10: ffff8a756e862258 R11: ffff8a756e862248 R12: ffff8a65582d7bc0 [90342.906440] R13: ffff8a756f2d2840 R14: ffff8a65582d7b80 R15: ffffffffc0d2a5f8 [90342.906441] FS: 0000000000000000(0000) GS:ffff8a657e480000(0000) knlGS:0000000000000000 [90342.906441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.906442] CR2: 00007f7ca3d3bbd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.906442] Call Trace: [90342.906445] [] queued_spin_lock_slowpath+0xb/0xf [90342.906447] [] _raw_spin_lock+0x20/0x30 [90342.906476] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.906494] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.906515] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.906537] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.906547] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.906566] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.906585] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.906592] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.906610] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.906634] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.906658] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.906660] [] kthread+0xd1/0xe0 [90342.906662] [] ? insert_kthread_work+0x40/0x40 [90342.906664] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.906665] [] ? insert_kthread_work+0x40/0x40 [90342.906679] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.909388] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [ldlm_bl_06:27291] [90342.909409] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.909417] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.909419] CPU: 21 PID: 27291 Comm: ldlm_bl_06 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.909420] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.909420] task: ffff8a5642ba5280 ti: ffff8a564f3a4000 task.ti: ffff8a564f3a4000 [90342.909423] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90342.909423] RSP: 0018:ffff8a564f3a7c10 EFLAGS: 00000246 [90342.909424] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a90000 [90342.909424] RDX: ffff8a757db9b8c0 RSI: 0000000000690001 RDI: ffff8a56673df99c [90342.909425] RBP: ffff8a564f3a7c10 R08: ffff8a757dc9b8c0 R09: 0000000000000000 [90342.909426] R10: ffff8a663db72258 R11: ffff8a663db72248 R12: ffff8a564f3a7bc0 [90342.909426] R13: ffff8a56bd752480 R14: ffff8a564f3a7b80 R15: ffffffffc0d2a5f8 [90342.909427] FS: 0000000000000000(0000) GS:ffff8a757dc80000(0000) knlGS:0000000000000000 [90342.909428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.909428] CR2: 00000000020ca6d8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.909429] Call Trace: [90342.909431] [] queued_spin_lock_slowpath+0xb/0xf [90342.909432] [] _raw_spin_lock+0x20/0x30 [90342.909450] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.909470] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.909489] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.909510] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.909516] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.909534] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.909553] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.909559] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.909577] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.909599] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.909601] [] ? wake_up_state+0x20/0x20 [90342.909622] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.909624] [] kthread+0xd1/0xe0 [90342.909625] [] ? insert_kthread_work+0x40/0x40 [90342.909627] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.909628] [] ? insert_kthread_work+0x40/0x40 [90342.909642] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90342.912390] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [ldlm_bl_24:28518] [90342.912418] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.912429] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.912430] CPU: 22 PID: 28518 Comm: ldlm_bl_24 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.912431] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.912432] task: ffff8a65cab98000 ti: ffff8a656905c000 task.ti: ffff8a656905c000 [90342.912436] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.912437] RSP: 0018:ffff8a656905fc10 EFLAGS: 00000246 [90342.912438] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b10000 [90342.912438] RDX: ffff8a657e49b8c0 RSI: 0000000000a10001 RDI: ffff8a56673df99c [90342.912439] RBP: ffff8a656905fc10 R08: ffff8a657e4db8c0 R09: 0000000000000000 [90342.912440] R10: ffff8a6637350058 R11: ffff8a6637350048 R12: ffff8a656905fbc0 [90342.912440] R13: ffff8a756f2d2b40 R14: ffff8a656905fb80 R15: ffffffffc0d2a5f8 [90342.912441] FS: 0000000000000000(0000) GS:ffff8a657e4c0000(0000) knlGS:0000000000000000 [90342.912442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.912442] CR2: 00007efc6a1ca000 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.912443] Call Trace: [90342.912446] [] queued_spin_lock_slowpath+0xb/0xf [90342.912448] [] _raw_spin_lock+0x20/0x30 [90342.912475] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.912494] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.912513] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.912534] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.912544] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.912563] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.912581] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.912588] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.912608] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.912632] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.912655] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.912657] [] kthread+0xd1/0xe0 [90342.912659] [] ? insert_kthread_work+0x40/0x40 [90342.912661] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.912663] [] ? insert_kthread_work+0x40/0x40 [90342.912676] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90342.915388] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_bl_22:28516] [90342.915409] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90342.915417] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90342.915419] CPU: 23 PID: 28516 Comm: ldlm_bl_22 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90342.915420] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90342.915420] task: ffff8a65cab9a100 ti: ffff8a65582d0000 task.ti: ffff8a65582d0000 [90342.915423] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90342.915423] RSP: 0018:ffff8a65582d3c10 EFLAGS: 00000246 [90342.915424] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b90000 [90342.915425] RDX: ffff8a757db1b8c0 RSI: 0000000000490001 RDI: ffff8a56673df99c [90342.915425] RBP: ffff8a65582d3c10 R08: ffff8a757dcdb8c0 R09: 0000000000000000 [90342.915426] R10: ffff8a65cbbf8458 R11: ffff8a65cbbf8448 R12: ffff8a65582d3bc0 [90342.915427] R13: ffff8a756f313e00 R14: ffff8a65582d3b80 R15: ffffffffc0d2a5f8 [90342.915427] FS: 0000000000000000(0000) GS:ffff8a757dcc0000(0000) knlGS:0000000000000000 [90342.915428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90342.915429] CR2: 00007f0a08c7a9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90342.915429] Call Trace: [90342.915431] [] queued_spin_lock_slowpath+0xb/0xf [90342.915433] [] _raw_spin_lock+0x20/0x30 [90342.915453] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90342.915474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90342.915492] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90342.915513] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90342.915519] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90342.915538] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.915557] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90342.915563] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90342.915581] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90342.915603] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90342.915626] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90342.915628] [] kthread+0xd1/0xe0 [90342.915629] [] ? insert_kthread_work+0x40/0x40 [90342.915631] [] ret_from_fork_nospec_begin+0x7/0x21 [90342.915632] [] ? insert_kthread_work+0x40/0x40 [90342.915646] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90351.852776] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90351.932104] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90351.967605] CPU: 1 PID: 28509 Comm: ldlm_bl_15 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90351.981770] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90351.990215] task: ffff8a65c37ba100 ti: ffff8a65582ac000 task.ti: ffff8a65582ac000 [90351.998562] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90352.009636] RSP: 0018:ffff8a65582afc10 EFLAGS: 00000246 [90352.015559] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000090000 [90352.023519] RDX: ffff8a657e31b8c0 RSI: 0000000000410001 RDI: ffff8a56673df99c [90352.031479] RBP: ffff8a65582afc10 R08: ffff8a757da1b8c0 R09: 0000000000000000 [90352.039439] R10: ffff8a5650be3658 R11: ffff8a5650be3648 R12: ffff8a65582afbc0 [90352.047400] R13: ffff8a56673dfe00 R14: ffff8a65582afb80 R15: ffffffffc0d2a5f8 [90352.055360] FS: 0000000000000000(0000) GS:ffff8a757da00000(0000) knlGS:0000000000000000 [90352.064387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90352.070795] CR2: 00007efff91a9000 CR3: 000000010d78e000 CR4: 00000000001607e0 [90352.078755] Call Trace: [90352.081482] [] queued_spin_lock_slowpath+0xb/0xf [90352.088473] [] _raw_spin_lock+0x20/0x30 [90352.094611] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90352.101914] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90352.109217] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90352.118362] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90352.127006] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90352.135074] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90352.144507] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90352.153927] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90352.161896] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90352.169976] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90352.177668] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90352.185920] [] kthread+0xd1/0xe0 [90352.191361] [] ? insert_kthread_work+0x40/0x40 [90352.198159] [] ret_from_fork_nospec_begin+0x7/0x21 [90352.205344] [] ? insert_kthread_work+0x40/0x40 [90352.212140] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.831896] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ldlm_bl_18:28512] [90370.835896] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ldlm_bl_07:27292] [90370.835921] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.835931] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.835933] CPU: 2 PID: 27292 Comm: ldlm_bl_07 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.835934] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.835935] task: ffff8a5642ba2100 ti: ffff8a6560504000 task.ti: ffff8a6560504000 [90370.835939] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.835940] RSP: 0018:ffff8a6560507c10 EFLAGS: 00000246 [90370.835941] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000110000 [90370.835941] RDX: ffff8a657e41b8c0 RSI: 0000000000810001 RDI: ffff8a56673df99c [90370.835942] RBP: ffff8a6560507c10 R08: ffff8a657e25b8c0 R09: 0000000000000000 [90370.835943] R10: ffff8a56497f4458 R11: ffff8a56497f4448 R12: ffff8a6560507bc0 [90370.835943] R13: ffff8a756f2d2e40 R14: ffff8a6560507b80 R15: ffffffffc0d2a5f8 [90370.835944] FS: 0000000000000000(0000) GS:ffff8a657e240000(0000) knlGS:0000000000000000 [90370.835944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.835945] CR2: 00007fef15016140 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.835946] Call Trace: [90370.835948] [] queued_spin_lock_slowpath+0xb/0xf [90370.835951] [] _raw_spin_lock+0x20/0x30 [90370.835980] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.836001] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.836022] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.836043] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.836054] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.836073] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.836091] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.836098] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.836116] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.836139] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.836142] [] ? wake_up_state+0x20/0x20 [90370.836165] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.836167] [] kthread+0xd1/0xe0 [90370.836168] [] ? insert_kthread_work+0x40/0x40 [90370.836170] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.836172] [] ? insert_kthread_work+0x40/0x40 [90370.836185] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.840894] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_17:28511] [90370.840914] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.840922] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.840923] CPU: 3 PID: 28511 Comm: ldlm_bl_17 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.840924] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.840925] task: ffff8a56517ae300 ti: ffff8a65582b4000 task.ti: ffff8a65582b4000 [90370.840927] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.840927] RSP: 0018:ffff8a65582b7c10 EFLAGS: 00000246 [90370.840928] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000190000 [90370.840929] RDX: ffff8a657e25b8c0 RSI: 0000000000110001 RDI: ffff8a56673df99c [90370.840929] RBP: ffff8a65582b7c10 R08: ffff8a757da5b8c0 R09: 0000000000000000 [90370.840930] R10: ffff8a756cdedc58 R11: ffff8a756cdedc48 R12: ffff8a65582b7bc0 [90370.840930] R13: ffff8a56673de000 R14: ffff8a65582b7b80 R15: ffffffffc0d2a5f8 [90370.840931] FS: 0000000000000000(0000) GS:ffff8a757da40000(0000) knlGS:0000000000000000 [90370.840932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.840932] CR2: 00000000006dee20 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.840933] Call Trace: [90370.840934] [] queued_spin_lock_slowpath+0xb/0xf [90370.840936] [] _raw_spin_lock+0x20/0x30 [90370.840956] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.840977] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.840995] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.841016] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.841022] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.841041] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.841060] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.841065] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.841085] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.841107] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.841129] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.841130] [] kthread+0xd1/0xe0 [90370.841132] [] ? insert_kthread_work+0x40/0x40 [90370.841133] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.841135] [] ? insert_kthread_work+0x40/0x40 [90370.841148] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.844896] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ldlm_bl_13:28507] [90370.844922] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.844932] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.844934] CPU: 4 PID: 28507 Comm: ldlm_bl_13 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.844935] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.844936] task: ffff8a65c47a2100 ti: ffff8a6558294000 task.ti: ffff8a6558294000 [90370.844941] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.844942] RSP: 0018:ffff8a6558297c10 EFLAGS: 00000246 [90370.844942] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000210000 [90370.844943] RDX: ffff8a757dc5b8c0 RSI: 0000000000990001 RDI: ffff8a56673df99c [90370.844944] RBP: ffff8a6558297c10 R08: ffff8a657e29b8c0 R09: 0000000000000000 [90370.844944] R10: ffff8a6633765258 R11: ffff8a6633765248 R12: ffff8a6558297bc0 [90370.844945] R13: ffff8a756f313b00 R14: ffff8a6558297b80 R15: ffffffffc0d2a5f8 [90370.844946] FS: 0000000000000000(0000) GS:ffff8a657e280000(0000) knlGS:0000000000000000 [90370.844946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.844947] CR2: 00007fdec83450a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.844947] Call Trace: [90370.844950] [] queued_spin_lock_slowpath+0xb/0xf [90370.844953] [] _raw_spin_lock+0x20/0x30 [90370.844983] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.845005] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.845026] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.845047] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.845058] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.845076] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.845095] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.845102] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.845120] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.845143] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.845167] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.845169] [] kthread+0xd1/0xe0 [90370.845170] [] ? insert_kthread_work+0x40/0x40 [90370.845172] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.845174] [] ? insert_kthread_work+0x40/0x40 [90370.845187] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.849894] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ldlm_bl_02:3841] [90370.849914] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.849922] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.849923] CPU: 5 PID: 3841 Comm: ldlm_bl_02 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.849924] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.849924] task: ffff8a6569fec200 ti: ffff8a6569ffc000 task.ti: ffff8a6569ffc000 [90370.849927] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.849927] RSP: 0018:ffff8a6569fffc10 EFLAGS: 00000246 [90370.849928] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000290000 [90370.849929] RDX: ffff8a657e39b8c0 RSI: 0000000000610001 RDI: ffff8a56673df99c [90370.849929] RBP: ffff8a6569fffc10 R08: ffff8a757da9b8c0 R09: 0000000000000000 [90370.849930] R10: ffff8a7568932058 R11: ffff8a7568932048 R12: ffff8a6569fffbc0 [90370.849931] R13: ffff8a56bd753d40 R14: ffff8a6569fffb80 R15: ffffffffc0d2a5f8 [90370.849932] FS: 0000000000000000(0000) GS:ffff8a757da80000(0000) knlGS:0000000000000000 [90370.849932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.849933] CR2: 00007f0a08ce1c70 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.849933] Call Trace: [90370.849935] [] queued_spin_lock_slowpath+0xb/0xf [90370.849936] [] _raw_spin_lock+0x20/0x30 [90370.849956] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.849975] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.849996] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.850017] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.850023] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.850041] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.850060] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.850066] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.850085] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.850106] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.850108] [] ? wake_up_state+0x20/0x20 [90370.850132] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.850133] [] kthread+0xd1/0xe0 [90370.850135] [] ? insert_kthread_work+0x40/0x40 [90370.850136] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.850138] [] ? insert_kthread_work+0x40/0x40 [90370.850151] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.853895] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ldlm_bl_04:22908] [90370.853921] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.853932] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.853934] CPU: 6 PID: 22908 Comm: ldlm_bl_04 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.853934] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.853935] task: ffff8a5644bca100 ti: ffff8a6560430000 task.ti: ffff8a6560430000 [90370.853967] RIP: 0010:[] [] ldlm_process_plain_lock+0x491/0xab0 [ptlrpc] [90370.853968] RSP: 0018:ffff8a6560433b88 EFLAGS: 00000287 [90370.853968] RAX: ffff8a7563a18440 RBX: 0000000000000000 RCX: 0000000000000010 [90370.853969] RDX: 0000000000000000 RSI: ffff8a7562d5af40 RDI: ffff8a756a714ec0 [90370.853969] RBP: ffff8a6560433bd0 R08: ffff8a6560433c28 R09: ffff8a657e39b8c0 [90370.853970] R10: ffff8a56633e4e58 R11: ffff8a7562d5af40 R12: 0000000000000002 [90370.853970] R13: ffff8a7566cc06c0 R14: 0000000000000000 R15: 0000000000000000 [90370.853971] FS: 0000000000000000(0000) GS:ffff8a657e2c0000(0000) knlGS:0000000000000000 [90370.853972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.853972] CR2: 00007fff7e854fc8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.853973] Call Trace: [90370.853996] [] ? ldlm_errno2error+0x60/0x60 [ptlrpc] [90370.854017] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [90370.854036] [] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [90370.854058] [] ldlm_reprocess_res+0x28/0x30 [ptlrpc] [90370.854068] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.854086] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90370.854107] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90370.854114] [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] [90370.854134] [] ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90370.854155] [] ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90370.854178] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.854181] [] ? wake_up_state+0x20/0x20 [90370.854205] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.854206] [] kthread+0xd1/0xe0 [90370.854208] [] ? insert_kthread_work+0x40/0x40 [90370.854210] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.854212] [] ? insert_kthread_work+0x40/0x40 [90370.854226] Code: a0 0f 84 98 02 00 00 4c 39 fb ba 01 00 00 00 0f 84 fd 04 00 00 0f 1f 44 00 00 49 8b 87 08 02 00 00 8b 4d cc 4c 8b b0 60 fe ff ff <41> 8b 87 98 00 00 00 85 0c 85 40 70 37 c1 75 64 4d 85 e4 0f 84 [90370.858894] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ldlm_bl_01:3840] [90370.858913] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.858921] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.858923] CPU: 7 PID: 3840 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.858923] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.858924] task: ffff8a6569feb180 ti: ffff8a6569ff8000 task.ti: ffff8a6569ff8000 [90370.858926] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.858927] RSP: 0018:ffff8a6569ffbc10 EFLAGS: 00000246 [90370.858927] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000390000 [90370.858928] RDX: ffff8a657e29b8c0 RSI: 0000000000210001 RDI: ffff8a56673df99c [90370.858929] RBP: ffff8a6569ffbc10 R08: ffff8a757dadb8c0 R09: 0000000000000000 [90370.858929] R10: ffff8a75692b7658 R11: ffff8a75692b7648 R12: ffff8a6569ffbbc0 [90370.858930] R13: ffff8a756f2d3d40 R14: ffff8a6569ffbb80 R15: ffffffffc0d2a5f8 [90370.858931] FS: 0000000000000000(0000) GS:ffff8a757dac0000(0000) knlGS:0000000000000000 [90370.858931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.858932] CR2: 00007f0a09682978 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.858932] Call Trace: [90370.858934] [] queued_spin_lock_slowpath+0xb/0xf [90370.858936] [] _raw_spin_lock+0x20/0x30 [90370.858956] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.858977] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.858995] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.859016] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.859022] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.859041] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.859059] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.859065] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.859086] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.859110] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.859111] [] ? wake_up_state+0x20/0x20 [90370.859135] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.859136] [] kthread+0xd1/0xe0 [90370.859138] [] ? insert_kthread_work+0x40/0x40 [90370.859139] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.859141] [] ? insert_kthread_work+0x40/0x40 [90370.859154] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.862895] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [ldlm_bl_03:22779] [90370.862919] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.862929] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.862931] CPU: 8 PID: 22779 Comm: ldlm_bl_03 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.862932] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.862933] task: ffff8a65e97a9080 ti: ffff8a6561218000 task.ti: ffff8a6561218000 [90370.862937] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.862938] RSP: 0018:ffff8a656121bc10 EFLAGS: 00000246 [90370.862938] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000410000 [90370.862939] RDX: ffff8a757dc9b8c0 RSI: 0000000000a90001 RDI: ffff8a56673df99c [90370.862939] RBP: ffff8a656121bc10 R08: ffff8a657e31b8c0 R09: 0000000000000000 [90370.862940] R10: ffff8a564c3c6c58 R11: ffff8a564c3c6c48 R12: ffff8a656121bbc0 [90370.862941] R13: ffff8a756f2d2180 R14: ffff8a656121bb80 R15: ffffffffc0d2a5f8 [90370.862942] FS: 0000000000000000(0000) GS:ffff8a657e300000(0000) knlGS:0000000000000000 [90370.862942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.862943] CR2: 0000563d758c22e8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.862943] Call Trace: [90370.862946] [] queued_spin_lock_slowpath+0xb/0xf [90370.862949] [] _raw_spin_lock+0x20/0x30 [90370.862980] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.863000] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.863020] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.863041] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.863053] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.863071] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.863090] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.863096] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.863114] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.863137] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.863140] [] ? wake_up_state+0x20/0x20 [90370.863163] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.863165] [] kthread+0xd1/0xe0 [90370.863167] [] ? insert_kthread_work+0x40/0x40 [90370.863169] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.863170] [] ? insert_kthread_work+0x40/0x40 [90370.863184] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.867894] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_21:28515] [90370.867913] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.867921] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.867923] CPU: 9 PID: 28515 Comm: ldlm_bl_21 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.867923] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.867924] task: ffff8a65cab9c200 ti: ffff8a65582c4000 task.ti: ffff8a65582c4000 [90370.867926] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.867927] RSP: 0018:ffff8a65582c7c10 EFLAGS: 00000246 [90370.867927] RAX: 0000000000000000 RBX: ffffd11580c1b340 RCX: 0000000000490000 [90370.867928] RDX: ffff8a657e45b8c0 RSI: 0000000000910001 RDI: ffff8a56673df99c [90370.867929] RBP: ffff8a65582c7c10 R08: ffff8a757db1b8c0 R09: 0000000000000000 [90370.867929] R10: ffff8a7568f2a458 R11: ffff8a7568f2a448 R12: ffff8a757db1acc0 [90370.867930] R13: ffffffffaa4e05ac R14: ffff8a65582c7b98 R15: ffffffffaa4e23f9 [90370.867931] FS: 0000000000000000(0000) GS:ffff8a757db00000(0000) knlGS:0000000000000000 [90370.867931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.867932] CR2: 00000000022f64b8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.867932] Call Trace: [90370.867934] [] queued_spin_lock_slowpath+0xb/0xf [90370.867936] [] _raw_spin_lock+0x20/0x30 [90370.867954] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.867973] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.867993] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.867995] [] ? __kmalloc+0x1eb/0x230 [90370.868014] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.868020] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.868038] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.868057] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.868062] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.868081] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.868103] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.868124] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.868126] [] kthread+0xd1/0xe0 [90370.868127] [] ? insert_kthread_work+0x40/0x40 [90370.868129] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.868130] [] ? insert_kthread_work+0x40/0x40 [90370.868144] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.871896] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [ldlm_bl_12:28506] [90370.871920] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.871929] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.871931] CPU: 10 PID: 28506 Comm: ldlm_bl_12 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.871932] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.871933] task: ffff8a56bdb63180 ti: ffff8a6558290000 task.ti: ffff8a6558290000 [90370.871937] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.871937] RSP: 0018:ffff8a6558293c10 EFLAGS: 00000246 [90370.871938] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000510000 [90370.871939] RDX: ffff8a757db5b8c0 RSI: 0000000000590001 RDI: ffff8a56673df99c [90370.871939] RBP: ffff8a6558293c10 R08: ffff8a657e35b8c0 R09: 0000000000000000 [90370.871940] R10: ffff8a56943b5e58 R11: ffff8a56943b5e48 R12: ffff8a6558293bc0 [90370.871940] R13: ffff8a756f313800 R14: ffff8a6558293b80 R15: ffffffffc0d2a5f8 [90370.871941] FS: 0000000000000000(0000) GS:ffff8a657e340000(0000) knlGS:0000000000000000 [90370.871942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.871942] CR2: 00007fef13b0c1e4 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.871943] Call Trace: [90370.871946] [] queued_spin_lock_slowpath+0xb/0xf [90370.871948] [] _raw_spin_lock+0x20/0x30 [90370.871977] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.871998] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.872019] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.872040] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.872051] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.872070] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.872088] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.872095] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.872116] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.872140] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.872163] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.872165] [] kthread+0xd1/0xe0 [90370.872167] [] ? insert_kthread_work+0x40/0x40 [90370.872169] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.872170] [] ? insert_kthread_work+0x40/0x40 [90370.872184] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.876893] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [ldlm_bl_09:27294] [90370.876913] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.876921] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.876922] CPU: 11 PID: 27294 Comm: ldlm_bl_09 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.876923] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.876924] task: ffff8a5642ba4200 ti: ffff8a6561074000 task.ti: ffff8a6561074000 [90370.876926] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.876926] RSP: 0018:ffff8a6561077c10 EFLAGS: 00000246 [90370.876927] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000590000 [90370.876928] RDX: ffff8a757dc1b8c0 RSI: 0000000000890001 RDI: ffff8a56673df99c [90370.876928] RBP: ffff8a6561077c10 R08: ffff8a757db5b8c0 R09: 0000000000000000 [90370.876929] R10: ffff8a7568f4e458 R11: ffff8a7568f4e448 R12: ffff8a6561077bc0 [90370.876929] R13: ffff8a56bd7520c0 R14: ffff8a6561077b80 R15: ffffffffc0d2a5f8 [90370.876930] FS: 0000000000000000(0000) GS:ffff8a757db40000(0000) knlGS:0000000000000000 [90370.876931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.876931] CR2: 000056004b843070 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.876932] Call Trace: [90370.876933] [] queued_spin_lock_slowpath+0xb/0xf [90370.876935] [] _raw_spin_lock+0x20/0x30 [90370.876953] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.876972] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.876991] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.877011] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.877017] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.877036] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.877054] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.877060] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.877080] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.877102] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.877104] [] ? wake_up_state+0x20/0x20 [90370.877127] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.877129] [] kthread+0xd1/0xe0 [90370.877130] [] ? insert_kthread_work+0x40/0x40 [90370.877132] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.877133] [] ? insert_kthread_work+0x40/0x40 [90370.877147] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.879894] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [ldlm_bl_08:27293] [90370.879915] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.879924] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.879925] CPU: 12 PID: 27293 Comm: ldlm_bl_08 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.879926] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.879927] task: ffff8a5642ba3180 ti: ffff8a564e3c8000 task.ti: ffff8a564e3c8000 [90370.879931] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x156/0x200 [90370.879932] RSP: 0018:ffff8a564e3cbb90 EFLAGS: 00000202 [90370.879932] RAX: 0000000000000001 RBX: ffff8a564e3cbbc8 RCX: 0000000000610000 [90370.879933] RDX: 0000000000b10001 RSI: 0000000000310000 RDI: ffff8a56673df99c [90370.879933] RBP: ffff8a564e3cbb90 R08: ffff8a657e39b8c0 R09: 0000000000000000 [90370.879934] R10: ffff8a6561009a58 R11: ffff8a6561009a48 R12: 0000000000000000 [90370.879934] R13: 0000000000000000 R14: 0000000000000002 R15: ffff8a7562d77980 [90370.879935] FS: 0000000000000000(0000) GS:ffff8a657e380000(0000) knlGS:0000000000000000 [90370.879936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.879936] CR2: 00007f1c986fc9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.879937] Call Trace: [90370.879939] [] queued_spin_lock_slowpath+0xb/0xf [90370.879942] [] _raw_spin_lock+0x20/0x30 [90370.879967] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.879985] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [90370.880007] [] ? class_handle_unhash_nolock+0xd8/0x150 [obdclass] [90370.880015] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [90370.880031] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [90370.880050] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.880067] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.880073] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.880090] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.880108] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.880114] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.880131] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.880153] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.880155] [] ? wake_up_state+0x20/0x20 [90370.880176] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.880177] [] kthread+0xd1/0xe0 [90370.880179] [] ? insert_kthread_work+0x40/0x40 [90370.880180] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.880182] [] ? insert_kthread_work+0x40/0x40 [90370.880193] Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 [90370.884893] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [ldlm_bl_16:28510] [90370.884913] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.884921] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.884922] CPU: 13 PID: 28510 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.884923] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.884924] task: ffff8a56517ad280 ti: ffff8a65582b0000 task.ti: ffff8a65582b0000 [90370.884926] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.884926] RSP: 0018:ffff8a65582b3c10 EFLAGS: 00000246 [90370.884927] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000690000 [90370.884928] RDX: ffff8a757da9b8c0 RSI: 0000000000290001 RDI: ffff8a56673df99c [90370.884928] RBP: ffff8a65582b3c10 R08: ffff8a757db9b8c0 R09: 0000000000000000 [90370.884929] R10: ffff8a65c3382858 R11: ffff8a65c3382848 R12: ffff8a65582b3bc0 [90370.884930] R13: ffff8a56673dfa40 R14: ffff8a65582b3b80 R15: ffffffffc0d2a5f8 [90370.884930] FS: 0000000000000000(0000) GS:ffff8a757db80000(0000) knlGS:0000000000000000 [90370.884931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.884932] CR2: 00007f2096ed4bd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.884932] Call Trace: [90370.884934] [] queued_spin_lock_slowpath+0xb/0xf [90370.884936] [] _raw_spin_lock+0x20/0x30 [90370.884954] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.884973] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.884994] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.885015] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.885021] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.885039] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.885057] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.885063] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.885081] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.885105] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.885127] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.885128] [] kthread+0xd1/0xe0 [90370.885130] [] ? insert_kthread_work+0x40/0x40 [90370.885132] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.885133] [] ? insert_kthread_work+0x40/0x40 [90370.885147] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.887895] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [ldlm_bl_19:28513] [90370.887920] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.887930] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.887931] CPU: 14 PID: 28513 Comm: ldlm_bl_19 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.887932] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.887933] task: ffff8a65cab9e300 ti: ffff8a65582bc000 task.ti: ffff8a65582bc000 [90370.887937] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.887938] RSP: 0018:ffff8a65582bfc10 EFLAGS: 00000246 [90370.887938] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000710000 [90370.887939] RDX: ffff8a757dadb8c0 RSI: 0000000000390001 RDI: ffff8a56673df99c [90370.887940] RBP: ffff8a65582bfc10 R08: ffff8a657e3db8c0 R09: 0000000000000000 [90370.887940] R10: ffff8a65cb3bca58 R11: ffff8a65cb3bca48 R12: ffff8a65582bfbc0 [90370.887941] R13: ffff8a756f313980 R14: ffff8a65582bfb80 R15: ffffffffc0d2a5f8 [90370.887942] FS: 0000000000000000(0000) GS:ffff8a657e3c0000(0000) knlGS:0000000000000000 [90370.887942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.887943] CR2: 00007fefa676b810 CR3: 000000101bf80000 CR4: 00000000001607e0 [90370.887943] Call Trace: [90370.887946] [] queued_spin_lock_slowpath+0xb/0xf [90370.887949] [] _raw_spin_lock+0x20/0x30 [90370.887976] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.887995] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.888016] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.888037] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.888047] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.888066] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.888084] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.888091] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.888111] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.888135] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.888159] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.888161] [] kthread+0xd1/0xe0 [90370.888162] [] ? insert_kthread_work+0x40/0x40 [90370.888164] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.888166] [] ? insert_kthread_work+0x40/0x40 [90370.888180] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.890893] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_10:27295] [90370.890912] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.890920] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.890921] CPU: 15 PID: 27295 Comm: ldlm_bl_10 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.890922] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.890923] task: ffff8a5664fa6300 ti: ffff8a56917dc000 task.ti: ffff8a56917dc000 [90370.890925] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.890925] RSP: 0018:ffff8a56917dfc10 EFLAGS: 00000246 [90370.890926] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000790000 [90370.890927] RDX: ffff8a757da5b8c0 RSI: 0000000000190001 RDI: ffff8a56673df99c [90370.890927] RBP: ffff8a56917dfc10 R08: ffff8a757dbdb8c0 R09: 0000000000000000 [90370.890928] R10: ffff8a7569378e58 R11: ffff8a7569378e48 R12: ffff8a56917dfbc0 [90370.890928] R13: ffff8a56bd7523c0 R14: ffff8a56917dfb80 R15: ffffffffc0d2a5f8 [90370.890929] FS: 0000000000000000(0000) GS:ffff8a757dbc0000(0000) knlGS:0000000000000000 [90370.890930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.890930] CR2: 00007f6df7e16871 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.890931] Call Trace: [90370.890932] [] queued_spin_lock_slowpath+0xb/0xf [90370.890934] [] _raw_spin_lock+0x20/0x30 [90370.890952] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.890970] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.890989] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.891010] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.891016] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.891034] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.891053] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.891058] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.891079] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.891100] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.891102] [] ? wake_up_state+0x20/0x20 [90370.891123] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.891125] [] kthread+0xd1/0xe0 [90370.891126] [] ? insert_kthread_work+0x40/0x40 [90370.891128] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.891129] [] ? insert_kthread_work+0x40/0x40 [90370.891143] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.893894] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ldlm_bl_14:28508] [90370.893916] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.893926] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.893927] CPU: 16 PID: 28508 Comm: ldlm_bl_14 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.893928] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.893929] task: ffff8a65c37bb180 ti: ffff8a65582a8000 task.ti: ffff8a65582a8000 [90370.893932] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.893933] RSP: 0018:ffff8a65582abc10 EFLAGS: 00000246 [90370.893933] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000810000 [90370.893934] RDX: ffff8a657e21b8c0 RSI: 0000000000010001 RDI: ffff8a56673df99c [90370.893935] RBP: ffff8a65582abc10 R08: ffff8a657e41b8c0 R09: 0000000000000000 [90370.893935] R10: ffff8a756f2f5c58 R11: ffff8a756f2f5c48 R12: ffff8a65582abbc0 [90370.893936] R13: ffff8a756f3120c0 R14: ffff8a65582abb80 R15: ffffffffc0d2a5f8 [90370.893937] FS: 0000000000000000(0000) GS:ffff8a657e400000(0000) knlGS:0000000000000000 [90370.893937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.893938] CR2: 00007f266f2d3daa CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.893938] Call Trace: [90370.893940] [] queued_spin_lock_slowpath+0xb/0xf [90370.893943] [] _raw_spin_lock+0x20/0x30 [90370.893966] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.893985] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.894006] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.894025] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.894033] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.894051] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.894069] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.894076] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.894094] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.894116] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.894138] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.894140] [] kthread+0xd1/0xe0 [90370.894141] [] ? insert_kthread_work+0x40/0x40 [90370.894143] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.894145] [] ? insert_kthread_work+0x40/0x40 [90370.894158] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.896893] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ldlm_bl_11:28505] [90370.896912] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.896920] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.896921] CPU: 17 PID: 28505 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.896922] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.896922] task: ffff8a56bdb60000 ti: ffff8a655829c000 task.ti: ffff8a655829c000 [90370.896924] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.896925] RSP: 0018:ffff8a655829fc10 EFLAGS: 00000246 [90370.896926] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000890000 [90370.896926] RDX: ffff8a757da1b8c0 RSI: 0000000000090001 RDI: ffff8a56673df99c [90370.896927] RBP: ffff8a655829fc10 R08: ffff8a757dc1b8c0 R09: 0000000000000000 [90370.896927] R10: ffff8a661a792e58 R11: ffff8a661a792e48 R12: ffff8a655829fbc0 [90370.896928] R13: ffff8a756f312f00 R14: ffff8a655829fb80 R15: ffffffffc0d2a5f8 [90370.896929] FS: 0000000000000000(0000) GS:ffff8a757dc00000(0000) knlGS:0000000000000000 [90370.896929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.896930] CR2: 0000000000412515 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.896930] Call Trace: [90370.896932] [] queued_spin_lock_slowpath+0xb/0xf [90370.896933] [] _raw_spin_lock+0x20/0x30 [90370.896954] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.896972] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.896993] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.897012] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.897018] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.897036] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.897055] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.897061] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.897079] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.897100] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.897122] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.897123] [] kthread+0xd1/0xe0 [90370.897125] [] ? insert_kthread_work+0x40/0x40 [90370.897127] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.897128] [] ? insert_kthread_work+0x40/0x40 [90370.897142] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90370.899893] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [ldlm_bl_20:28514] [90370.899919] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.899930] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.899932] CPU: 18 PID: 28514 Comm: ldlm_bl_20 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.899933] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.899934] task: ffff8a65cab9d280 ti: ffff8a65582c0000 task.ti: ffff8a65582c0000 [90370.899937] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x120/0x200 [90370.899938] RSP: 0018:ffff8a65582c3c10 EFLAGS: 00000246 [90370.899939] RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000910000 [90370.899939] RDX: ffff8a657e3db8c0 RSI: 0000000000710001 RDI: ffff8a56673df99c [90370.899940] RBP: ffff8a65582c3c10 R08: ffff8a657e45b8c0 R09: 0000000000000000 [90370.899941] R10: ffff8a756f2bac58 R11: ffff8a756f2bac48 R12: ffff8a657e45acc0 [90370.899942] R13: ffffffffaa4e05ac R14: ffff8a65582c3b98 R15: ffffffffaa4e23f9 [90370.899943] FS: 0000000000000000(0000) GS:ffff8a657e440000(0000) knlGS:0000000000000000 [90370.899943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.899944] CR2: 0000000000ef1090 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.899945] Call Trace: [90370.899947] [] queued_spin_lock_slowpath+0xb/0xf [90370.899949] [] _raw_spin_lock+0x20/0x30 [90370.899974] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.899998] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.900022] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.900025] [] ? __kmalloc+0x1eb/0x230 [90370.900048] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.900057] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.900079] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.900103] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.900110] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.900135] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.900162] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.900188] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.900189] [] kthread+0xd1/0xe0 [90370.900191] [] ? insert_kthread_work+0x40/0x40 [90370.900194] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.900195] [] ? insert_kthread_work+0x40/0x40 [90370.900211] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [90370.902893] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [ldlm_bl_05:27290] [90370.902912] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.902920] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.902921] CPU: 19 PID: 27290 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.902922] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.902922] task: ffff8a5642ba0000 ti: ffff8a65693d0000 task.ti: ffff8a65693d0000 [90370.902925] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.902925] RSP: 0018:ffff8a65693d3c10 EFLAGS: 00000246 [90370.902926] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000990000 [90370.902926] RDX: ffff8a757dbdb8c0 RSI: 0000000000790001 RDI: ffff8a56673df99c [90370.902927] RBP: ffff8a65693d3c10 R08: ffff8a757dc5b8c0 R09: 0000000000000000 [90370.902928] R10: ffff8a7568900c58 R11: ffff8a7568900c48 R12: ffff8a65693d3bc0 [90370.902928] R13: ffff8a756f2d3140 R14: ffff8a65693d3b80 R15: ffffffffc0d2a5f8 [90370.902929] FS: 0000000000000000(0000) GS:ffff8a757dc40000(0000) knlGS:0000000000000000 [90370.902930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.902930] CR2: 00007f99a2b321a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.902931] Call Trace: [90370.902932] [] queued_spin_lock_slowpath+0xb/0xf [90370.902934] [] _raw_spin_lock+0x20/0x30 [90370.902951] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.902970] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.902989] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.903010] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.903016] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.903034] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.903053] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.903058] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.903077] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.903100] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.903102] [] ? wake_up_state+0x20/0x20 [90370.903125] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.903127] [] kthread+0xd1/0xe0 [90370.903128] [] ? insert_kthread_work+0x40/0x40 [90370.903130] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.903131] [] ? insert_kthread_work+0x40/0x40 [90370.903145] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.905894] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ldlm_bl_23:28517] [90370.905916] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.905926] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.905928] CPU: 20 PID: 28517 Comm: ldlm_bl_23 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.905929] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.905929] task: ffff8a65cab99080 ti: ffff8a65582d4000 task.ti: ffff8a65582d4000 [90370.905932] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.905933] RSP: 0018:ffff8a65582d7c10 EFLAGS: 00000246 [90370.905934] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a10000 [90370.905934] RDX: ffff8a757dcdb8c0 RSI: 0000000000b90001 RDI: ffff8a56673df99c [90370.905935] RBP: ffff8a65582d7c10 R08: ffff8a657e49b8c0 R09: 0000000000000000 [90370.905936] R10: ffff8a756e862258 R11: ffff8a756e862248 R12: ffff8a65582d7bc0 [90370.905936] R13: ffff8a756f2d2840 R14: ffff8a65582d7b80 R15: ffffffffc0d2a5f8 [90370.905937] FS: 0000000000000000(0000) GS:ffff8a657e480000(0000) knlGS:0000000000000000 [90370.905938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.905938] CR2: 00007f7ca3d3bbd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.905939] Call Trace: [90370.905941] [] queued_spin_lock_slowpath+0xb/0xf [90370.905943] [] _raw_spin_lock+0x20/0x30 [90370.905966] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.905986] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.906006] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.906027] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.906035] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.906053] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.906072] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.906078] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.906097] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.906120] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.906142] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.906144] [] kthread+0xd1/0xe0 [90370.906145] [] ? insert_kthread_work+0x40/0x40 [90370.906147] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.906149] [] ? insert_kthread_work+0x40/0x40 [90370.906162] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.908893] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [ldlm_bl_06:27291] [90370.908912] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.908920] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.908922] CPU: 21 PID: 27291 Comm: ldlm_bl_06 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.908922] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.908923] task: ffff8a5642ba5280 ti: ffff8a564f3a4000 task.ti: ffff8a564f3a4000 [90370.908925] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.908926] RSP: 0018:ffff8a564f3a7c10 EFLAGS: 00000246 [90370.908926] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a90000 [90370.908927] RDX: ffff8a757db9b8c0 RSI: 0000000000690001 RDI: ffff8a56673df99c [90370.908927] RBP: ffff8a564f3a7c10 R08: ffff8a757dc9b8c0 R09: 0000000000000000 [90370.908928] R10: ffff8a663db72258 R11: ffff8a663db72248 R12: ffff8a564f3a7bc0 [90370.908929] R13: ffff8a56bd752480 R14: ffff8a564f3a7b80 R15: ffffffffc0d2a5f8 [90370.908929] FS: 0000000000000000(0000) GS:ffff8a757dc80000(0000) knlGS:0000000000000000 [90370.908930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.908931] CR2: 00000000020ca6d8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.908931] Call Trace: [90370.908932] [] queued_spin_lock_slowpath+0xb/0xf [90370.908934] [] _raw_spin_lock+0x20/0x30 [90370.908953] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.908975] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.908995] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.909016] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.909022] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.909040] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.909059] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.909065] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.909083] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.909106] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.909108] [] ? wake_up_state+0x20/0x20 [90370.909129] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.909131] [] kthread+0xd1/0xe0 [90370.909132] [] ? insert_kthread_work+0x40/0x40 [90370.909134] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.909136] [] ? insert_kthread_work+0x40/0x40 [90370.909149] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.911893] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [ldlm_bl_24:28518] [90370.911916] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.911925] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.911926] CPU: 22 PID: 28518 Comm: ldlm_bl_24 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.911927] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.911928] task: ffff8a65cab98000 ti: ffff8a656905c000 task.ti: ffff8a656905c000 [90370.911931] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90370.911932] RSP: 0018:ffff8a656905fc10 EFLAGS: 00000246 [90370.911932] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b10000 [90370.911933] RDX: ffff8a657e49b8c0 RSI: 0000000000a10001 RDI: ffff8a56673df99c [90370.911934] RBP: ffff8a656905fc10 R08: ffff8a657e4db8c0 R09: 0000000000000000 [90370.911934] R10: ffff8a6637350058 R11: ffff8a6637350048 R12: ffff8a656905fbc0 [90370.911935] R13: ffff8a756f2d2b40 R14: ffff8a656905fb80 R15: ffffffffc0d2a5f8 [90370.911936] FS: 0000000000000000(0000) GS:ffff8a657e4c0000(0000) knlGS:0000000000000000 [90370.911936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.911937] CR2: 00007efc6a1ca000 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.911937] Call Trace: [90370.911940] [] queued_spin_lock_slowpath+0xb/0xf [90370.911942] [] _raw_spin_lock+0x20/0x30 [90370.911965] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.911984] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.912003] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.912022] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.912030] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.912048] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.912067] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.912074] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.912094] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.912116] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.912138] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.912140] [] kthread+0xd1/0xe0 [90370.912141] [] ? insert_kthread_work+0x40/0x40 [90370.912143] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.912145] [] ? insert_kthread_work+0x40/0x40 [90370.912158] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90370.914893] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_bl_22:28516] [90370.914912] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90370.914920] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90370.914921] CPU: 23 PID: 28516 Comm: ldlm_bl_22 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90370.914922] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90370.914923] task: ffff8a65cab9a100 ti: ffff8a65582d0000 task.ti: ffff8a65582d0000 [90370.914925] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90370.914925] RSP: 0018:ffff8a65582d3c10 EFLAGS: 00000246 [90370.914926] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b90000 [90370.914926] RDX: ffff8a757db1b8c0 RSI: 0000000000490001 RDI: ffff8a56673df99c [90370.914927] RBP: ffff8a65582d3c10 R08: ffff8a757dcdb8c0 R09: 0000000000000000 [90370.914928] R10: ffff8a65cbbf8458 R11: ffff8a65cbbf8448 R12: ffff8a65582d3bc0 [90370.914928] R13: ffff8a756f313e00 R14: ffff8a65582d3b80 R15: ffffffffc0d2a5f8 [90370.914929] FS: 0000000000000000(0000) GS:ffff8a757dcc0000(0000) knlGS:0000000000000000 [90370.914930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90370.914930] CR2: 00007f0a08c7a9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90370.914930] Call Trace: [90370.914932] [] queued_spin_lock_slowpath+0xb/0xf [90370.914934] [] _raw_spin_lock+0x20/0x30 [90370.914951] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90370.914972] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90370.914991] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90370.915012] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90370.915018] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90370.915036] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.915055] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90370.915061] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90370.915081] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90370.915103] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90370.915124] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90370.915126] [] kthread+0xd1/0xe0 [90370.915127] [] ? insert_kthread_work+0x40/0x40 [90370.915129] [] ret_from_fork_nospec_begin+0x7/0x21 [90370.915130] [] ? insert_kthread_work+0x40/0x40 [90370.915144] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90374.830824] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ldlm_bl_15:28509] [90374.830842] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90374.830850] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90374.830852] CPU: 1 PID: 28509 Comm: ldlm_bl_15 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90374.830852] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90374.830853] task: ffff8a65c37ba100 ti: ffff8a65582ac000 task.ti: ffff8a65582ac000 [90374.830855] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90374.830856] RSP: 0018:ffff8a65582afc10 EFLAGS: 00000246 [90374.830856] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000090000 [90374.830857] RDX: ffff8a657e31b8c0 RSI: 0000000000410001 RDI: ffff8a56673df99c [90374.830858] RBP: ffff8a65582afc10 R08: ffff8a757da1b8c0 R09: 0000000000000000 [90374.830858] R10: ffff8a5650be3658 R11: ffff8a5650be3648 R12: ffff8a65582afbc0 [90374.830859] R13: ffff8a56673dfe00 R14: ffff8a65582afb80 R15: ffffffffc0d2a5f8 [90374.830860] FS: 0000000000000000(0000) GS:ffff8a757da00000(0000) knlGS:0000000000000000 [90374.830860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90374.830861] CR2: 00007efff91a9000 CR3: 000000010d78e000 CR4: 00000000001607e0 [90374.830861] Call Trace: [90374.830863] [] queued_spin_lock_slowpath+0xb/0xf [90374.830865] [] _raw_spin_lock+0x20/0x30 [90374.830885] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90374.830904] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90374.830925] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90374.830944] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90374.830950] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90374.830968] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90374.830987] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90374.830993] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90374.831011] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90374.831035] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90374.831056] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90374.831058] [] kthread+0xd1/0xe0 [90374.831059] [] ? insert_kthread_work+0x40/0x40 [90374.831061] [] ret_from_fork_nospec_begin+0x7/0x21 [90374.831062] [] ? insert_kthread_work+0x40/0x40 [90374.831076] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90377.855786] INFO: rcu_sched self-detected stall on CPU { 12} (t=60000 jiffies g=678150 c=678149 q=2698) [90377.855787] Task dump for CPU 0: [90377.855789] ldlm_bl_18 R running task 0 28512 2 0x00000088 [90377.855789] Call Trace: [90377.855794] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.855796] [] ? _raw_spin_lock+0x20/0x30 [90377.855830] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.855847] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.855865] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.855882] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.855894] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.855911] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.855928] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.855934] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.855952] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.855973] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.855994] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.855996] [] ? kthread+0xd1/0xe0 [90377.855997] [] ? insert_kthread_work+0x40/0x40 [90377.855999] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856000] [] ? insert_kthread_work+0x40/0x40 [90377.856001] Task dump for CPU 1: [90377.856002] ldlm_bl_15 R running task 0 28509 2 0x00000088 [90377.856002] Call Trace: [90377.856004] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.856005] [] ? _raw_spin_lock+0x20/0x30 [90377.856022] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.856039] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.856057] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.856074] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.856079] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.856096] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856114] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856119] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.856136] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856156] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856176] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856177] [] ? kthread+0xd1/0xe0 [90377.856179] [] ? insert_kthread_work+0x40/0x40 [90377.856180] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856181] [] ? insert_kthread_work+0x40/0x40 [90377.856182] Task dump for CPU 2: [90377.856183] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.856183] Call Trace: [90377.856200] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856220] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856222] [] ? wake_up_state+0x20/0x20 [90377.856242] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856243] [] ? kthread+0xd1/0xe0 [90377.856244] [] ? insert_kthread_work+0x40/0x40 [90377.856246] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856247] [] ? insert_kthread_work+0x40/0x40 [90377.856247] Task dump for CPU 3: [90377.856249] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.856249] Call Trace: [90377.856250] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.856252] [] ? _raw_spin_lock+0x20/0x30 [90377.856269] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.856286] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.856303] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.856321] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.856326] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.856343] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856361] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856366] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.856383] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856403] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856424] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856425] [] ? kthread+0xd1/0xe0 [90377.856426] [] ? insert_kthread_work+0x40/0x40 [90377.856428] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856429] [] ? insert_kthread_work+0x40/0x40 [90377.856429] Task dump for CPU 4: [90377.856430] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.856430] Call Trace: [90377.856432] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.856433] [] ? _raw_spin_lock+0x20/0x30 [90377.856450] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.856467] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.856485] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.856503] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.856508] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.856524] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856542] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856547] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.856564] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856585] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856605] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856606] [] ? kthread+0xd1/0xe0 [90377.856607] [] ? insert_kthread_work+0x40/0x40 [90377.856609] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856610] [] ? insert_kthread_work+0x40/0x40 [90377.856610] Task dump for CPU 5: [90377.856612] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.856612] Call Trace: [90377.856613] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.856615] [] ? _raw_spin_lock+0x20/0x30 [90377.856632] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.856650] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.856667] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.856686] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.856691] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.856708] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856725] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.856730] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.856748] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856768] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856770] [] ? wake_up_state+0x20/0x20 [90377.856773] INFO: rcu_sched self-detected stall on CPU [90377.856774] INFO: rcu_sched self-detected stall on CPU [90377.856775] INFO: rcu_sched self-detected stall on CPU [90377.856776] INFO: rcu_sched self-detected stall on CPU [90377.856777] INFO: rcu_sched self-detected stall on CPU [90377.856778] INFO: rcu_sched self-detected stall on CPU [90377.856778] INFO: rcu_sched self-detected stall on CPU [90377.856779] INFO: rcu_sched self-detected stall on CPU [90377.856780] INFO: rcu_sched self-detected stall on CPU [90377.856781] INFO: rcu_sched self-detected stall on CPU [90377.856783] INFO: rcu_sched self-detected stall on CPU [90377.856783] INFO: rcu_sched self-detected stall on CPU [90377.856784] INFO: rcu_sched self-detected stall on CPU [90377.856804] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856805] INFO: rcu_sched self-detected stall on CPU [90377.856805] { [90377.856806] { [90377.856806] { [90377.856807] { [90377.856808] { [90377.856809] { [90377.856809] { [90377.856809] { [90377.856810] { [90377.856811] { [90377.856812] { [90377.856813] { [90377.856813] { [90377.856815] [] ? kthread+0xd1/0xe0 [90377.856816] { [90377.856816] 19 [90377.856817] 5 [90377.856818] 17 [90377.856819] 3 [90377.856819] 16 [90377.856820] 4 [90377.856821] 7 [90377.856821] 1 [90377.856823] 13 [90377.856824] 6 [90377.856825] 21 [90377.856825] 2 [90377.856826] 8 [90377.856827] [] ? insert_kthread_work+0x40/0x40 [90377.856829] 10 [90377.856829] } [90377.856829] } [90377.856830] } [90377.856830] } [90377.856831] } [90377.856831] } [90377.856832] } [90377.856832] } [90377.856833] } [90377.856833] } [90377.856834] } [90377.856835] } [90377.856835] } [90377.856837] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856838] [] ? insert_kthread_work+0x40/0x40 [90377.856839] Task dump for CPU 6: [90377.856842] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856843] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856844] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856845] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856846] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856847] } (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856849] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856850] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856851] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856852] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856852] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856853] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856854] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856855] (t=60001 jiffies g=678150 c=678149 q=2698) [90377.856857] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.856857] Call Trace: [90377.856875] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.856892] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.856912] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856914] [] ? wake_up_state+0x20/0x20 [90377.856933] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.856935] [] ? kthread+0xd1/0xe0 [90377.856936] [] ? insert_kthread_work+0x40/0x40 [90377.856937] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.856938] [] ? insert_kthread_work+0x40/0x40 [90377.856939] Task dump for CPU 7: [90377.856940] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.856940] Call Trace: [90377.856958] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.856978] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.856979] [] ? wake_up_state+0x20/0x20 [90377.856999] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.857000] [] ? kthread+0xd1/0xe0 [90377.857001] [] ? insert_kthread_work+0x40/0x40 [90377.857003] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.857004] [] ? insert_kthread_work+0x40/0x40 [90377.857005] Task dump for CPU 8: [90377.857006] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.857006] Call Trace: [90377.857024] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.857044] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.857045] [] ? wake_up_state+0x20/0x20 [90377.857064] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.857066] [] ? kthread+0xd1/0xe0 [90377.857067] [] ? insert_kthread_work+0x40/0x40 [90377.857068] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.857069] [] ? insert_kthread_work+0x40/0x40 [90377.857070] Task dump for CPU 10: [90377.857071] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.857071] Call Trace: [90377.857073] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.857074] [] ? _raw_spin_lock+0x20/0x30 [90377.857091] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.857108] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.857126] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.857144] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.857149] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.857166] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857184] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857190] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.857206] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.857226] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.857246] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.857247] [] ? kthread+0xd1/0xe0 [90377.857248] [] ? insert_kthread_work+0x40/0x40 [90377.857250] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.857251] [] ? insert_kthread_work+0x40/0x40 [90377.857252] Task dump for CPU 12: [90377.857253] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.857253] Call Trace: [90377.857255] [] sched_show_task+0xa8/0x110 [90377.857257] [] dump_cpu_task+0x39/0x70 [90377.857261] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.857263] [] rcu_check_callbacks+0x442/0x730 [90377.857267] [] ? tick_sched_do_timer+0x50/0x50 [90377.857271] [] update_process_times+0x46/0x80 [90377.857272] [] tick_sched_handle+0x30/0x70 [90377.857274] [] tick_sched_timer+0x39/0x80 [90377.857275] [] __hrtimer_run_queues+0x10e/0x270 [90377.857277] [] hrtimer_interrupt+0xaf/0x1d0 [90377.857279] [] local_apic_timer_interrupt+0x3b/0x60 [90377.857281] [] smp_apic_timer_interrupt+0x43/0x60 [90377.857283] [] apic_timer_interrupt+0x16a/0x170 [90377.857285] [] ? native_queued_spin_lock_slowpath+0x158/0x200 [90377.857286] [] queued_spin_lock_slowpath+0xb/0xf [90377.857287] [] _raw_spin_lock+0x20/0x30 [90377.857303] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.857321] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [90377.857342] [] ? class_handle_unhash_nolock+0xd8/0x150 [obdclass] [90377.857347] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [90377.857364] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [90377.857381] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.857399] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.857404] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.857421] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857439] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857444] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.857461] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.857482] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.857483] [] ? wake_up_state+0x20/0x20 [90377.857503] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.857505] [] kthread+0xd1/0xe0 [90377.857506] [] ? insert_kthread_work+0x40/0x40 [90377.857507] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.857509] [] ? insert_kthread_work+0x40/0x40 [90377.857509] Task dump for CPU 13: [90377.857510] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.857510] Call Trace: [90377.857512] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.857513] [] ? _raw_spin_lock+0x20/0x30 [90377.857530] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.857548] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.857565] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.857583] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.857588] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.857605] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857623] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857628] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.857645] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.857665] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.857685] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.857686] [] ? kthread+0xd1/0xe0 [90377.857687] [] ? insert_kthread_work+0x40/0x40 [90377.857689] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.857690] [] ? insert_kthread_work+0x40/0x40 [90377.857690] Task dump for CPU 16: [90377.857691] Task dump for CPU 0: [90377.857693] ldlm_bl_14 R [90377.857694] ldlm_bl_18 R [90377.857695] running task [90377.857696] 0 28508 2 0x00000088 [90377.857697] running task 0 28512 2 0x00000088 [90377.857698] Call Trace: [90377.857699] Call Trace: [90377.857701] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.857704] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.857706] [] ? _raw_spin_lock+0x20/0x30 [90377.857708] [] ? _raw_spin_lock+0x20/0x30 [90377.857725] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.857744] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.857762] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.857781] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.857799] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.857821] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.857839] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.857859] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.857864] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.857869] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.857886] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857905] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857922] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857941] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.857946] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.857952] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.857969] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.857988] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858008] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858041] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858061] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858095] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858097] [] ? kthread+0xd1/0xe0 [90377.858099] [] ? kthread+0xd1/0xe0 [90377.858101] [] ? insert_kthread_work+0x40/0x40 [90377.858103] [] ? insert_kthread_work+0x40/0x40 [90377.858105] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858107] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858109] [] ? insert_kthread_work+0x40/0x40 [90377.858111] [] ? insert_kthread_work+0x40/0x40 [90377.858111] Task dump for CPU 17: [90377.858112] Task dump for CPU 1: [90377.858113] INFO: rcu_sched detected stalls on CPUs/tasks: { [90377.858114] ldlm_bl_11 R [90377.858116] ldlm_bl_15 R [90377.858116] running task [90377.858117] 0 28505 2 0x00000088 [90377.858119] running task 0 28509 2 0x00000088 [90377.858120] Call Trace: [90377.858121] Call Trace: [90377.858122] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.858125] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.858126] [] ? _raw_spin_lock+0x20/0x30 [90377.858129] [] ? _raw_spin_lock+0x20/0x30 [90377.858145] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.858177] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.858195] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.858227] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.858245] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.858277] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.858296] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.858327] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.858332] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.858340] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.858357] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.858389] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.858407] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.858439] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.858444] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.858451] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.858468] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858500] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858520] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858553] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858574] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858607] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858609] [] ? kthread+0xd1/0xe0 [90377.858611] [] ? kthread+0xd1/0xe0 [90377.858613] [] ? insert_kthread_work+0x40/0x40 [90377.858615] [] ? insert_kthread_work+0x40/0x40 [90377.858617] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858619] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858620] [] ? insert_kthread_work+0x40/0x40 [90377.858622] [] ? insert_kthread_work+0x40/0x40 [90377.858623] Task dump for CPU 19: [90377.858624] Task dump for CPU 2: [90377.858626] ldlm_bl_05 R [90377.858627] ldlm_bl_07 R [90377.858627] running task [90377.858628] 0 27290 2 0x00000088 [90377.858630] running task 0 27292 2 0x00000088 [90377.858631] Call Trace: [90377.858631] Call Trace: [90377.858649] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858681] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858701] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858734] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858736] [] ? wake_up_state+0x20/0x20 [90377.858738] [] ? wake_up_state+0x20/0x20 [90377.858757] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858791] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858793] [] ? kthread+0xd1/0xe0 [90377.858795] [] ? kthread+0xd1/0xe0 [90377.858797] [] ? insert_kthread_work+0x40/0x40 [90377.858799] [] ? insert_kthread_work+0x40/0x40 [90377.858801] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858803] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.858804] [] ? insert_kthread_work+0x40/0x40 [90377.858806] [] ? insert_kthread_work+0x40/0x40 [90377.858807] Task dump for CPU 21: [90377.858808] Task dump for CPU 3: [90377.858810] ldlm_bl_06 R [90377.858811] ldlm_bl_17 R [90377.858812] running task [90377.858813] 0 27291 2 0x00000088 [90377.858815] running task 0 28511 2 0x00000088 [90377.858815] Call Trace: [90377.858816] Call Trace: [90377.858834] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.858836] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.858856] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.858858] [] ? _raw_spin_lock+0x20/0x30 [90377.858860] [] ? wake_up_state+0x20/0x20 [90377.858891] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.858911] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.858943] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.858945] [] ? kthread+0xd1/0xe0 [90377.858977] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.858978] [] ? insert_kthread_work+0x40/0x40 [90377.859010] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.859012] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.859019] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.859021] [] ? insert_kthread_work+0x40/0x40 [90377.859053] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859085] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859092] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.859123] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.859157] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.859190] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.859192] [] ? kthread+0xd1/0xe0 [90377.859193] [] ? insert_kthread_work+0x40/0x40 [90377.859195] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.859196] [] ? insert_kthread_work+0x40/0x40 [90377.859197] Task dump for CPU 4: [90377.859198] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.859199] Call Trace: [90377.859200] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.859202] [] ? _raw_spin_lock+0x20/0x30 [90377.859233] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.859264] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.859296] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.859328] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.859335] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.859366] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859398] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859405] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.859437] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.859470] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.859504] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.859505] [] ? kthread+0xd1/0xe0 [90377.859507] [] ? insert_kthread_work+0x40/0x40 [90377.859508] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.859510] [] ? insert_kthread_work+0x40/0x40 [90377.859510] Task dump for CPU 5: [90377.859512] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.859512] Call Trace: [90377.859513] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.859515] [] ? _raw_spin_lock+0x20/0x30 [90377.859546] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.859578] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.859609] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.859641] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.859648] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.859679] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859711] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.859718] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.859750] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.859783] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.859785] [] ? wake_up_state+0x20/0x20 [90377.859818] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.859820] [] ? kthread+0xd1/0xe0 [90377.859821] [] ? insert_kthread_work+0x40/0x40 [90377.859823] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.859824] [] ? insert_kthread_work+0x40/0x40 [90377.859825] Task dump for CPU 6: [90377.859827] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.859827] Call Trace: [90377.859859] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.859890] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.859924] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.859925] [] ? wake_up_state+0x20/0x20 [90377.859959] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.859960] [] ? kthread+0xd1/0xe0 [90377.859962] [] ? insert_kthread_work+0x40/0x40 [90377.859963] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.859965] [] ? insert_kthread_work+0x40/0x40 [90377.859965] Task dump for CPU 7: [90377.859967] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.859967] Call Trace: [90377.859999] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.860032] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.860034] [] ? wake_up_state+0x20/0x20 [90377.860067] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.860068] [] ? kthread+0xd1/0xe0 [90377.860070] [] ? insert_kthread_work+0x40/0x40 [90377.860071] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.860073] [] ? insert_kthread_work+0x40/0x40 [90377.860073] Task dump for CPU 8: [90377.860075] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.860075] Call Trace: [90377.860107] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.860140] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.860142] [] ? wake_up_state+0x20/0x20 [90377.860175] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.860177] [] ? kthread+0xd1/0xe0 [90377.860178] [] ? insert_kthread_work+0x40/0x40 [90377.860180] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.860181] [] ? insert_kthread_work+0x40/0x40 [90377.860182] Task dump for CPU 10: [90377.860183] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.860183] Call Trace: [90377.860185] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.860186] [] ? _raw_spin_lock+0x20/0x30 [90377.860217] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.860249] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.860280] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.860312] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.860319] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.860351] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.860382] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.860390] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.860421] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.860454] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.860488] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.860489] [] ? kthread+0xd1/0xe0 [90377.860491] [] ? insert_kthread_work+0x40/0x40 [90377.860492] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.860494] [] ? insert_kthread_work+0x40/0x40 [90377.860494] Task dump for CPU 12: [90377.860496] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.860498] Call Trace: [90377.860530] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.860563] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.860565] [] ? wake_up_state+0x20/0x20 [90377.860598] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.860600] [] ? kthread+0xd1/0xe0 [90377.860601] [] ? insert_kthread_work+0x40/0x40 [90377.860603] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.860604] [] ? insert_kthread_work+0x40/0x40 [90377.860605] Task dump for CPU 13: [90377.860606] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.860606] Call Trace: [90377.860608] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.860610] [] ? _raw_spin_lock+0x20/0x30 [90377.860641] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.860672] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.860704] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.860735] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.860743] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.860774] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.860806] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.860813] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.860845] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.860878] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.860911] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.860913] [] ? kthread+0xd1/0xe0 [90377.860914] [] ? insert_kthread_work+0x40/0x40 [90377.860916] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.860917] [] ? insert_kthread_work+0x40/0x40 [90377.860919] Task dump for CPU 16: [90377.860919] Task dump for CPU 0: [90377.860921] ldlm_bl_14 R [90377.860922] ldlm_bl_18 R [90377.860923] running task [90377.860925] 0 28508 2 0x00000088 [90377.860926] running task 0 28512 2 0x00000088 [90377.860927] Call Trace: [90377.860928] Call Trace: [90377.860930] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.860932] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.860934] [] ? _raw_spin_lock+0x20/0x30 [90377.860936] [] ? _raw_spin_lock+0x20/0x30 [90377.860968] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.860987] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.861019] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.861039] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.861072] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.861091] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.861123] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.861143] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.861150] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.861156] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.861187] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861206] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861238] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861258] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861265] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.861271] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.861303] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.861323] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.861356] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.861379] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.861413] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.861435] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.861437] [] ? kthread+0xd1/0xe0 [90377.861439] [] ? kthread+0xd1/0xe0 [90377.861441] [] ? insert_kthread_work+0x40/0x40 [90377.861442] [] ? insert_kthread_work+0x40/0x40 [90377.861445] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.861447] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.861448] [] ? insert_kthread_work+0x40/0x40 [90377.861450] [] ? insert_kthread_work+0x40/0x40 [90377.861451] Task dump for CPU 17: [90377.861452] Task dump for CPU 1: [90377.861454] ldlm_bl_11 R [90377.861455] ldlm_bl_15 R [90377.861456] running task [90377.861457] 0 28505 2 0x00000088 [90377.861459] running task 0 28509 2 0x00000088 [90377.861460] Call Trace: [90377.861460] Call Trace: [90377.861463] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.861464] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.861467] [] ? _raw_spin_lock+0x20/0x30 [90377.861469] [] ? _raw_spin_lock+0x20/0x30 [90377.861500] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.861519] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.861550] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.861570] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.861602] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.861621] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.861653] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.861673] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.861680] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.861685] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.861717] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861737] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861769] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861789] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.861796] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.861802] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.861834] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.861853] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.861886] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.861908] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.861942] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.861963] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.861965] [] ? kthread+0xd1/0xe0 [90377.861967] [] ? kthread+0xd1/0xe0 [90377.861969] [] ? insert_kthread_work+0x40/0x40 [90377.861971] [] ? insert_kthread_work+0x40/0x40 [90377.861973] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.861975] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.861977] [] ? insert_kthread_work+0x40/0x40 [90377.861979] [] ? insert_kthread_work+0x40/0x40 [90377.861980] Task dump for CPU 19: [90377.861981] Task dump for CPU 2: [90377.861983] ldlm_bl_05 R [90377.861984] ldlm_bl_07 R [90377.861984] running task [90377.861986] 0 27290 2 0x00000088 [90377.861987] running task 0 27292 2 0x00000088 [90377.861988] Call Trace: [90377.861989] Call Trace: [90377.862021] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.862041] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.862074] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.862096] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.862099] [] ? wake_up_state+0x20/0x20 [90377.862100] [] ? wake_up_state+0x20/0x20 [90377.862134] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.862156] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.862158] [] ? kthread+0xd1/0xe0 [90377.862160] [] ? kthread+0xd1/0xe0 [90377.862162] [] ? insert_kthread_work+0x40/0x40 [90377.862164] [] ? insert_kthread_work+0x40/0x40 [90377.862166] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.862168] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.862170] [] ? insert_kthread_work+0x40/0x40 [90377.862172] [] ? insert_kthread_work+0x40/0x40 [90377.862173] Task dump for CPU 21: [90377.862173] Task dump for CPU 3: [90377.862175] ldlm_bl_06 R [90377.862176] ldlm_bl_17 R [90377.862177] running task [90377.862178] 0 27291 2 0x00000088 [90377.862180] running task 0 28511 2 0x00000088 [90377.862181] Call Trace: [90377.862181] Call Trace: [90377.862184] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.862186] [] sched_show_task+0xa8/0x110 [90377.862188] [] ? _raw_spin_lock+0x20/0x30 [90377.862191] [] dump_cpu_task+0x39/0x70 [90377.862210] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.862214] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.862233] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.862235] [] rcu_check_callbacks+0x442/0x730 [90377.862255] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.862258] [] ? tick_sched_do_timer+0x50/0x50 [90377.862279] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.862283] [] update_process_times+0x46/0x80 [90377.862288] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.862290] [] tick_sched_handle+0x30/0x70 [90377.862309] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862311] [] tick_sched_timer+0x39/0x80 [90377.862331] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862333] [] __hrtimer_run_queues+0x10e/0x270 [90377.862338] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.862340] [] hrtimer_interrupt+0xaf/0x1d0 [90377.862359] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.862362] [] local_apic_timer_interrupt+0x3b/0x60 [90377.862384] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.862386] [] smp_apic_timer_interrupt+0x43/0x60 [90377.862408] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.862410] [] apic_timer_interrupt+0x16a/0x170 [90377.862412] [] ? kthread+0xd1/0xe0 [90377.862414] [] ? insert_kthread_work+0x40/0x40 [90377.862416] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.862418] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.862421] [] queued_spin_lock_slowpath+0xb/0xf [90377.862422] [] ? insert_kthread_work+0x40/0x40 [90377.862425] [] _raw_spin_lock+0x20/0x30 [90377.862425] Task dump for CPU 4: [90377.862457] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.862489] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.862521] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.862522] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.862554] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.862555] Call Trace: [90377.862562] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.862595] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862597] [] sched_show_task+0xa8/0x110 [90377.862629] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862631] [] dump_cpu_task+0x39/0x70 [90377.862638] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.862640] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.862672] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.862674] [] rcu_check_callbacks+0x442/0x730 [90377.862707] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.862709] [] ? tick_sched_do_timer+0x50/0x50 [90377.862711] [] ? wake_up_state+0x20/0x20 [90377.862713] [] update_process_times+0x46/0x80 [90377.862747] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.862749] [] tick_sched_handle+0x30/0x70 [90377.862751] [] kthread+0xd1/0xe0 [90377.862752] [] tick_sched_timer+0x39/0x80 [90377.862754] [] ? insert_kthread_work+0x40/0x40 [90377.862756] [] __hrtimer_run_queues+0x10e/0x270 [90377.862758] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.862760] [] hrtimer_interrupt+0xaf/0x1d0 [90377.862762] [] ? insert_kthread_work+0x40/0x40 [90377.862763] [] local_apic_timer_interrupt+0x3b/0x60 [90377.862764] [] smp_apic_timer_interrupt+0x43/0x60 [90377.862766] [] apic_timer_interrupt+0x16a/0x170 [90377.862768] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.862770] [] queued_spin_lock_slowpath+0xb/0xf [90377.862772] [] _raw_spin_lock+0x20/0x30 [90377.862790] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.862810] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.862829] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.862849] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.862855] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.862874] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862893] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.862899] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.862918] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.862939] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.862962] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.862963] [] kthread+0xd1/0xe0 [90377.862965] [] ? insert_kthread_work+0x40/0x40 [90377.862967] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.862968] [] ? insert_kthread_work+0x40/0x40 [90377.862969] Task dump for CPU 5: [90377.862970] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.862970] Call Trace: [90377.862972] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.862973] [] ? _raw_spin_lock+0x20/0x30 [90377.862992] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.863012] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.863031] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.863051] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.863057] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.863076] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863095] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863100] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.863120] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863143] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863144] [] ? wake_up_state+0x20/0x20 [90377.863166] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863167] [] ? kthread+0xd1/0xe0 [90377.863169] [] ? insert_kthread_work+0x40/0x40 [90377.863171] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863172] [] ? insert_kthread_work+0x40/0x40 [90377.863172] Task dump for CPU 6: [90377.863174] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.863174] Call Trace: [90377.863194] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.863213] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.863235] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863237] [] ? wake_up_state+0x20/0x20 [90377.863258] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863260] [] ? kthread+0xd1/0xe0 [90377.863261] [] ? insert_kthread_work+0x40/0x40 [90377.863263] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863264] [] ? insert_kthread_work+0x40/0x40 [90377.863265] Task dump for CPU 7: [90377.863266] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.863266] Call Trace: [90377.863285] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863307] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863309] [] ? wake_up_state+0x20/0x20 [90377.863330] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863332] [] ? kthread+0xd1/0xe0 [90377.863333] [] ? insert_kthread_work+0x40/0x40 [90377.863335] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863336] [] ? insert_kthread_work+0x40/0x40 [90377.863337] Task dump for CPU 8: [90377.863338] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.863338] Call Trace: [90377.863358] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863380] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863381] [] ? wake_up_state+0x20/0x20 [90377.863402] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863404] [] ? kthread+0xd1/0xe0 [90377.863405] [] ? insert_kthread_work+0x40/0x40 [90377.863407] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863408] [] ? insert_kthread_work+0x40/0x40 [90377.863409] Task dump for CPU 10: [90377.863410] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.863410] Call Trace: [90377.863412] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.863413] [] ? _raw_spin_lock+0x20/0x30 [90377.863432] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.863452] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.863472] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.863491] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.863497] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.863515] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863536] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863541] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.863560] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863581] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863604] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863605] [] ? kthread+0xd1/0xe0 [90377.863607] [] ? insert_kthread_work+0x40/0x40 [90377.863608] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863609] [] ? insert_kthread_work+0x40/0x40 [90377.863610] Task dump for CPU 12: [90377.863611] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.863611] Call Trace: [90377.863631] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863652] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863654] [] ? wake_up_state+0x20/0x20 [90377.863675] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863677] [] ? kthread+0xd1/0xe0 [90377.863678] [] ? insert_kthread_work+0x40/0x40 [90377.863680] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863681] [] ? insert_kthread_work+0x40/0x40 [90377.863682] Task dump for CPU 13: [90377.863683] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.863683] Call Trace: [90377.863685] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.863686] [] ? _raw_spin_lock+0x20/0x30 [90377.863705] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.863724] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.863743] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.863763] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.863768] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.863787] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863806] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.863812] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.863831] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.863853] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.863875] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.863877] [] ? kthread+0xd1/0xe0 [90377.863878] [] ? insert_kthread_work+0x40/0x40 [90377.863880] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.863881] [] ? insert_kthread_work+0x40/0x40 [90377.863882] Task dump for CPU 16: [90377.863882] Task dump for CPU 0: [90377.863883] ldlm_bl_14 R running task [90377.863884] 0 28508 2 0x00000088 [90377.863884] Call Trace: [90377.863884] ldlm_bl_18 R running task 0 28512 2 0x00000088 [90377.863886] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.863886] Call Trace: [90377.863888] [] ? _raw_spin_lock+0x20/0x30 [90377.863889] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.863908] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.863909] [] ? _raw_spin_lock+0x20/0x30 [90377.863928] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.863947] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.863966] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.863986] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.864005] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.864024] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.864029] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.864049] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.864069] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864074] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.864093] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864111] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864116] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.864135] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864154] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864160] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.864181] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864199] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864220] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864242] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864243] [] ? kthread+0xd1/0xe0 [90377.864265] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864267] [] ? insert_kthread_work+0x40/0x40 [90377.864268] [] ? kthread+0xd1/0xe0 [90377.864269] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864271] [] ? insert_kthread_work+0x40/0x40 [90377.864272] [] ? insert_kthread_work+0x40/0x40 [90377.864273] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864274] Task dump for CPU 17: [90377.864275] [] ? insert_kthread_work+0x40/0x40 [90377.864276] Task dump for CPU 1: [90377.864276] ldlm_bl_11 R running task 0 28505 2 0x00000088 [90377.864277] Call Trace: [90377.864278] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.864279] ldlm_bl_15 R running task 0 28509 2 0x00000088 [90377.864281] [] ? _raw_spin_lock+0x20/0x30 [90377.864281] Call Trace: [90377.864300] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.864301] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.864320] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.864322] [] ? _raw_spin_lock+0x20/0x30 [90377.864342] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.864360] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.864380] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.864399] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.864404] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.864423] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.864441] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864460] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.864480] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864485] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.864489] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.864507] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864528] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864547] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.864569] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864574] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.864595] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864614] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864616] [] ? kthread+0xd1/0xe0 [90377.864638] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864639] [] ? insert_kthread_work+0x40/0x40 [90377.864660] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864662] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864663] [] ? kthread+0xd1/0xe0 [90377.864665] [] ? insert_kthread_work+0x40/0x40 [90377.864666] [] ? insert_kthread_work+0x40/0x40 [90377.864666] Task dump for CPU 19: [90377.864667] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864669] [] ? insert_kthread_work+0x40/0x40 [90377.864670] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.864670] Task dump for CPU 2: [90377.864670] Call Trace: [90377.864671] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.864691] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864691] Call Trace: [90377.864713] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864732] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864734] [] ? wake_up_state+0x20/0x20 [90377.864755] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864777] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864778] [] ? wake_up_state+0x20/0x20 [90377.864779] [] ? kthread+0xd1/0xe0 [90377.864801] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864802] [] ? insert_kthread_work+0x40/0x40 [90377.864803] [] ? kthread+0xd1/0xe0 [90377.864805] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864806] [] ? insert_kthread_work+0x40/0x40 [90377.864807] [] ? insert_kthread_work+0x40/0x40 [90377.864809] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864809] Task dump for CPU 21: [90377.864810] [] ? insert_kthread_work+0x40/0x40 [90377.864811] Task dump for CPU 3: [90377.864811] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.864812] Call Trace: [90377.864812] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.864832] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.864833] Call Trace: [90377.864834] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.864856] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.864858] [] ? _raw_spin_lock+0x20/0x30 [90377.864859] [] ? wake_up_state+0x20/0x20 [90377.864877] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.864899] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.864918] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.864920] [] ? kthread+0xd1/0xe0 [90377.864939] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.864941] [] ? insert_kthread_work+0x40/0x40 [90377.864961] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.864962] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.864967] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.864969] [] ? insert_kthread_work+0x40/0x40 [90377.864987] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865006] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865012] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.865032] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865053] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865074] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865076] [] ? kthread+0xd1/0xe0 [90377.865077] [] ? insert_kthread_work+0x40/0x40 [90377.865079] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865080] [] ? insert_kthread_work+0x40/0x40 [90377.865081] Task dump for CPU 4: [90377.865082] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.865082] Call Trace: [90377.865083] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.865085] [] ? _raw_spin_lock+0x20/0x30 [90377.865103] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.865122] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.865142] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.865163] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.865168] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.865187] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865205] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865211] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.865231] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865254] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865275] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865277] [] ? kthread+0xd1/0xe0 [90377.865278] [] ? insert_kthread_work+0x40/0x40 [90377.865280] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865281] [] ? insert_kthread_work+0x40/0x40 [90377.865281] Task dump for CPU 5: [90377.865283] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.865283] Call Trace: [90377.865284] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.865286] [] ? _raw_spin_lock+0x20/0x30 [90377.865303] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.865322] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.865343] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.865364] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.865369] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.865387] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865408] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865414] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.865432] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865453] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865455] [] ? wake_up_state+0x20/0x20 [90377.865476] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865477] [] ? kthread+0xd1/0xe0 [90377.865479] [] ? insert_kthread_work+0x40/0x40 [90377.865480] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865481] [] ? insert_kthread_work+0x40/0x40 [90377.865482] Task dump for CPU 6: [90377.865483] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.865483] Call Trace: [90377.865504] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.865525] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.865547] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865549] [] ? wake_up_state+0x20/0x20 [90377.865570] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865572] [] ? kthread+0xd1/0xe0 [90377.865573] [] ? insert_kthread_work+0x40/0x40 [90377.865575] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865576] [] ? insert_kthread_work+0x40/0x40 [90377.865576] Task dump for CPU 7: [90377.865577] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.865578] Call Trace: [90377.865597] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865618] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865619] [] ? wake_up_state+0x20/0x20 [90377.865640] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865642] [] ? kthread+0xd1/0xe0 [90377.865643] [] ? insert_kthread_work+0x40/0x40 [90377.865645] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865646] [] ? insert_kthread_work+0x40/0x40 [90377.865646] Task dump for CPU 8: [90377.865647] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.865648] Call Trace: [90377.865667] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865687] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865689] [] ? wake_up_state+0x20/0x20 [90377.865710] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865711] [] ? kthread+0xd1/0xe0 [90377.865712] [] ? insert_kthread_work+0x40/0x40 [90377.865714] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865715] [] ? insert_kthread_work+0x40/0x40 [90377.865716] Task dump for CPU 10: [90377.865717] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.865717] Call Trace: [90377.865718] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.865720] [] ? _raw_spin_lock+0x20/0x30 [90377.865738] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.865756] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.865777] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.865796] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.865801] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.865819] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865837] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.865843] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.865861] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865882] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865904] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865905] [] ? kthread+0xd1/0xe0 [90377.865906] [] ? insert_kthread_work+0x40/0x40 [90377.865908] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865909] [] ? insert_kthread_work+0x40/0x40 [90377.865910] Task dump for CPU 12: [90377.865911] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.865911] Call Trace: [90377.865930] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.865953] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.865955] [] ? wake_up_state+0x20/0x20 [90377.865976] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.865978] [] ? kthread+0xd1/0xe0 [90377.865979] [] ? insert_kthread_work+0x40/0x40 [90377.865981] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.865982] [] ? insert_kthread_work+0x40/0x40 [90377.865982] Task dump for CPU 13: [90377.865983] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.865984] Call Trace: [90377.865985] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.865987] [] ? _raw_spin_lock+0x20/0x30 [90377.866004] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.866023] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.866044] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.866065] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.866070] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.866089] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866108] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866113] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.866131] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.866152] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.866173] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.866175] [] ? kthread+0xd1/0xe0 [90377.866176] [] ? insert_kthread_work+0x40/0x40 [90377.866178] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.866179] [] ? insert_kthread_work+0x40/0x40 [90377.866180] Task dump for CPU 16: [90377.866180] Task dump for CPU 0: [90377.866182] ldlm_bl_14 R [90377.866182] ldlm_bl_18 R [90377.866182] running task [90377.866183] 0 28508 2 0x00000088 [90377.866184] running task 0 28512 2 0x00000088 [90377.866184] Call Trace: [90377.866185] Call Trace: [90377.866187] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.866189] [] sched_show_task+0xa8/0x110 [90377.866191] [] ? _raw_spin_lock+0x20/0x30 [90377.866193] [] dump_cpu_task+0x39/0x70 [90377.866213] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.866215] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.866236] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.866237] [] rcu_check_callbacks+0x442/0x730 [90377.866256] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.866258] [] ? tick_sched_do_timer+0x50/0x50 [90377.866278] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.866280] [] update_process_times+0x46/0x80 [90377.866285] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.866287] [] tick_sched_handle+0x30/0x70 [90377.866305] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866306] [] tick_sched_timer+0x39/0x80 [90377.866325] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866326] [] __hrtimer_run_queues+0x10e/0x270 [90377.866332] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.866333] [] hrtimer_interrupt+0xaf/0x1d0 [90377.866353] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.866354] [] local_apic_timer_interrupt+0x3b/0x60 [90377.866377] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.866378] [] smp_apic_timer_interrupt+0x43/0x60 [90377.866400] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.866401] [] apic_timer_interrupt+0x16a/0x170 [90377.866403] [] ? kthread+0xd1/0xe0 [90377.866405] [] ? insert_kthread_work+0x40/0x40 [90377.866406] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.866408] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.866409] [] queued_spin_lock_slowpath+0xb/0xf [90377.866410] [] ? insert_kthread_work+0x40/0x40 [90377.866412] [] _raw_spin_lock+0x20/0x30 [90377.866412] Task dump for CPU 1: [90377.866430] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.866449] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.866450] ldlm_bl_15 R running task 0 28509 2 0x00000088 [90377.866468] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.866469] Call Trace: [90377.866470] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.866491] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.866493] [] ? _raw_spin_lock+0x20/0x30 [90377.866498] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.866515] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.866533] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866552] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.866570] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866591] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.866596] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.866617] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.866635] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.866640] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.866661] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.866679] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866700] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.866718] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866719] [] kthread+0xd1/0xe0 [90377.866725] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.866726] [] ? insert_kthread_work+0x40/0x40 [90377.866746] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.866748] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.866771] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.866772] [] ? insert_kthread_work+0x40/0x40 [90377.866795] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.866796] Task dump for CPU 17: [90377.866797] [] ? kthread+0xd1/0xe0 [90377.866799] [] ? insert_kthread_work+0x40/0x40 [90377.866801] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.866801] ldlm_bl_11 R running task 0 28505 2 0x00000088 [90377.866803] [] ? insert_kthread_work+0x40/0x40 [90377.866803] Call Trace: [90377.866803] Task dump for CPU 2: [90377.866805] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.866807] [] ? _raw_spin_lock+0x20/0x30 [90377.866808] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.866825] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.866826] Call Trace: [90377.866844] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.866865] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.866885] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.866906] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.866926] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.866928] [] ? wake_up_state+0x20/0x20 [90377.866933] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.866954] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.866972] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866973] [] ? kthread+0xd1/0xe0 [90377.866991] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.866993] [] ? insert_kthread_work+0x40/0x40 [90377.866998] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.866999] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867017] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.867019] [] ? insert_kthread_work+0x40/0x40 [90377.867039] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.867040] Task dump for CPU 3: [90377.867061] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.867062] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.867063] [] ? kthread+0xd1/0xe0 [90377.867064] Call Trace: [90377.867065] [] ? insert_kthread_work+0x40/0x40 [90377.867066] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.867068] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867070] [] ? _raw_spin_lock+0x20/0x30 [90377.867071] [] ? insert_kthread_work+0x40/0x40 [90377.867088] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.867089] Task dump for CPU 19: [90377.867110] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.867128] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.867129] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.867150] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.867150] Call Trace: [90377.867155] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.867174] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.867192] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867213] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.867231] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867233] [] ? wake_up_state+0x20/0x20 [90377.867238] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.867258] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.867276] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.867278] [] ? kthread+0xd1/0xe0 [90377.867310] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.867311] [] ? insert_kthread_work+0x40/0x40 [90377.867344] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.867346] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867347] [] ? kthread+0xd1/0xe0 [90377.867349] [] ? insert_kthread_work+0x40/0x40 [90377.867350] [] ? insert_kthread_work+0x40/0x40 [90377.867350] Task dump for CPU 21: [90377.867352] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867354] [] ? insert_kthread_work+0x40/0x40 [90377.867355] Task dump for CPU 4: [90377.867355] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.867356] Call Trace: [90377.867357] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.867376] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.867376] Call Trace: [90377.867397] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.867398] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.867400] [] ? wake_up_state+0x20/0x20 [90377.867401] [] ? _raw_spin_lock+0x20/0x30 [90377.867422] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.867453] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.867454] [] ? kthread+0xd1/0xe0 [90377.867486] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.867487] [] ? insert_kthread_work+0x40/0x40 [90377.867518] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.867520] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867551] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.867553] [] ? insert_kthread_work+0x40/0x40 [90377.867559] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.867591] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867623] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867630] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.867661] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.867695] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.867728] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.867730] [] ? kthread+0xd1/0xe0 [90377.867731] [] ? insert_kthread_work+0x40/0x40 [90377.867733] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.867734] [] ? insert_kthread_work+0x40/0x40 [90377.867735] Task dump for CPU 5: [90377.867736] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.867737] Call Trace: [90377.867738] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.867740] [] ? _raw_spin_lock+0x20/0x30 [90377.867771] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.867802] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.867834] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.867866] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.867873] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.867904] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867936] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.867943] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.867975] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.868008] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.868010] [] ? wake_up_state+0x20/0x20 [90377.868043] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.868044] [] ? kthread+0xd1/0xe0 [90377.868046] [] ? insert_kthread_work+0x40/0x40 [90377.868047] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.868049] [] ? insert_kthread_work+0x40/0x40 [90377.868049] Task dump for CPU 6: [90377.868051] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.868051] Call Trace: [90377.868053] [] sched_show_task+0xa8/0x110 [90377.868055] [] dump_cpu_task+0x39/0x70 [90377.868057] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.868058] [] rcu_check_callbacks+0x442/0x730 [90377.868060] [] ? tick_sched_do_timer+0x50/0x50 [90377.868062] [] update_process_times+0x46/0x80 [90377.868064] [] tick_sched_handle+0x30/0x70 [90377.868065] [] tick_sched_timer+0x39/0x80 [90377.868066] [] __hrtimer_run_queues+0x10e/0x270 [90377.868068] [] hrtimer_interrupt+0xaf/0x1d0 [90377.868069] [] local_apic_timer_interrupt+0x3b/0x60 [90377.868071] [] smp_apic_timer_interrupt+0x43/0x60 [90377.868072] [] apic_timer_interrupt+0x16a/0x170 [90377.868106] [] ? ldlm_process_plain_lock+0x487/0xab0 [ptlrpc] [90377.868138] [] ? ldlm_process_plain_lock+0x4c2/0xab0 [ptlrpc] [90377.868170] [] ? ldlm_errno2error+0x60/0x60 [ptlrpc] [90377.868202] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [90377.868234] [] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [90377.868265] [] ldlm_reprocess_res+0x28/0x30 [ptlrpc] [90377.868273] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.868304] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90377.868336] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90377.868343] [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] [90377.868375] [] ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.868406] [] ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.868440] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.868442] [] ? wake_up_state+0x20/0x20 [90377.868475] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.868476] [] kthread+0xd1/0xe0 [90377.868478] [] ? insert_kthread_work+0x40/0x40 [90377.868480] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.868481] [] ? insert_kthread_work+0x40/0x40 [90377.868482] Task dump for CPU 7: [90377.868483] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.868484] Call Trace: [90377.868515] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.868549] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.868551] [] ? wake_up_state+0x20/0x20 [90377.868584] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.868585] [] ? kthread+0xd1/0xe0 [90377.868587] [] ? insert_kthread_work+0x40/0x40 [90377.868588] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.868590] [] ? insert_kthread_work+0x40/0x40 [90377.868590] Task dump for CPU 8: [90377.868591] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.868592] Call Trace: [90377.868623] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.868657] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.868659] [] ? wake_up_state+0x20/0x20 [90377.868692] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.868693] [] ? kthread+0xd1/0xe0 [90377.868695] [] ? insert_kthread_work+0x40/0x40 [90377.868696] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.868698] [] ? insert_kthread_work+0x40/0x40 [90377.868698] Task dump for CPU 10: [90377.868700] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.868700] Call Trace: [90377.868701] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.868703] [] ? _raw_spin_lock+0x20/0x30 [90377.868734] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.868765] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.868797] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.868829] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.868836] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.868867] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.868899] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.868906] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.868938] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.868971] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.869004] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.869006] [] ? kthread+0xd1/0xe0 [90377.869008] [] ? insert_kthread_work+0x40/0x40 [90377.869009] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.869011] [] ? insert_kthread_work+0x40/0x40 [90377.869011] Task dump for CPU 12: [90377.869012] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.869013] Call Trace: [90377.869045] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.869078] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.869080] [] ? wake_up_state+0x20/0x20 [90377.869113] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.869114] [] ? kthread+0xd1/0xe0 [90377.869116] [] ? insert_kthread_work+0x40/0x40 [90377.869117] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.869119] [] ? insert_kthread_work+0x40/0x40 [90377.869119] Task dump for CPU 13: [90377.869120] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.869121] Call Trace: [90377.869122] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.869124] [] ? _raw_spin_lock+0x20/0x30 [90377.869155] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.869187] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.869218] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.869250] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.869257] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.869288] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869320] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869327] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.869359] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.869392] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.869425] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.869427] [] ? kthread+0xd1/0xe0 [90377.869428] [] ? insert_kthread_work+0x40/0x40 [90377.869430] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.869431] [] ? insert_kthread_work+0x40/0x40 [90377.869432] Task dump for CPU 16: [90377.869433] Task dump for CPU 0: [90377.869434] ldlm_bl_14 R [90377.869434] ldlm_bl_18 R [90377.869435] running task [90377.869435] 0 28508 2 0x00000088 [90377.869436] running task 0 28512 2 0x00000088 [90377.869437] Call Trace: [90377.869437] Call Trace: [90377.869438] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.869440] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.869442] [] ? _raw_spin_lock+0x20/0x30 [90377.869444] [] ? _raw_spin_lock+0x20/0x30 [90377.869475] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.869495] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.869526] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.869544] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.869576] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.869594] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.869625] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.869646] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.869652] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.869658] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.869689] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869707] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869739] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869759] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.869766] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.869771] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.869803] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.869820] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.869853] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.869874] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.869907] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.869930] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.869932] [] ? kthread+0xd1/0xe0 [90377.869933] [] ? kthread+0xd1/0xe0 [90377.869935] [] ? insert_kthread_work+0x40/0x40 [90377.869936] [] ? insert_kthread_work+0x40/0x40 [90377.869938] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.869940] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.869941] [] ? insert_kthread_work+0x40/0x40 [90377.869942] [] ? insert_kthread_work+0x40/0x40 [90377.869943] Task dump for CPU 17: [90377.869944] Task dump for CPU 1: [90377.869944] ldlm_bl_11 R running task [90377.869945] 0 28505 2 0x00000088 [90377.869946] Call Trace: [90377.869946] ldlm_bl_15 R running task 0 28509 2 0x00000088 [90377.869948] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.869948] Call Trace: [90377.869950] [] ? _raw_spin_lock+0x20/0x30 [90377.869951] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.869982] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.869984] [] ? _raw_spin_lock+0x20/0x30 [90377.870015] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.870032] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.870064] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.870082] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.870114] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.870134] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.870141] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.870159] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.870191] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870196] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.870227] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870245] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870252] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.870273] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870304] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870309] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.870342] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870360] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870393] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870414] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870416] [] ? kthread+0xd1/0xe0 [90377.870437] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870438] [] ? insert_kthread_work+0x40/0x40 [90377.870439] [] ? kthread+0xd1/0xe0 [90377.870441] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870442] [] ? insert_kthread_work+0x40/0x40 [90377.870444] [] ? insert_kthread_work+0x40/0x40 [90377.870445] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870446] Task dump for CPU 19: [90377.870447] [] ? insert_kthread_work+0x40/0x40 [90377.870448] Task dump for CPU 2: [90377.870449] ldlm_bl_05 R running task [90377.870449] 0 27290 2 0x00000088 [90377.870450] Call Trace: [90377.870451] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.870482] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870482] Call Trace: [90377.870515] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870534] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870536] [] ? wake_up_state+0x20/0x20 [90377.870556] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870589] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870591] [] ? wake_up_state+0x20/0x20 [90377.870593] [] ? kthread+0xd1/0xe0 [90377.870613] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870614] [] ? insert_kthread_work+0x40/0x40 [90377.870616] [] ? kthread+0xd1/0xe0 [90377.870618] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870619] [] ? insert_kthread_work+0x40/0x40 [90377.870620] [] ? insert_kthread_work+0x40/0x40 [90377.870622] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870622] Task dump for CPU 21: [90377.870624] [] ? insert_kthread_work+0x40/0x40 [90377.870624] Task dump for CPU 3: [90377.870625] ldlm_bl_06 R running task [90377.870626] 0 27291 2 0x00000088 [90377.870626] Call Trace: [90377.870627] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.870658] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870659] Call Trace: [90377.870692] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870693] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.870695] [] ? wake_up_state+0x20/0x20 [90377.870696] [] ? _raw_spin_lock+0x20/0x30 [90377.870729] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870746] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.870748] [] ? kthread+0xd1/0xe0 [90377.870766] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.870768] [] ? insert_kthread_work+0x40/0x40 [90377.870787] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.870788] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870807] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.870809] [] ? insert_kthread_work+0x40/0x40 [90377.870814] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.870832] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870853] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.870859] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.870879] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.870900] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.870921] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.870923] [] ? kthread+0xd1/0xe0 [90377.870924] [] ? insert_kthread_work+0x40/0x40 [90377.870926] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.870927] [] ? insert_kthread_work+0x40/0x40 [90377.870927] Task dump for CPU 4: [90377.870929] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.870929] Call Trace: [90377.870931] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.870932] [] ? _raw_spin_lock+0x20/0x30 [90377.870950] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.870968] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.870989] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.871008] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.871014] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.871032] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871050] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871056] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.871076] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871097] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871119] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871120] [] ? kthread+0xd1/0xe0 [90377.871121] [] ? insert_kthread_work+0x40/0x40 [90377.871123] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871124] [] ? insert_kthread_work+0x40/0x40 [90377.871125] Task dump for CPU 5: [90377.871126] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.871126] Call Trace: [90377.871128] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.871129] [] ? _raw_spin_lock+0x20/0x30 [90377.871147] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.871165] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.871184] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.871203] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.871208] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.871226] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871245] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871251] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.871269] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871290] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871291] [] ? wake_up_state+0x20/0x20 [90377.871314] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871315] [] ? kthread+0xd1/0xe0 [90377.871317] [] ? insert_kthread_work+0x40/0x40 [90377.871318] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871320] [] ? insert_kthread_work+0x40/0x40 [90377.871320] Task dump for CPU 6: [90377.871321] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.871322] Call Trace: [90377.871341] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.871359] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.871380] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871382] [] ? wake_up_state+0x20/0x20 [90377.871403] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871404] [] ? kthread+0xd1/0xe0 [90377.871405] [] ? insert_kthread_work+0x40/0x40 [90377.871407] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871408] [] ? insert_kthread_work+0x40/0x40 [90377.871409] Task dump for CPU 7: [90377.871410] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.871410] Call Trace: [90377.871429] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871450] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871452] [] ? wake_up_state+0x20/0x20 [90377.871473] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871474] [] ? kthread+0xd1/0xe0 [90377.871475] [] ? insert_kthread_work+0x40/0x40 [90377.871477] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871478] [] ? insert_kthread_work+0x40/0x40 [90377.871479] Task dump for CPU 8: [90377.871480] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.871480] Call Trace: [90377.871483] [] sched_show_task+0xa8/0x110 [90377.871484] [] dump_cpu_task+0x39/0x70 [90377.871486] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.871487] [] rcu_check_callbacks+0x442/0x730 [90377.871489] [] ? tick_sched_do_timer+0x50/0x50 [90377.871491] [] update_process_times+0x46/0x80 [90377.871493] [] tick_sched_handle+0x30/0x70 [90377.871494] [] tick_sched_timer+0x39/0x80 [90377.871495] [] __hrtimer_run_queues+0x10e/0x270 [90377.871497] [] hrtimer_interrupt+0xaf/0x1d0 [90377.871498] [] local_apic_timer_interrupt+0x3b/0x60 [90377.871499] [] smp_apic_timer_interrupt+0x43/0x60 [90377.871501] [] apic_timer_interrupt+0x16a/0x170 [90377.871503] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.871505] [] queued_spin_lock_slowpath+0xb/0xf [90377.871506] [] _raw_spin_lock+0x20/0x30 [90377.871524] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.871543] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.871564] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.871583] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.871589] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.871607] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871625] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871631] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.871651] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871673] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871675] [] ? wake_up_state+0x20/0x20 [90377.871696] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871697] [] kthread+0xd1/0xe0 [90377.871699] [] ? insert_kthread_work+0x40/0x40 [90377.871700] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.871702] [] ? insert_kthread_work+0x40/0x40 [90377.871703] Task dump for CPU 10: [90377.871704] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.871704] Call Trace: [90377.871705] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.871707] [] ? _raw_spin_lock+0x20/0x30 [90377.871724] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.871743] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.871764] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.871783] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.871788] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.871806] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871825] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.871830] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.871849] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871870] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871891] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871893] [] ? kthread+0xd1/0xe0 [90377.871894] [] ? insert_kthread_work+0x40/0x40 [90377.871896] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871897] [] ? insert_kthread_work+0x40/0x40 [90377.871898] Task dump for CPU 12: [90377.871899] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.871899] Call Trace: [90377.871918] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.871939] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.871941] [] ? wake_up_state+0x20/0x20 [90377.871962] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.871963] [] ? kthread+0xd1/0xe0 [90377.871965] [] ? insert_kthread_work+0x40/0x40 [90377.871966] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.871968] [] ? insert_kthread_work+0x40/0x40 [90377.871968] Task dump for CPU 13: [90377.871969] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.871969] Call Trace: [90377.871971] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.871972] [] ? _raw_spin_lock+0x20/0x30 [90377.871990] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.872009] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.872030] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.872048] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.872054] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.872072] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872091] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872096] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.872115] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872136] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.872157] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.872158] [] ? kthread+0xd1/0xe0 [90377.872160] [] ? insert_kthread_work+0x40/0x40 [90377.872161] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.872163] [] ? insert_kthread_work+0x40/0x40 [90377.872163] Task dump for CPU 16: [90377.872164] Task dump for CPU 0: [90377.872165] ldlm_bl_14 R [90377.872165] ldlm_bl_18 R [90377.872166] running task [90377.872166] 0 28508 2 0x00000088 [90377.872167] running task 0 28512 2 0x00000088 [90377.872168] Call Trace: [90377.872168] Call Trace: [90377.872170] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.872171] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.872173] [] ? _raw_spin_lock+0x20/0x30 [90377.872175] [] ? _raw_spin_lock+0x20/0x30 [90377.872192] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.872212] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.872230] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.872251] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.872269] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.872290] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.872308] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.872329] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.872334] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.872339] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.872357] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872375] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872393] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872411] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872416] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.872422] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.872440] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872458] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872478] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.872501] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.872522] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.872545] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.872546] [] ? kthread+0xd1/0xe0 [90377.872548] [] ? kthread+0xd1/0xe0 [90377.872549] [] ? insert_kthread_work+0x40/0x40 [90377.872550] [] ? insert_kthread_work+0x40/0x40 [90377.872552] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.872554] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.872555] [] ? insert_kthread_work+0x40/0x40 [90377.872556] [] ? insert_kthread_work+0x40/0x40 [90377.872557] Task dump for CPU 17: [90377.872557] Task dump for CPU 1: [90377.872558] ldlm_bl_11 R [90377.872558] ldlm_bl_15 R [90377.872559] running task [90377.872559] 0 28505 2 0x00000088 [90377.872560] running task 0 28509 2 0x00000088 [90377.872560] Call Trace: [90377.872561] Call Trace: [90377.872562] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.872563] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.872565] [] ? _raw_spin_lock+0x20/0x30 [90377.872567] [] ? _raw_spin_lock+0x20/0x30 [90377.872584] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.872601] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.872621] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.872642] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.872663] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.872681] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.872701] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.872722] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.872727] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.872732] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.872749] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872768] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872787] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872806] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.872811] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.872816] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.872834] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872854] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872875] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.872897] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.872918] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.872942] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.872943] [] ? kthread+0xd1/0xe0 [90377.872945] [] ? kthread+0xd1/0xe0 [90377.872946] [] ? insert_kthread_work+0x40/0x40 [90377.872947] [] ? insert_kthread_work+0x40/0x40 [90377.872949] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.872951] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.872952] [] ? insert_kthread_work+0x40/0x40 [90377.872953] [] ? insert_kthread_work+0x40/0x40 [90377.872954] Task dump for CPU 19: [90377.872954] Task dump for CPU 2: [90377.872955] ldlm_bl_05 R [90377.872955] ldlm_bl_07 R [90377.872956] running task [90377.872956] 0 27290 2 0x00000088 [90377.872957] running task 0 27292 2 0x00000088 [90377.872957] Call Trace: [90377.872958] Call Trace: [90377.872976] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.872997] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873018] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873038] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873040] [] ? wake_up_state+0x20/0x20 [90377.873042] [] ? wake_up_state+0x20/0x20 [90377.873062] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873083] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873085] [] ? kthread+0xd1/0xe0 [90377.873086] [] ? kthread+0xd1/0xe0 [90377.873087] [] ? insert_kthread_work+0x40/0x40 [90377.873089] [] ? insert_kthread_work+0x40/0x40 [90377.873090] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873092] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873093] [] ? insert_kthread_work+0x40/0x40 [90377.873094] [] ? insert_kthread_work+0x40/0x40 [90377.873095] Task dump for CPU 21: [90377.873095] Task dump for CPU 3: [90377.873096] ldlm_bl_06 R [90377.873096] ldlm_bl_17 R [90377.873097] running task [90377.873097] 0 27291 2 0x00000088 [90377.873098] running task 0 28511 2 0x00000088 [90377.873099] Call Trace: [90377.873099] Call Trace: [90377.873100] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.873121] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873123] [] ? _raw_spin_lock+0x20/0x30 [90377.873143] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873160] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.873162] [] ? wake_up_state+0x20/0x20 [90377.873183] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.873203] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873222] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.873223] [] ? kthread+0xd1/0xe0 [90377.873244] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.873245] [] ? insert_kthread_work+0x40/0x40 [90377.873250] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.873252] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873270] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873271] [] ? insert_kthread_work+0x40/0x40 [90377.873289] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873295] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.873313] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873335] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873356] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873357] [] ? kthread+0xd1/0xe0 [90377.873359] [] ? insert_kthread_work+0x40/0x40 [90377.873360] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873362] [] ? insert_kthread_work+0x40/0x40 [90377.873362] Task dump for CPU 4: [90377.873363] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.873364] Call Trace: [90377.873365] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.873367] [] ? _raw_spin_lock+0x20/0x30 [90377.873384] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.873403] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.873421] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.873443] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.873448] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.873466] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873485] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873490] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.873511] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873532] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873554] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873555] [] ? kthread+0xd1/0xe0 [90377.873557] [] ? insert_kthread_work+0x40/0x40 [90377.873558] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873559] [] ? insert_kthread_work+0x40/0x40 [90377.873560] Task dump for CPU 5: [90377.873561] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.873561] Call Trace: [90377.873563] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.873564] [] ? _raw_spin_lock+0x20/0x30 [90377.873582] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.873603] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.873622] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.873643] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.873648] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.873666] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873685] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.873690] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.873710] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873733] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873734] [] ? wake_up_state+0x20/0x20 [90377.873757] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873759] [] ? kthread+0xd1/0xe0 [90377.873760] [] ? insert_kthread_work+0x40/0x40 [90377.873762] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873763] [] ? insert_kthread_work+0x40/0x40 [90377.873764] Task dump for CPU 6: [90377.873765] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.873765] Call Trace: [90377.873784] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.873805] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.873828] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873830] [] ? wake_up_state+0x20/0x20 [90377.873852] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873854] [] ? kthread+0xd1/0xe0 [90377.873855] [] ? insert_kthread_work+0x40/0x40 [90377.873857] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873858] [] ? insert_kthread_work+0x40/0x40 [90377.873859] Task dump for CPU 7: [90377.873860] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.873860] Call Trace: [90377.873881] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873903] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873905] [] ? wake_up_state+0x20/0x20 [90377.873926] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.873927] [] ? kthread+0xd1/0xe0 [90377.873929] [] ? insert_kthread_work+0x40/0x40 [90377.873930] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.873932] [] ? insert_kthread_work+0x40/0x40 [90377.873932] Task dump for CPU 8: [90377.873934] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.873934] Call Trace: [90377.873955] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.873977] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.873979] [] ? wake_up_state+0x20/0x20 [90377.874002] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874003] [] ? kthread+0xd1/0xe0 [90377.874004] [] ? insert_kthread_work+0x40/0x40 [90377.874006] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.874007] [] ? insert_kthread_work+0x40/0x40 [90377.874008] Task dump for CPU 10: [90377.874009] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.874009] Call Trace: [90377.874011] [] sched_show_task+0xa8/0x110 [90377.874013] [] dump_cpu_task+0x39/0x70 [90377.874015] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.874016] [] rcu_check_callbacks+0x442/0x730 [90377.874018] [] ? tick_sched_do_timer+0x50/0x50 [90377.874020] [] update_process_times+0x46/0x80 [90377.874022] [] tick_sched_handle+0x30/0x70 [90377.874023] [] tick_sched_timer+0x39/0x80 [90377.874024] [] __hrtimer_run_queues+0x10e/0x270 [90377.874026] [] hrtimer_interrupt+0xaf/0x1d0 [90377.874027] [] local_apic_timer_interrupt+0x3b/0x60 [90377.874028] [] smp_apic_timer_interrupt+0x43/0x60 [90377.874030] [] apic_timer_interrupt+0x16a/0x170 [90377.874032] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.874033] [] queued_spin_lock_slowpath+0xb/0xf [90377.874035] [] _raw_spin_lock+0x20/0x30 [90377.874053] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874073] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.874092] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.874113] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.874119] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.874137] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874156] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874161] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.874181] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.874204] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.874227] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874229] [] kthread+0xd1/0xe0 [90377.874230] [] ? insert_kthread_work+0x40/0x40 [90377.874232] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.874233] [] ? insert_kthread_work+0x40/0x40 [90377.874234] Task dump for CPU 12: [90377.874235] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.874235] Call Trace: [90377.874256] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.874279] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.874281] [] ? wake_up_state+0x20/0x20 [90377.874301] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874303] [] ? kthread+0xd1/0xe0 [90377.874304] [] ? insert_kthread_work+0x40/0x40 [90377.874306] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.874307] [] ? insert_kthread_work+0x40/0x40 [90377.874308] Task dump for CPU 13: [90377.874309] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.874309] Call Trace: [90377.874310] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.874312] [] ? _raw_spin_lock+0x20/0x30 [90377.874330] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874351] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.874369] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.874390] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.874395] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.874414] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874432] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874438] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.874458] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.874481] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.874504] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874505] [] ? kthread+0xd1/0xe0 [90377.874507] [] ? insert_kthread_work+0x40/0x40 [90377.874508] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.874509] [] ? insert_kthread_work+0x40/0x40 [90377.874510] Task dump for CPU 16: [90377.874511] Task dump for CPU 0: [90377.874512] ldlm_bl_14 R [90377.874512] ldlm_bl_18 R [90377.874513] running task [90377.874513] 0 28508 2 0x00000088 [90377.874514] running task 0 28512 2 0x00000088 [90377.874515] Call Trace: [90377.874515] Call Trace: [90377.874516] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.874518] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.874520] [] ? _raw_spin_lock+0x20/0x30 [90377.874522] [] ? _raw_spin_lock+0x20/0x30 [90377.874539] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874559] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874579] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.874599] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.874618] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.874636] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.874656] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.874677] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.874682] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.874687] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.874705] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874723] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874742] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874760] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.874765] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.874770] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.874790] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.874810] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.874833] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.874856] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.874878] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874899] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.874901] [] ? kthread+0xd1/0xe0 [90377.874902] [] ? kthread+0xd1/0xe0 [90377.874904] [] ? insert_kthread_work+0x40/0x40 [90377.874905] [] ? insert_kthread_work+0x40/0x40 [90377.874907] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.874908] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.874910] [] ? insert_kthread_work+0x40/0x40 [90377.874911] [] ? insert_kthread_work+0x40/0x40 [90377.874911] Task dump for CPU 17: [90377.874912] Task dump for CPU 1: [90377.874913] ldlm_bl_11 R [90377.874913] ldlm_bl_15 R [90377.874914] running task [90377.874914] 0 28505 2 0x00000088 [90377.874915] running task 0 28509 2 0x00000088 [90377.874916] Call Trace: [90377.874916] Call Trace: [90377.874918] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.874919] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.874921] [] ? _raw_spin_lock+0x20/0x30 [90377.874922] [] ? _raw_spin_lock+0x20/0x30 [90377.874940] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874958] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.874978] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.874996] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.875015] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.875033] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.875054] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.875075] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.875080] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.875085] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.875103] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875121] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875140] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875158] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875163] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.875168] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.875188] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875206] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875229] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875252] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875274] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875295] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875296] [] ? kthread+0xd1/0xe0 [90377.875298] [] ? kthread+0xd1/0xe0 [90377.875299] [] ? insert_kthread_work+0x40/0x40 [90377.875301] [] ? insert_kthread_work+0x40/0x40 [90377.875302] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.875304] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.875305] [] ? insert_kthread_work+0x40/0x40 [90377.875307] [] ? insert_kthread_work+0x40/0x40 [90377.875307] Task dump for CPU 19: [90377.875307] Task dump for CPU 2: [90377.875308] ldlm_bl_05 R [90377.875309] ldlm_bl_07 R [90377.875309] running task [90377.875310] 0 27290 2 0x00000088 [90377.875310] running task 0 27292 2 0x00000088 [90377.875311] Call Trace: [90377.875311] Call Trace: [90377.875332] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875354] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875356] [] sched_show_task+0xa8/0x110 [90377.875358] [] ? wake_up_state+0x20/0x20 [90377.875359] [] dump_cpu_task+0x39/0x70 [90377.875382] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875384] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.875385] [] ? kthread+0xd1/0xe0 [90377.875387] [] rcu_check_callbacks+0x442/0x730 [90377.875388] [] ? insert_kthread_work+0x40/0x40 [90377.875390] [] ? tick_sched_do_timer+0x50/0x50 [90377.875391] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.875393] [] update_process_times+0x46/0x80 [90377.875394] [] ? insert_kthread_work+0x40/0x40 [90377.875396] [] tick_sched_handle+0x30/0x70 [90377.875396] Task dump for CPU 21: [90377.875397] [] tick_sched_timer+0x39/0x80 [90377.875399] [] __hrtimer_run_queues+0x10e/0x270 [90377.875400] [] hrtimer_interrupt+0xaf/0x1d0 [90377.875401] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.875402] [] local_apic_timer_interrupt+0x3b/0x60 [90377.875403] Call Trace: [90377.875404] [] smp_apic_timer_interrupt+0x43/0x60 [90377.875423] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875424] [] apic_timer_interrupt+0x16a/0x170 [90377.875445] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875447] [] ? wake_up_state+0x20/0x20 [90377.875448] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.875471] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875472] [] queued_spin_lock_slowpath+0xb/0xf [90377.875473] [] ? kthread+0xd1/0xe0 [90377.875475] [] _raw_spin_lock+0x20/0x30 [90377.875476] [] ? insert_kthread_work+0x40/0x40 [90377.875494] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.875495] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.875516] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.875517] [] ? insert_kthread_work+0x40/0x40 [90377.875536] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.875557] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.875563] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.875581] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875600] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875605] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.875626] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875649] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875651] [] ? wake_up_state+0x20/0x20 [90377.875672] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875674] [] kthread+0xd1/0xe0 [90377.875676] [] ? insert_kthread_work+0x40/0x40 [90377.875677] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.875678] [] ? insert_kthread_work+0x40/0x40 [90377.875679] Task dump for CPU 3: [90377.875680] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.875681] Call Trace: [90377.875682] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.875683] [] ? _raw_spin_lock+0x20/0x30 [90377.875701] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.875722] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.875740] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.875761] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.875767] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.875785] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875803] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.875809] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.875829] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.875852] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.875875] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.875876] [] ? kthread+0xd1/0xe0 [90377.875878] [] ? insert_kthread_work+0x40/0x40 [90377.875879] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.875880] [] ? insert_kthread_work+0x40/0x40 [90377.875881] Task dump for CPU 4: [90377.875882] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.875882] Call Trace: [90377.875884] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.875885] [] ? _raw_spin_lock+0x20/0x30 [90377.875903] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.875922] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.875943] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.875964] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.875969] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.875987] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876006] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876011] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.876032] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876055] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876078] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876080] [] ? kthread+0xd1/0xe0 [90377.876081] [] ? insert_kthread_work+0x40/0x40 [90377.876083] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876084] [] ? insert_kthread_work+0x40/0x40 [90377.876084] Task dump for CPU 5: [90377.876086] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.876086] Call Trace: [90377.876087] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.876089] [] ? _raw_spin_lock+0x20/0x30 [90377.876107] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.876126] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.876145] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.876166] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.876171] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.876189] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876208] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876214] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.876234] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876257] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876259] [] ? wake_up_state+0x20/0x20 [90377.876281] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876283] [] ? kthread+0xd1/0xe0 [90377.876284] [] ? insert_kthread_work+0x40/0x40 [90377.876286] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876287] [] ? insert_kthread_work+0x40/0x40 [90377.876288] Task dump for CPU 6: [90377.876289] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.876289] Call Trace: [90377.876309] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.876330] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.876352] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876354] [] ? wake_up_state+0x20/0x20 [90377.876375] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876376] [] ? kthread+0xd1/0xe0 [90377.876378] [] ? insert_kthread_work+0x40/0x40 [90377.876379] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876381] [] ? insert_kthread_work+0x40/0x40 [90377.876381] Task dump for CPU 7: [90377.876382] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.876382] Call Trace: [90377.876403] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876426] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876427] [] ? wake_up_state+0x20/0x20 [90377.876450] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876452] [] ? kthread+0xd1/0xe0 [90377.876453] [] ? insert_kthread_work+0x40/0x40 [90377.876455] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876456] [] ? insert_kthread_work+0x40/0x40 [90377.876456] Task dump for CPU 8: [90377.876457] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.876458] Call Trace: [90377.876477] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876500] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876501] [] ? wake_up_state+0x20/0x20 [90377.876524] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876525] [] ? kthread+0xd1/0xe0 [90377.876526] [] ? insert_kthread_work+0x40/0x40 [90377.876528] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876529] [] ? insert_kthread_work+0x40/0x40 [90377.876530] Task dump for CPU 10: [90377.876531] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.876531] Call Trace: [90377.876533] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.876534] [] ? _raw_spin_lock+0x20/0x30 [90377.876552] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.876573] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.876591] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.876612] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.876618] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.876636] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876655] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876660] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.876680] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876703] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876726] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876727] [] ? kthread+0xd1/0xe0 [90377.876728] [] ? insert_kthread_work+0x40/0x40 [90377.876730] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876731] [] ? insert_kthread_work+0x40/0x40 [90377.876732] Task dump for CPU 12: [90377.876733] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.876733] Call Trace: [90377.876754] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876777] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876778] [] ? wake_up_state+0x20/0x20 [90377.876801] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.876802] [] ? kthread+0xd1/0xe0 [90377.876804] [] ? insert_kthread_work+0x40/0x40 [90377.876805] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.876807] [] ? insert_kthread_work+0x40/0x40 [90377.876807] Task dump for CPU 13: [90377.876808] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.876808] Call Trace: [90377.876810] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.876811] [] ? _raw_spin_lock+0x20/0x30 [90377.876829] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.876849] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.876868] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.876889] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.876895] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.876913] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876931] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.876937] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.876957] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.876978] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.876999] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877000] [] ? kthread+0xd1/0xe0 [90377.877002] [] ? insert_kthread_work+0x40/0x40 [90377.877003] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877005] [] ? insert_kthread_work+0x40/0x40 [90377.877005] Task dump for CPU 16: [90377.877006] Task dump for CPU 0: [90377.877008] ldlm_bl_14 R [90377.877009] ldlm_bl_18 R [90377.877010] running task [90377.877011] 0 28508 2 0x00000088 [90377.877013] running task 0 28512 2 0x00000088 [90377.877013] Call Trace: [90377.877015] Call Trace: [90377.877016] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.877019] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.877021] [] ? _raw_spin_lock+0x20/0x30 [90377.877023] [] ? _raw_spin_lock+0x20/0x30 [90377.877041] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.877060] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.877079] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.877099] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.877117] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.877137] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.877158] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.877178] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.877183] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.877189] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.877207] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877227] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877246] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877265] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877271] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.877276] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.877297] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877317] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877340] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877362] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877385] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877407] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877409] [] ? kthread+0xd1/0xe0 [90377.877411] [] ? kthread+0xd1/0xe0 [90377.877412] [] ? insert_kthread_work+0x40/0x40 [90377.877414] [] ? insert_kthread_work+0x40/0x40 [90377.877416] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877418] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877420] [] ? insert_kthread_work+0x40/0x40 [90377.877422] [] ? insert_kthread_work+0x40/0x40 [90377.877423] Task dump for CPU 17: [90377.877424] Task dump for CPU 1: [90377.877426] ldlm_bl_11 R [90377.877426] ldlm_bl_15 R [90377.877427] running task [90377.877429] 0 28505 2 0x00000088 [90377.877430] running task 0 28509 2 0x00000088 [90377.877431] Call Trace: [90377.877432] Call Trace: [90377.877433] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.877435] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.877437] [] ? _raw_spin_lock+0x20/0x30 [90377.877439] [] ? _raw_spin_lock+0x20/0x30 [90377.877457] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.877476] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.877497] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.877516] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.877535] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.877555] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.877576] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.877595] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.877600] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.877606] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.877624] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877643] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877662] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877682] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.877688] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.877693] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.877714] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877733] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877754] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877776] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877797] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877820] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877822] [] ? kthread+0xd1/0xe0 [90377.877824] [] ? kthread+0xd1/0xe0 [90377.877825] [] ? insert_kthread_work+0x40/0x40 [90377.877827] [] ? insert_kthread_work+0x40/0x40 [90377.877829] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877831] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877833] [] ? insert_kthread_work+0x40/0x40 [90377.877835] [] ? insert_kthread_work+0x40/0x40 [90377.877836] Task dump for CPU 19: [90377.877837] Task dump for CPU 2: [90377.877838] ldlm_bl_05 R [90377.877839] ldlm_bl_07 R [90377.877840] running task [90377.877841] 0 27290 2 0x00000088 [90377.877843] running task 0 27292 2 0x00000088 [90377.877843] Call Trace: [90377.877844] Call Trace: [90377.877865] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877885] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.877908] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877930] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.877932] [] ? wake_up_state+0x20/0x20 [90377.877934] [] ? wake_up_state+0x20/0x20 [90377.877955] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877977] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.877979] [] ? kthread+0xd1/0xe0 [90377.877981] [] ? kthread+0xd1/0xe0 [90377.877983] [] ? insert_kthread_work+0x40/0x40 [90377.877985] [] ? insert_kthread_work+0x40/0x40 [90377.877987] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877989] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.877991] [] ? insert_kthread_work+0x40/0x40 [90377.877992] [] ? insert_kthread_work+0x40/0x40 [90377.877993] Task dump for CPU 21: [90377.877994] Task dump for CPU 3: [90377.877996] ldlm_bl_06 R [90377.877997] ldlm_bl_17 R [90377.877998] running task [90377.877999] 0 27291 2 0x00000088 [90377.878000] running task 0 28511 2 0x00000088 [90377.878001] Call Trace: [90377.878002] Call Trace: [90377.878023] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878025] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.878046] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878048] [] ? _raw_spin_lock+0x20/0x30 [90377.878050] [] ? wake_up_state+0x20/0x20 [90377.878068] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.878091] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878110] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.878112] [] ? kthread+0xd1/0xe0 [90377.878132] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.878133] [] ? insert_kthread_work+0x40/0x40 [90377.878154] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.878156] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878161] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.878163] [] ? insert_kthread_work+0x40/0x40 [90377.878183] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878202] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878208] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.878228] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878249] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878271] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878272] [] ? kthread+0xd1/0xe0 [90377.878274] [] ? insert_kthread_work+0x40/0x40 [90377.878275] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878277] [] ? insert_kthread_work+0x40/0x40 [90377.878277] Task dump for CPU 4: [90377.878279] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.878279] Call Trace: [90377.878280] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.878282] [] ? _raw_spin_lock+0x20/0x30 [90377.878300] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.878320] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.878339] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.878358] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.878364] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.878384] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878403] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878409] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.878427] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878449] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878471] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878473] [] ? kthread+0xd1/0xe0 [90377.878474] [] ? insert_kthread_work+0x40/0x40 [90377.878476] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878477] [] ? insert_kthread_work+0x40/0x40 [90377.878478] Task dump for CPU 5: [90377.878479] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.878479] Call Trace: [90377.878481] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.878482] [] ? _raw_spin_lock+0x20/0x30 [90377.878501] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.878520] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.878539] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.878559] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.878565] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.878583] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878602] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.878608] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.878627] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878649] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878650] [] ? wake_up_state+0x20/0x20 [90377.878672] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878673] [] ? kthread+0xd1/0xe0 [90377.878675] [] ? insert_kthread_work+0x40/0x40 [90377.878676] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878678] [] ? insert_kthread_work+0x40/0x40 [90377.878678] Task dump for CPU 6: [90377.878680] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.878680] Call Trace: [90377.878699] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.878718] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.878741] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878742] [] ? wake_up_state+0x20/0x20 [90377.878764] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878765] [] ? kthread+0xd1/0xe0 [90377.878767] [] ? insert_kthread_work+0x40/0x40 [90377.878768] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878770] [] ? insert_kthread_work+0x40/0x40 [90377.878770] Task dump for CPU 7: [90377.878771] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.878772] Call Trace: [90377.878791] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878813] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878815] [] ? wake_up_state+0x20/0x20 [90377.878836] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878837] [] ? kthread+0xd1/0xe0 [90377.878839] [] ? insert_kthread_work+0x40/0x40 [90377.878840] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878842] [] ? insert_kthread_work+0x40/0x40 [90377.878842] Task dump for CPU 8: [90377.878844] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.878844] Call Trace: [90377.878864] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.878886] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.878888] [] ? wake_up_state+0x20/0x20 [90377.878909] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.878910] [] ? kthread+0xd1/0xe0 [90377.878911] [] ? insert_kthread_work+0x40/0x40 [90377.878913] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.878914] [] ? insert_kthread_work+0x40/0x40 [90377.878915] Task dump for CPU 10: [90377.878916] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.878916] Call Trace: [90377.878918] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.878920] [] ? _raw_spin_lock+0x20/0x30 [90377.878938] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.878957] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.878976] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.878996] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.879001] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.879020] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879039] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879045] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.879065] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879087] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879108] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.879110] [] ? kthread+0xd1/0xe0 [90377.879111] [] ? insert_kthread_work+0x40/0x40 [90377.879113] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.879114] [] ? insert_kthread_work+0x40/0x40 [90377.879115] Task dump for CPU 12: [90377.879116] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.879116] Call Trace: [90377.879137] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879158] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879160] [] ? wake_up_state+0x20/0x20 [90377.879181] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.879183] [] ? kthread+0xd1/0xe0 [90377.879184] [] ? insert_kthread_work+0x40/0x40 [90377.879186] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.879187] [] ? insert_kthread_work+0x40/0x40 [90377.879188] Task dump for CPU 13: [90377.879189] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.879189] Call Trace: [90377.879192] [] sched_show_task+0xa8/0x110 [90377.879193] [] dump_cpu_task+0x39/0x70 [90377.879195] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.879196] [] rcu_check_callbacks+0x442/0x730 [90377.879198] [] ? tick_sched_do_timer+0x50/0x50 [90377.879200] [] update_process_times+0x46/0x80 [90377.879202] [] tick_sched_handle+0x30/0x70 [90377.879203] [] tick_sched_timer+0x39/0x80 [90377.879204] [] __hrtimer_run_queues+0x10e/0x270 [90377.879206] [] hrtimer_interrupt+0xaf/0x1d0 [90377.879207] [] local_apic_timer_interrupt+0x3b/0x60 [90377.879209] [] smp_apic_timer_interrupt+0x43/0x60 [90377.879210] [] apic_timer_interrupt+0x16a/0x170 [90377.879212] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.879214] [] queued_spin_lock_slowpath+0xb/0xf [90377.879216] [] _raw_spin_lock+0x20/0x30 [90377.879234] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.879254] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.879273] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.879293] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.879298] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.879317] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879337] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879342] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.879361] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879383] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879405] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.879407] [] kthread+0xd1/0xe0 [90377.879409] [] ? insert_kthread_work+0x40/0x40 [90377.879410] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.879412] [] ? insert_kthread_work+0x40/0x40 [90377.879413] Task dump for CPU 16: [90377.879413] Task dump for CPU 0: [90377.879413] ldlm_bl_14 R [90377.879414] ldlm_bl_18 R [90377.879414] running task [90377.879414] 0 28508 2 0x00000088 [90377.879415] running task 0 28512 2 0x00000088 [90377.879415] Call Trace: [90377.879415] Call Trace: [90377.879417] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.879418] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.879419] [] ? _raw_spin_lock+0x20/0x30 [90377.879421] [] ? _raw_spin_lock+0x20/0x30 [90377.879440] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.879458] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.879477] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.879496] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.879516] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.879536] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.879555] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.879574] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.879579] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.879584] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.879602] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879621] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879641] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879660] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879665] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.879669] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.879688] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879707] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879729] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879752] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879774] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.879795] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.879796] [] ? kthread+0xd1/0xe0 [90377.879798] [] ? kthread+0xd1/0xe0 [90377.879799] [] ? insert_kthread_work+0x40/0x40 [90377.879800] [] ? insert_kthread_work+0x40/0x40 [90377.879802] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.879803] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.879804] [] ? insert_kthread_work+0x40/0x40 [90377.879805] [] ? insert_kthread_work+0x40/0x40 [90377.879806] Task dump for CPU 17: [90377.879806] Task dump for CPU 1: [90377.879807] ldlm_bl_11 R [90377.879807] ldlm_bl_15 R [90377.879808] running task [90377.879808] 0 28505 2 0x00000088 [90377.879809] running task 0 28509 2 0x00000088 [90377.879809] Call Trace: [90377.879809] Call Trace: [90377.879810] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.879812] [] ? _raw_spin_lock+0x20/0x30 [90377.879814] [] sched_show_task+0xa8/0x110 [90377.879832] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.879834] [] dump_cpu_task+0x39/0x70 [90377.879853] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.879855] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.879874] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.879876] [] rcu_check_callbacks+0x442/0x730 [90377.879895] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.879897] [] ? tick_sched_do_timer+0x50/0x50 [90377.879902] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.879903] [] update_process_times+0x46/0x80 [90377.879922] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879924] [] tick_sched_handle+0x30/0x70 [90377.879943] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.879945] [] tick_sched_timer+0x39/0x80 [90377.879950] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.879951] [] __hrtimer_run_queues+0x10e/0x270 [90377.879969] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.879971] [] hrtimer_interrupt+0xaf/0x1d0 [90377.879994] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.879995] [] local_apic_timer_interrupt+0x3b/0x60 [90377.880017] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880019] [] smp_apic_timer_interrupt+0x43/0x60 [90377.880020] [] ? kthread+0xd1/0xe0 [90377.880022] [] apic_timer_interrupt+0x16a/0x170 [90377.880023] [] ? insert_kthread_work+0x40/0x40 [90377.880025] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880026] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.880027] [] ? insert_kthread_work+0x40/0x40 [90377.880029] [] queued_spin_lock_slowpath+0xb/0xf [90377.880029] Task dump for CPU 19: [90377.880030] [] _raw_spin_lock+0x20/0x30 [90377.880050] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.880050] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.880069] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.880069] Call Trace: [90377.880089] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.880108] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880127] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.880148] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880154] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.880155] [] ? wake_up_state+0x20/0x20 [90377.880173] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880194] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880213] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880215] [] ? kthread+0xd1/0xe0 [90377.880220] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.880221] [] ? insert_kthread_work+0x40/0x40 [90377.880239] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880241] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880263] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880264] [] ? insert_kthread_work+0x40/0x40 [90377.880285] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880286] Task dump for CPU 21: [90377.880287] [] kthread+0xd1/0xe0 [90377.880289] [] ? insert_kthread_work+0x40/0x40 [90377.880289] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.880291] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.880291] Call Trace: [90377.880293] [] ? insert_kthread_work+0x40/0x40 [90377.880312] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880312] Task dump for CPU 2: [90377.880334] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880336] [] ? wake_up_state+0x20/0x20 [90377.880337] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.880358] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880359] Call Trace: [90377.880360] [] ? kthread+0xd1/0xe0 [90377.880379] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880381] [] ? insert_kthread_work+0x40/0x40 [90377.880402] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880403] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880405] [] ? wake_up_state+0x20/0x20 [90377.880406] [] ? insert_kthread_work+0x40/0x40 [90377.880426] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880427] [] ? kthread+0xd1/0xe0 [90377.880429] [] ? insert_kthread_work+0x40/0x40 [90377.880430] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880432] [] ? insert_kthread_work+0x40/0x40 [90377.880432] Task dump for CPU 3: [90377.880433] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.880434] Call Trace: [90377.880435] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.880437] [] ? _raw_spin_lock+0x20/0x30 [90377.880454] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.880473] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.880493] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.880513] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.880518] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.880536] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880555] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880560] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.880578] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880601] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880622] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880624] [] ? kthread+0xd1/0xe0 [90377.880625] [] ? insert_kthread_work+0x40/0x40 [90377.880627] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880628] [] ? insert_kthread_work+0x40/0x40 [90377.880629] Task dump for CPU 4: [90377.880630] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.880630] Call Trace: [90377.880632] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.880633] [] ? _raw_spin_lock+0x20/0x30 [90377.880651] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.880669] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.880690] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.880711] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.880716] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.880734] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880753] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880758] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.880778] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880801] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.880824] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.880825] [] ? kthread+0xd1/0xe0 [90377.880827] [] ? insert_kthread_work+0x40/0x40 [90377.880828] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.880829] [] ? insert_kthread_work+0x40/0x40 [90377.880830] Task dump for CPU 5: [90377.880831] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.880831] Call Trace: [90377.880833] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.880834] [] ? _raw_spin_lock+0x20/0x30 [90377.880852] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.880870] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.880891] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.880912] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.880917] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.880935] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880954] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.880959] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.880977] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.880999] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881000] [] ? wake_up_state+0x20/0x20 [90377.881021] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881023] [] ? kthread+0xd1/0xe0 [90377.881024] [] ? insert_kthread_work+0x40/0x40 [90377.881026] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881027] [] ? insert_kthread_work+0x40/0x40 [90377.881028] Task dump for CPU 6: [90377.881029] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.881029] Call Trace: [90377.881048] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.881069] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.881090] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881091] [] ? wake_up_state+0x20/0x20 [90377.881112] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881114] [] ? kthread+0xd1/0xe0 [90377.881115] [] ? insert_kthread_work+0x40/0x40 [90377.881117] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881118] [] ? insert_kthread_work+0x40/0x40 [90377.881118] Task dump for CPU 7: [90377.881119] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.881120] Call Trace: [90377.881139] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.881161] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881163] [] ? wake_up_state+0x20/0x20 [90377.881195] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881197] [] ? kthread+0xd1/0xe0 [90377.881199] [] ? insert_kthread_work+0x40/0x40 [90377.881200] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881202] [] ? insert_kthread_work+0x40/0x40 [90377.881202] Task dump for CPU 8: [90377.881204] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.881204] Call Trace: [90377.881235] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.881269] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881271] [] ? wake_up_state+0x20/0x20 [90377.881304] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881305] [] ? kthread+0xd1/0xe0 [90377.881307] [] ? insert_kthread_work+0x40/0x40 [90377.881309] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881310] [] ? insert_kthread_work+0x40/0x40 [90377.881311] Task dump for CPU 10: [90377.881312] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.881312] Call Trace: [90377.881314] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.881315] [] ? _raw_spin_lock+0x20/0x30 [90377.881346] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.881378] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.881410] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.881441] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.881449] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.881480] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.881512] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.881519] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.881550] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.881584] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881617] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881619] [] ? kthread+0xd1/0xe0 [90377.881620] [] ? insert_kthread_work+0x40/0x40 [90377.881622] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881623] [] ? insert_kthread_work+0x40/0x40 [90377.881624] Task dump for CPU 12: [90377.881625] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.881625] Call Trace: [90377.881657] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.881690] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.881692] [] ? wake_up_state+0x20/0x20 [90377.881725] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.881727] [] ? kthread+0xd1/0xe0 [90377.881728] [] ? insert_kthread_work+0x40/0x40 [90377.881730] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.881731] [] ? insert_kthread_work+0x40/0x40 [90377.881732] Task dump for CPU 13: [90377.881733] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.881733] Call Trace: [90377.881735] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.881736] [] ? _raw_spin_lock+0x20/0x30 [90377.881767] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.881799] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.881831] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.881862] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.881869] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.881901] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.881933] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.881940] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.881971] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.882005] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.882038] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.882039] [] ? kthread+0xd1/0xe0 [90377.882041] [] ? insert_kthread_work+0x40/0x40 [90377.882043] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.882044] [] ? insert_kthread_work+0x40/0x40 [90377.882045] Task dump for CPU 16: [90377.882045] Task dump for CPU 0: [90377.882046] ldlm_bl_14 R [90377.882047] ldlm_bl_18 R [90377.882047] running task [90377.882048] 0 28508 2 0x00000088 [90377.882049] running task 0 28512 2 0x00000088 [90377.882049] Call Trace: [90377.882050] Call Trace: [90377.882051] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.882053] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.882055] [] ? _raw_spin_lock+0x20/0x30 [90377.882056] [] ? _raw_spin_lock+0x20/0x30 [90377.882087] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.882105] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.882137] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.882156] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.882187] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.882206] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.882237] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.882257] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.882263] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.882269] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.882300] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882320] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882351] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882371] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882377] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.882382] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.882414] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.882432] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.882465] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.882487] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.882519] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.882541] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.882543] [] ? kthread+0xd1/0xe0 [90377.882544] [] ? kthread+0xd1/0xe0 [90377.882546] [] ? insert_kthread_work+0x40/0x40 [90377.882547] [] ? insert_kthread_work+0x40/0x40 [90377.882549] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.882550] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.882552] [] ? insert_kthread_work+0x40/0x40 [90377.882553] [] ? insert_kthread_work+0x40/0x40 [90377.882554] Task dump for CPU 17: [90377.882554] Task dump for CPU 1: [90377.882555] ldlm_bl_11 R [90377.882556] ldlm_bl_15 R [90377.882556] running task [90377.882557] 0 28505 2 0x00000088 [90377.882557] running task 0 28509 2 0x00000088 [90377.882558] Call Trace: [90377.882558] Call Trace: [90377.882560] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.882561] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.882563] [] ? _raw_spin_lock+0x20/0x30 [90377.882564] [] ? _raw_spin_lock+0x20/0x30 [90377.882595] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.882613] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.882644] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.882663] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.882695] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.882714] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.882745] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.882764] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.882771] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.882776] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.882808] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882827] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882858] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882877] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.882884] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.882889] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.882920] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.882939] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.882972] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.882994] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883027] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883048] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883049] [] ? kthread+0xd1/0xe0 [90377.883051] [] ? kthread+0xd1/0xe0 [90377.883052] [] ? insert_kthread_work+0x40/0x40 [90377.883054] [] ? insert_kthread_work+0x40/0x40 [90377.883055] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883057] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883058] [] ? insert_kthread_work+0x40/0x40 [90377.883060] [] ? insert_kthread_work+0x40/0x40 [90377.883060] Task dump for CPU 19: [90377.883060] Task dump for CPU 2: [90377.883061] ldlm_bl_05 R [90377.883062] ldlm_bl_07 R [90377.883062] running task [90377.883063] 0 27290 2 0x00000088 [90377.883063] running task 0 27292 2 0x00000088 [90377.883064] Call Trace: [90377.883064] Call Trace: [90377.883096] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883115] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883148] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883169] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883171] [] ? wake_up_state+0x20/0x20 [90377.883173] [] ? wake_up_state+0x20/0x20 [90377.883205] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883227] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883228] [] ? kthread+0xd1/0xe0 [90377.883230] [] ? kthread+0xd1/0xe0 [90377.883231] [] ? insert_kthread_work+0x40/0x40 [90377.883232] [] ? insert_kthread_work+0x40/0x40 [90377.883234] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883236] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883237] [] ? insert_kthread_work+0x40/0x40 [90377.883238] [] ? insert_kthread_work+0x40/0x40 [90377.883239] Task dump for CPU 21: [90377.883239] Task dump for CPU 3: [90377.883240] ldlm_bl_06 R [90377.883240] ldlm_bl_17 R [90377.883241] running task [90377.883242] 0 27291 2 0x00000088 [90377.883242] running task 0 28511 2 0x00000088 [90377.883243] Call Trace: [90377.883243] Call Trace: [90377.883274] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883276] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.883309] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883310] [] ? _raw_spin_lock+0x20/0x30 [90377.883312] [] ? wake_up_state+0x20/0x20 [90377.883330] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.883363] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883383] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.883384] [] ? kthread+0xd1/0xe0 [90377.883403] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.883405] [] ? insert_kthread_work+0x40/0x40 [90377.883424] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.883426] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883431] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.883432] [] ? insert_kthread_work+0x40/0x40 [90377.883450] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883470] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883476] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.883495] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883518] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883540] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883541] [] ? kthread+0xd1/0xe0 [90377.883542] [] ? insert_kthread_work+0x40/0x40 [90377.883544] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883545] [] ? insert_kthread_work+0x40/0x40 [90377.883546] Task dump for CPU 4: [90377.883547] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.883548] Call Trace: [90377.883549] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.883551] [] ? _raw_spin_lock+0x20/0x30 [90377.883569] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.883588] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.883608] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.883627] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.883633] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.883652] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883671] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883677] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.883695] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883717] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883738] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883740] [] ? kthread+0xd1/0xe0 [90377.883741] [] ? insert_kthread_work+0x40/0x40 [90377.883743] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883744] [] ? insert_kthread_work+0x40/0x40 [90377.883745] Task dump for CPU 5: [90377.883746] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.883746] Call Trace: [90377.883748] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.883749] [] ? _raw_spin_lock+0x20/0x30 [90377.883768] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.883787] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.883807] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.883827] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.883832] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.883852] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883872] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.883877] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.883897] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.883919] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.883920] [] ? wake_up_state+0x20/0x20 [90377.883942] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.883944] [] ? kthread+0xd1/0xe0 [90377.883945] [] ? insert_kthread_work+0x40/0x40 [90377.883947] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.883948] [] ? insert_kthread_work+0x40/0x40 [90377.883949] Task dump for CPU 6: [90377.883950] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.883950] Call Trace: [90377.883970] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.883989] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.884011] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884012] [] ? wake_up_state+0x20/0x20 [90377.884033] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884035] [] ? kthread+0xd1/0xe0 [90377.884036] [] ? insert_kthread_work+0x40/0x40 [90377.884038] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.884039] [] ? insert_kthread_work+0x40/0x40 [90377.884040] Task dump for CPU 7: [90377.884041] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.884041] Call Trace: [90377.884044] [] sched_show_task+0xa8/0x110 [90377.884045] [] dump_cpu_task+0x39/0x70 [90377.884047] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.884049] [] rcu_check_callbacks+0x442/0x730 [90377.884051] [] ? tick_sched_do_timer+0x50/0x50 [90377.884052] [] update_process_times+0x46/0x80 [90377.884054] [] tick_sched_handle+0x30/0x70 [90377.884055] [] tick_sched_timer+0x39/0x80 [90377.884057] [] __hrtimer_run_queues+0x10e/0x270 [90377.884058] [] hrtimer_interrupt+0xaf/0x1d0 [90377.884059] [] local_apic_timer_interrupt+0x3b/0x60 [90377.884061] [] smp_apic_timer_interrupt+0x43/0x60 [90377.884062] [] apic_timer_interrupt+0x16a/0x170 [90377.884064] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.884066] [] queued_spin_lock_slowpath+0xb/0xf [90377.884067] [] _raw_spin_lock+0x20/0x30 [90377.884086] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.884106] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.884126] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.884145] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.884151] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.884171] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884190] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884196] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.884214] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.884237] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884239] [] ? wake_up_state+0x20/0x20 [90377.884261] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884263] [] kthread+0xd1/0xe0 [90377.884264] [] ? insert_kthread_work+0x40/0x40 [90377.884266] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.884267] [] ? insert_kthread_work+0x40/0x40 [90377.884268] Task dump for CPU 8: [90377.884269] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.884269] Call Trace: [90377.884289] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.884311] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884313] [] ? wake_up_state+0x20/0x20 [90377.884334] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884336] [] ? kthread+0xd1/0xe0 [90377.884337] [] ? insert_kthread_work+0x40/0x40 [90377.884339] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.884340] [] ? insert_kthread_work+0x40/0x40 [90377.884341] Task dump for CPU 10: [90377.884342] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.884342] Call Trace: [90377.884344] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.884345] [] ? _raw_spin_lock+0x20/0x30 [90377.884364] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.884383] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.884403] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.884423] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.884428] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.884448] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884468] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884474] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.884493] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.884515] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884536] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884538] [] ? kthread+0xd1/0xe0 [90377.884539] [] ? insert_kthread_work+0x40/0x40 [90377.884541] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.884542] [] ? insert_kthread_work+0x40/0x40 [90377.884543] Task dump for CPU 12: [90377.884544] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.884544] Call Trace: [90377.884564] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.884585] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884587] [] ? wake_up_state+0x20/0x20 [90377.884608] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884610] [] ? kthread+0xd1/0xe0 [90377.884611] [] ? insert_kthread_work+0x40/0x40 [90377.884613] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.884614] [] ? insert_kthread_work+0x40/0x40 [90377.884615] Task dump for CPU 13: [90377.884616] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.884616] Call Trace: [90377.884617] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.884619] [] ? _raw_spin_lock+0x20/0x30 [90377.884637] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.884656] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.884676] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.884696] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.884701] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.884721] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884740] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.884746] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.884764] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.884786] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.884807] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.884809] [] ? kthread+0xd1/0xe0 [90377.884810] [] ? insert_kthread_work+0x40/0x40 [90377.884812] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.884813] [] ? insert_kthread_work+0x40/0x40 [90377.884814] Task dump for CPU 16: [90377.884814] Task dump for CPU 0: [90377.884815] ldlm_bl_14 R [90377.884815] ldlm_bl_18 R [90377.884815] running task [90377.884816] 0 28508 2 0x00000088 [90377.884816] running task 0 28512 2 0x00000088 [90377.884817] Call Trace: [90377.884817] Call Trace: [90377.884818] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.884819] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.884821] [] ? _raw_spin_lock+0x20/0x30 [90377.884822] [] ? _raw_spin_lock+0x20/0x30 [90377.884840] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.884859] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.884878] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.884898] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.884917] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.884937] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.884957] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.884976] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.884981] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.884986] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.885005] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885024] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885043] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885063] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885068] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.885072] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.885090] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885110] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885131] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885154] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885176] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885197] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885199] [] ? kthread+0xd1/0xe0 [90377.885200] [] ? kthread+0xd1/0xe0 [90377.885201] [] ? insert_kthread_work+0x40/0x40 [90377.885203] [] ? insert_kthread_work+0x40/0x40 [90377.885204] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885206] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885207] [] ? insert_kthread_work+0x40/0x40 [90377.885208] [] ? insert_kthread_work+0x40/0x40 [90377.885208] Task dump for CPU 17: [90377.885209] Task dump for CPU 1: [90377.885209] ldlm_bl_11 R [90377.885209] ldlm_bl_15 R [90377.885210] running task [90377.885210] 0 28505 2 0x00000088 [90377.885211] running task 0 28509 2 0x00000088 [90377.885211] Call Trace: [90377.885211] Call Trace: [90377.885212] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.885214] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.885215] [] ? _raw_spin_lock+0x20/0x30 [90377.885216] [] ? _raw_spin_lock+0x20/0x30 [90377.885235] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.885253] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.885272] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.885291] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.885310] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.885330] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.885350] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.885369] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.885374] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.885378] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.885397] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885416] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885436] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885455] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885460] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.885465] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.885483] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885503] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885525] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885547] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885568] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885590] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885591] [] ? kthread+0xd1/0xe0 [90377.885592] [] ? kthread+0xd1/0xe0 [90377.885594] [] ? insert_kthread_work+0x40/0x40 [90377.885595] [] ? insert_kthread_work+0x40/0x40 [90377.885596] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885598] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885599] [] ? insert_kthread_work+0x40/0x40 [90377.885600] [] ? insert_kthread_work+0x40/0x40 [90377.885601] Task dump for CPU 19: [90377.885601] Task dump for CPU 2: [90377.885601] ldlm_bl_05 R [90377.885602] ldlm_bl_07 R [90377.885602] running task [90377.885602] 0 27290 2 0x00000088 [90377.885603] running task 0 27292 2 0x00000088 [90377.885603] Call Trace: [90377.885604] Call Trace: [90377.885624] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885643] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885665] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885686] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885687] [] ? wake_up_state+0x20/0x20 [90377.885689] [] ? wake_up_state+0x20/0x20 [90377.885710] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885732] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885733] [] ? kthread+0xd1/0xe0 [90377.885735] [] ? kthread+0xd1/0xe0 [90377.885736] [] ? insert_kthread_work+0x40/0x40 [90377.885737] [] ? insert_kthread_work+0x40/0x40 [90377.885739] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885740] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885741] [] ? insert_kthread_work+0x40/0x40 [90377.885742] [] ? insert_kthread_work+0x40/0x40 [90377.885743] Task dump for CPU 21: [90377.885743] Task dump for CPU 3: [90377.885743] ldlm_bl_06 R [90377.885744] ldlm_bl_17 R [90377.885744] running task [90377.885744] 0 27291 2 0x00000088 [90377.885745] running task 0 28511 2 0x00000088 [90377.885745] Call Trace: [90377.885746] Call Trace: [90377.885747] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.885766] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885768] [] ? _raw_spin_lock+0x20/0x30 [90377.885789] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.885807] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.885809] [] ? wake_up_state+0x20/0x20 [90377.885828] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.885850] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.885870] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.885871] [] ? kthread+0xd1/0xe0 [90377.885890] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.885892] [] ? insert_kthread_work+0x40/0x40 [90377.885897] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.885898] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.885918] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885919] [] ? insert_kthread_work+0x40/0x40 [90377.885938] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.885943] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.885961] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.885984] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886007] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886009] [] ? kthread+0xd1/0xe0 [90377.886010] [] ? insert_kthread_work+0x40/0x40 [90377.886011] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886013] [] ? insert_kthread_work+0x40/0x40 [90377.886013] Task dump for CPU 4: [90377.886015] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.886015] Call Trace: [90377.886016] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.886018] [] ? _raw_spin_lock+0x20/0x30 [90377.886035] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.886056] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.886075] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.886096] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.886101] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.886119] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886138] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886143] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.886164] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886185] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886208] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886210] [] ? kthread+0xd1/0xe0 [90377.886211] [] ? insert_kthread_work+0x40/0x40 [90377.886213] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886214] [] ? insert_kthread_work+0x40/0x40 [90377.886215] Task dump for CPU 5: [90377.886216] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.886216] Call Trace: [90377.886217] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.886219] [] ? _raw_spin_lock+0x20/0x30 [90377.886236] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.886255] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.886276] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.886297] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.886302] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.886320] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886339] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886344] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.886362] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886385] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886387] [] ? wake_up_state+0x20/0x20 [90377.886408] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886410] [] ? kthread+0xd1/0xe0 [90377.886411] [] ? insert_kthread_work+0x40/0x40 [90377.886412] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886414] [] ? insert_kthread_work+0x40/0x40 [90377.886414] Task dump for CPU 6: [90377.886415] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.886416] Call Trace: [90377.886436] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.886457] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.886480] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886481] [] ? wake_up_state+0x20/0x20 [90377.886502] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886504] [] ? kthread+0xd1/0xe0 [90377.886505] [] ? insert_kthread_work+0x40/0x40 [90377.886507] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886508] [] ? insert_kthread_work+0x40/0x40 [90377.886509] Task dump for CPU 7: [90377.886510] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.886510] Call Trace: [90377.886529] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886550] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886552] [] ? wake_up_state+0x20/0x20 [90377.886572] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886574] [] ? kthread+0xd1/0xe0 [90377.886575] [] ? insert_kthread_work+0x40/0x40 [90377.886576] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886578] [] ? insert_kthread_work+0x40/0x40 [90377.886578] Task dump for CPU 8: [90377.886580] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.886580] Call Trace: [90377.886601] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886623] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886625] [] ? wake_up_state+0x20/0x20 [90377.886647] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886649] [] ? kthread+0xd1/0xe0 [90377.886650] [] ? insert_kthread_work+0x40/0x40 [90377.886652] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886653] [] ? insert_kthread_work+0x40/0x40 [90377.886654] Task dump for CPU 10: [90377.886655] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.886655] Call Trace: [90377.886656] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.886658] [] ? _raw_spin_lock+0x20/0x30 [90377.886676] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.886697] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.886715] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.886736] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.886741] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.886760] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886779] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.886787] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.886807] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886830] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886853] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886854] [] ? kthread+0xd1/0xe0 [90377.886856] [] ? insert_kthread_work+0x40/0x40 [90377.886857] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886858] [] ? insert_kthread_work+0x40/0x40 [90377.886859] Task dump for CPU 12: [90377.886860] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.886860] Call Trace: [90377.886881] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.886904] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.886906] [] ? wake_up_state+0x20/0x20 [90377.886928] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.886930] [] ? kthread+0xd1/0xe0 [90377.886931] [] ? insert_kthread_work+0x40/0x40 [90377.886932] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.886934] [] ? insert_kthread_work+0x40/0x40 [90377.886934] Task dump for CPU 13: [90377.886935] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.886936] Call Trace: [90377.886937] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.886939] [] ? _raw_spin_lock+0x20/0x30 [90377.886956] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.886977] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.886996] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.887017] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.887022] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.887040] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887059] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887064] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.887084] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887105] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887126] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.887128] [] ? kthread+0xd1/0xe0 [90377.887129] [] ? insert_kthread_work+0x40/0x40 [90377.887130] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.887132] [] ? insert_kthread_work+0x40/0x40 [90377.887132] Task dump for CPU 16: [90377.887133] Task dump for CPU 0: [90377.887134] ldlm_bl_14 R [90377.887134] ldlm_bl_18 R [90377.887135] running task [90377.887135] 0 28508 2 0x00000088 [90377.887136] running task 0 28512 2 0x00000088 [90377.887137] Call Trace: [90377.887137] Call Trace: [90377.887138] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.887140] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.887142] [] ? _raw_spin_lock+0x20/0x30 [90377.887144] [] ? _raw_spin_lock+0x20/0x30 [90377.887161] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.887179] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.887199] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.887218] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.887237] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.887256] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.887276] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.887295] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.887300] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.887305] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.887323] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887343] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887361] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887381] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887386] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.887391] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.887409] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887429] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887451] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887473] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887496] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.887517] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.887519] [] ? kthread+0xd1/0xe0 [90377.887521] [] ? kthread+0xd1/0xe0 [90377.887522] [] ? insert_kthread_work+0x40/0x40 [90377.887523] [] ? insert_kthread_work+0x40/0x40 [90377.887525] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.887526] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.887528] [] ? insert_kthread_work+0x40/0x40 [90377.887529] [] ? insert_kthread_work+0x40/0x40 [90377.887529] Task dump for CPU 17: [90377.887530] Task dump for CPU 1: [90377.887531] ldlm_bl_11 R [90377.887531] ldlm_bl_15 R [90377.887532] running task [90377.887532] 0 28505 2 0x00000088 [90377.887533] running task 0 28509 2 0x00000088 [90377.887533] Call Trace: [90377.887534] Call Trace: [90377.887535] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.887537] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.887538] [] ? _raw_spin_lock+0x20/0x30 [90377.887540] [] ? _raw_spin_lock+0x20/0x30 [90377.887557] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.887576] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.887596] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.887616] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.887637] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.887655] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.887676] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.887695] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.887700] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.887705] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.887723] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887741] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887759] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887778] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.887783] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.887788] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.887809] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887828] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887850] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887872] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887895] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.887917] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.887919] [] ? kthread+0xd1/0xe0 [90377.887920] [] ? kthread+0xd1/0xe0 [90377.887922] [] ? insert_kthread_work+0x40/0x40 [90377.887923] [] ? insert_kthread_work+0x40/0x40 [90377.887925] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.887926] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.887928] [] ? insert_kthread_work+0x40/0x40 [90377.887929] [] ? insert_kthread_work+0x40/0x40 [90377.887929] Task dump for CPU 19: [90377.887930] Task dump for CPU 2: [90377.887930] ldlm_bl_05 R [90377.887931] ldlm_bl_07 R [90377.887931] running task [90377.887932] 0 27290 2 0x00000088 [90377.887933] running task 0 27292 2 0x00000088 [90377.887933] Call Trace: [90377.887933] Call Trace: [90377.887954] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.887955] [] sched_show_task+0xa8/0x110 [90377.887977] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.887979] [] dump_cpu_task+0x39/0x70 [90377.887980] [] ? wake_up_state+0x20/0x20 [90377.887982] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.888003] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888004] [] rcu_check_callbacks+0x442/0x730 [90377.888006] [] ? kthread+0xd1/0xe0 [90377.888007] [] ? tick_sched_do_timer+0x50/0x50 [90377.888009] [] ? insert_kthread_work+0x40/0x40 [90377.888010] [] update_process_times+0x46/0x80 [90377.888012] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888013] [] tick_sched_handle+0x30/0x70 [90377.888014] [] ? insert_kthread_work+0x40/0x40 [90377.888016] [] tick_sched_timer+0x39/0x80 [90377.888016] Task dump for CPU 3: [90377.888017] [] __hrtimer_run_queues+0x10e/0x270 [90377.888019] [] hrtimer_interrupt+0xaf/0x1d0 [90377.888020] [] local_apic_timer_interrupt+0x3b/0x60 [90377.888021] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.888022] [] smp_apic_timer_interrupt+0x43/0x60 [90377.888023] Call Trace: [90377.888024] [] apic_timer_interrupt+0x16a/0x170 [90377.888026] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.888028] [] ? _raw_spin_lock+0x20/0x30 [90377.888029] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.888047] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.888049] [] queued_spin_lock_slowpath+0xb/0xf [90377.888068] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.888070] [] _raw_spin_lock+0x20/0x30 [90377.888089] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.888106] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.888126] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.888144] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.888149] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.888167] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.888186] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888207] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.888225] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888230] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.888235] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.888253] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888272] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.888291] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888312] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888317] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.888339] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888359] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.888361] [] ? kthread+0xd1/0xe0 [90377.888383] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888385] [] ? insert_kthread_work+0x40/0x40 [90377.888386] [] ? wake_up_state+0x20/0x20 [90377.888388] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888410] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888411] [] ? insert_kthread_work+0x40/0x40 [90377.888413] [] kthread+0xd1/0xe0 [90377.888413] Task dump for CPU 4: [90377.888414] [] ? insert_kthread_work+0x40/0x40 [90377.888416] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.888418] [] ? insert_kthread_work+0x40/0x40 [90377.888419] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.888419] Task dump for CPU 21: [90377.888419] Call Trace: [90377.888421] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.888423] [] ? _raw_spin_lock+0x20/0x30 [90377.888424] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.888443] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.888443] Call Trace: [90377.888462] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.888483] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.888502] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.888524] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888543] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.888545] [] ? wake_up_state+0x20/0x20 [90377.888550] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.888572] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888591] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888593] [] ? kthread+0xd1/0xe0 [90377.888611] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888613] [] ? insert_kthread_work+0x40/0x40 [90377.888618] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.888619] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888638] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.888640] [] ? insert_kthread_work+0x40/0x40 [90377.888661] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888684] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888685] [] ? kthread+0xd1/0xe0 [90377.888687] [] ? insert_kthread_work+0x40/0x40 [90377.888688] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888689] [] ? insert_kthread_work+0x40/0x40 [90377.888690] Task dump for CPU 5: [90377.888691] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.888692] Call Trace: [90377.888693] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.888695] [] ? _raw_spin_lock+0x20/0x30 [90377.888713] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.888733] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.888753] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.888773] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.888779] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.888798] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888817] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.888823] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.888842] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.888864] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888866] [] ? wake_up_state+0x20/0x20 [90377.888888] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888890] [] ? kthread+0xd1/0xe0 [90377.888891] [] ? insert_kthread_work+0x40/0x40 [90377.888893] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888894] [] ? insert_kthread_work+0x40/0x40 [90377.888895] Task dump for CPU 6: [90377.888896] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.888896] Call Trace: [90377.888916] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.888936] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.888957] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.888959] [] ? wake_up_state+0x20/0x20 [90377.888980] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.888982] [] ? kthread+0xd1/0xe0 [90377.888983] [] ? insert_kthread_work+0x40/0x40 [90377.888985] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.888986] [] ? insert_kthread_work+0x40/0x40 [90377.888987] Task dump for CPU 7: [90377.888988] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.888988] Call Trace: [90377.889008] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889030] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889032] [] ? wake_up_state+0x20/0x20 [90377.889053] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889054] [] ? kthread+0xd1/0xe0 [90377.889056] [] ? insert_kthread_work+0x40/0x40 [90377.889057] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889059] [] ? insert_kthread_work+0x40/0x40 [90377.889059] Task dump for CPU 8: [90377.889060] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.889061] Call Trace: [90377.889081] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889102] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889104] [] ? wake_up_state+0x20/0x20 [90377.889125] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889127] [] ? kthread+0xd1/0xe0 [90377.889128] [] ? insert_kthread_work+0x40/0x40 [90377.889130] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889131] [] ? insert_kthread_work+0x40/0x40 [90377.889132] Task dump for CPU 10: [90377.889133] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.889133] Call Trace: [90377.889135] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.889136] [] ? _raw_spin_lock+0x20/0x30 [90377.889154] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.889174] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.889193] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.889212] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.889218] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.889236] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889256] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889261] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.889280] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889301] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889323] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889325] [] ? kthread+0xd1/0xe0 [90377.889326] [] ? insert_kthread_work+0x40/0x40 [90377.889328] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889329] [] ? insert_kthread_work+0x40/0x40 [90377.889330] Task dump for CPU 12: [90377.889331] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.889331] Call Trace: [90377.889350] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889372] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889374] [] ? wake_up_state+0x20/0x20 [90377.889395] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889397] [] ? kthread+0xd1/0xe0 [90377.889398] [] ? insert_kthread_work+0x40/0x40 [90377.889400] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889401] [] ? insert_kthread_work+0x40/0x40 [90377.889402] Task dump for CPU 13: [90377.889403] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.889403] Call Trace: [90377.889404] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.889406] [] ? _raw_spin_lock+0x20/0x30 [90377.889424] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.889444] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.889464] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.889484] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.889489] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.889509] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889529] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889535] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.889553] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889576] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889597] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889599] [] ? kthread+0xd1/0xe0 [90377.889600] [] ? insert_kthread_work+0x40/0x40 [90377.889602] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889603] [] ? insert_kthread_work+0x40/0x40 [90377.889604] Task dump for CPU 16: [90377.889604] Task dump for CPU 0: [90377.889605] ldlm_bl_14 R [90377.889605] ldlm_bl_18 R [90377.889605] running task [90377.889606] 0 28508 2 0x00000088 [90377.889606] running task 0 28512 2 0x00000088 [90377.889606] Call Trace: [90377.889607] Call Trace: [90377.889608] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.889609] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.889611] [] ? _raw_spin_lock+0x20/0x30 [90377.889612] [] ? _raw_spin_lock+0x20/0x30 [90377.889630] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.889649] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.889668] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.889688] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.889707] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.889726] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.889746] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.889765] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.889770] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.889775] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.889794] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889814] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889833] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889852] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.889857] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.889862] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.889880] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889900] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.889921] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889943] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.889964] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889985] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.889986] [] ? kthread+0xd1/0xe0 [90377.889987] [] ? kthread+0xd1/0xe0 [90377.889989] [] ? insert_kthread_work+0x40/0x40 [90377.889990] [] ? insert_kthread_work+0x40/0x40 [90377.889992] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889993] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.889994] [] ? insert_kthread_work+0x40/0x40 [90377.889995] [] ? insert_kthread_work+0x40/0x40 [90377.889996] Task dump for CPU 17: [90377.889996] Task dump for CPU 1: [90377.889997] ldlm_bl_11 R [90377.889997] ldlm_bl_15 R [90377.889997] running task [90377.889998] 0 28505 2 0x00000088 [90377.889998] running task 0 28509 2 0x00000088 [90377.889998] Call Trace: [90377.889999] Call Trace: [90377.890000] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.890002] [] ? _raw_spin_lock+0x20/0x30 [90377.890004] [] sched_show_task+0xa8/0x110 [90377.890022] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.890024] [] dump_cpu_task+0x39/0x70 [90377.890043] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.890045] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.890064] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.890066] [] rcu_check_callbacks+0x442/0x730 [90377.890085] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.890087] [] ? tick_sched_do_timer+0x50/0x50 [90377.890092] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.890093] [] update_process_times+0x46/0x80 [90377.890113] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890114] [] tick_sched_handle+0x30/0x70 [90377.890134] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890135] [] tick_sched_timer+0x39/0x80 [90377.890140] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.890142] [] __hrtimer_run_queues+0x10e/0x270 [90377.890161] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890162] [] hrtimer_interrupt+0xaf/0x1d0 [90377.890184] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890185] [] local_apic_timer_interrupt+0x3b/0x60 [90377.890207] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890209] [] smp_apic_timer_interrupt+0x43/0x60 [90377.890210] [] ? kthread+0xd1/0xe0 [90377.890212] [] apic_timer_interrupt+0x16a/0x170 [90377.890213] [] ? insert_kthread_work+0x40/0x40 [90377.890215] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.890216] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.890217] [] ? insert_kthread_work+0x40/0x40 [90377.890219] [] queued_spin_lock_slowpath+0xb/0xf [90377.890219] Task dump for CPU 2: [90377.890221] [] _raw_spin_lock+0x20/0x30 [90377.890239] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.890240] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.890260] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.890260] Call Trace: [90377.890279] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.890298] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890318] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.890339] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890345] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.890346] [] ? wake_up_state+0x20/0x20 [90377.890364] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890385] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890405] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890406] [] ? kthread+0xd1/0xe0 [90377.890411] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.890412] [] ? insert_kthread_work+0x40/0x40 [90377.890431] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890433] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.890455] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890456] [] ? insert_kthread_work+0x40/0x40 [90377.890478] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890479] Task dump for CPU 3: [90377.890480] [] kthread+0xd1/0xe0 [90377.890482] [] ? insert_kthread_work+0x40/0x40 [90377.890482] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.890484] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.890484] Call Trace: [90377.890485] [] ? insert_kthread_work+0x40/0x40 [90377.890487] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.890487] Task dump for CPU 19: [90377.890488] [] ? _raw_spin_lock+0x20/0x30 [90377.890507] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.890507] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.890526] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.890527] Call Trace: [90377.890546] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.890565] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890585] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.890606] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890611] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.890613] [] ? wake_up_state+0x20/0x20 [90377.890631] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890653] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890672] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890674] [] ? kthread+0xd1/0xe0 [90377.890679] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.890680] [] ? insert_kthread_work+0x40/0x40 [90377.890699] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890701] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.890723] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890725] [] ? insert_kthread_work+0x40/0x40 [90377.890745] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890746] Task dump for CPU 21: [90377.890747] [] ? kthread+0xd1/0xe0 [90377.890749] [] ? insert_kthread_work+0x40/0x40 [90377.890750] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.890751] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.890752] [] ? insert_kthread_work+0x40/0x40 [90377.890752] Call Trace: [90377.890753] Task dump for CPU 4: [90377.890772] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890794] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.890795] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.890796] [] ? wake_up_state+0x20/0x20 [90377.890797] Call Trace: [90377.890817] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.890819] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.890820] [] ? kthread+0xd1/0xe0 [90377.890822] [] ? _raw_spin_lock+0x20/0x30 [90377.890823] [] ? insert_kthread_work+0x40/0x40 [90377.890842] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.890844] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.890863] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.890865] [] ? insert_kthread_work+0x40/0x40 [90377.890884] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.890905] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.890911] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.890929] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890947] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.890953] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.890971] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.890992] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891013] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891014] [] ? kthread+0xd1/0xe0 [90377.891016] [] ? insert_kthread_work+0x40/0x40 [90377.891017] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891019] [] ? insert_kthread_work+0x40/0x40 [90377.891019] Task dump for CPU 5: [90377.891020] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.891021] Call Trace: [90377.891023] [] sched_show_task+0xa8/0x110 [90377.891025] [] dump_cpu_task+0x39/0x70 [90377.891026] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.891028] [] rcu_check_callbacks+0x442/0x730 [90377.891030] [] ? tick_sched_do_timer+0x50/0x50 [90377.891031] [] update_process_times+0x46/0x80 [90377.891033] [] tick_sched_handle+0x30/0x70 [90377.891034] [] tick_sched_timer+0x39/0x80 [90377.891035] [] __hrtimer_run_queues+0x10e/0x270 [90377.891037] [] hrtimer_interrupt+0xaf/0x1d0 [90377.891038] [] local_apic_timer_interrupt+0x3b/0x60 [90377.891039] [] smp_apic_timer_interrupt+0x43/0x60 [90377.891041] [] apic_timer_interrupt+0x16a/0x170 [90377.891043] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90377.891044] [] queued_spin_lock_slowpath+0xb/0xf [90377.891046] [] _raw_spin_lock+0x20/0x30 [90377.891063] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.891082] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.891103] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.891122] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.891127] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.891146] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891164] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891170] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.891190] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891211] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891213] [] ? wake_up_state+0x20/0x20 [90377.891234] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891236] [] kthread+0xd1/0xe0 [90377.891237] [] ? insert_kthread_work+0x40/0x40 [90377.891239] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.891240] [] ? insert_kthread_work+0x40/0x40 [90377.891241] Task dump for CPU 6: [90377.891242] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.891242] Call Trace: [90377.891261] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.891282] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.891303] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891305] [] ? wake_up_state+0x20/0x20 [90377.891325] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891327] [] ? kthread+0xd1/0xe0 [90377.891328] [] ? insert_kthread_work+0x40/0x40 [90377.891330] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891331] [] ? insert_kthread_work+0x40/0x40 [90377.891332] Task dump for CPU 7: [90377.891333] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.891333] Call Trace: [90377.891354] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891375] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891376] [] ? wake_up_state+0x20/0x20 [90377.891397] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891399] [] ? kthread+0xd1/0xe0 [90377.891400] [] ? insert_kthread_work+0x40/0x40 [90377.891402] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891403] [] ? insert_kthread_work+0x40/0x40 [90377.891403] Task dump for CPU 8: [90377.891405] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.891405] Call Trace: [90377.891424] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891445] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891446] [] ? wake_up_state+0x20/0x20 [90377.891467] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891468] [] ? kthread+0xd1/0xe0 [90377.891470] [] ? insert_kthread_work+0x40/0x40 [90377.891471] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891472] [] ? insert_kthread_work+0x40/0x40 [90377.891473] Task dump for CPU 10: [90377.891474] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.891474] Call Trace: [90377.891476] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.891477] [] ? _raw_spin_lock+0x20/0x30 [90377.891495] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.891514] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.891535] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.891556] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.891561] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.891579] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891598] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891604] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.891622] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891643] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891664] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891665] [] ? kthread+0xd1/0xe0 [90377.891666] [] ? insert_kthread_work+0x40/0x40 [90377.891668] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891669] [] ? insert_kthread_work+0x40/0x40 [90377.891670] Task dump for CPU 12: [90377.891671] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.891671] Call Trace: [90377.891692] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891713] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891715] [] ? wake_up_state+0x20/0x20 [90377.891736] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891737] [] ? kthread+0xd1/0xe0 [90377.891738] [] ? insert_kthread_work+0x40/0x40 [90377.891740] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891741] [] ? insert_kthread_work+0x40/0x40 [90377.891742] Task dump for CPU 13: [90377.891743] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.891743] Call Trace: [90377.891744] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.891746] [] ? _raw_spin_lock+0x20/0x30 [90377.891764] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.891783] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.891803] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.891822] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.891827] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.891845] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891864] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.891870] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.891888] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.891909] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.891930] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.891932] [] ? kthread+0xd1/0xe0 [90377.891933] [] ? insert_kthread_work+0x40/0x40 [90377.891935] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.891936] [] ? insert_kthread_work+0x40/0x40 [90377.891937] Task dump for CPU 16: [90377.891937] Task dump for CPU 0: [90377.891938] ldlm_bl_14 R [90377.891939] ldlm_bl_18 R [90377.891939] running task [90377.891940] 0 28508 2 0x00000088 [90377.891941] running task 0 28512 2 0x00000088 [90377.891941] Call Trace: [90377.891941] Call Trace: [90377.891943] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.891945] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.891947] [] ? _raw_spin_lock+0x20/0x30 [90377.891948] [] ? _raw_spin_lock+0x20/0x30 [90377.891966] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.891986] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.892004] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.892025] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.892043] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.892062] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.892080] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.892100] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.892105] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.892111] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.892128] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892147] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892167] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892186] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892191] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.892196] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.892216] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892236] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892257] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892279] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892300] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892321] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892322] [] ? kthread+0xd1/0xe0 [90377.892324] [] ? kthread+0xd1/0xe0 [90377.892325] [] ? insert_kthread_work+0x40/0x40 [90377.892326] [] ? insert_kthread_work+0x40/0x40 [90377.892328] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892330] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892331] [] ? insert_kthread_work+0x40/0x40 [90377.892332] [] ? insert_kthread_work+0x40/0x40 [90377.892333] Task dump for CPU 17: [90377.892333] Task dump for CPU 1: [90377.892334] ldlm_bl_11 R [90377.892334] ldlm_bl_15 R [90377.892335] running task [90377.892335] 0 28505 2 0x00000088 [90377.892336] running task 0 28509 2 0x00000088 [90377.892337] Call Trace: [90377.892337] Call Trace: [90377.892338] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.892340] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.892341] [] ? _raw_spin_lock+0x20/0x30 [90377.892343] [] ? _raw_spin_lock+0x20/0x30 [90377.892360] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.892377] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.892396] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.892416] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.892434] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.892455] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.892473] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.892494] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.892499] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.892504] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.892522] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892540] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892558] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892576] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.892581] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.892586] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.892606] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892624] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892645] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892668] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892690] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892713] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892715] [] ? kthread+0xd1/0xe0 [90377.892716] [] ? kthread+0xd1/0xe0 [90377.892718] [] ? insert_kthread_work+0x40/0x40 [90377.892719] [] ? insert_kthread_work+0x40/0x40 [90377.892721] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892722] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892723] [] ? insert_kthread_work+0x40/0x40 [90377.892725] [] ? insert_kthread_work+0x40/0x40 [90377.892725] Task dump for CPU 19: [90377.892725] Task dump for CPU 2: [90377.892726] ldlm_bl_05 R [90377.892727] ldlm_bl_07 R [90377.892727] running task [90377.892728] 0 27290 2 0x00000088 [90377.892728] running task 0 27292 2 0x00000088 [90377.892729] Call Trace: [90377.892729] Call Trace: [90377.892750] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892768] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892789] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892812] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892813] [] ? wake_up_state+0x20/0x20 [90377.892815] [] ? wake_up_state+0x20/0x20 [90377.892835] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892858] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892859] [] ? kthread+0xd1/0xe0 [90377.892861] [] ? kthread+0xd1/0xe0 [90377.892862] [] ? insert_kthread_work+0x40/0x40 [90377.892864] [] ? insert_kthread_work+0x40/0x40 [90377.892865] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892867] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892868] [] ? insert_kthread_work+0x40/0x40 [90377.892869] [] ? insert_kthread_work+0x40/0x40 [90377.892870] Task dump for CPU 21: [90377.892870] Task dump for CPU 3: [90377.892871] ldlm_bl_06 R [90377.892871] ldlm_bl_17 R [90377.892872] running task [90377.892872] 0 27291 2 0x00000088 [90377.892873] running task 0 28511 2 0x00000088 [90377.892873] Call Trace: [90377.892874] Call Trace: [90377.892892] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.892894] [] sched_show_task+0xa8/0x110 [90377.892915] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.892917] [] dump_cpu_task+0x39/0x70 [90377.892918] [] ? wake_up_state+0x20/0x20 [90377.892920] [] rcu_dump_cpu_stacks+0x90/0xd0 [90377.892941] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.892942] [] rcu_check_callbacks+0x442/0x730 [90377.892944] [] ? kthread+0xd1/0xe0 [90377.892945] [] ? tick_sched_do_timer+0x50/0x50 [90377.892947] [] ? insert_kthread_work+0x40/0x40 [90377.892948] [] update_process_times+0x46/0x80 [90377.892950] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.892951] [] tick_sched_handle+0x30/0x70 [90377.892952] [] ? insert_kthread_work+0x40/0x40 [90377.892954] [] tick_sched_timer+0x39/0x80 [90377.892955] [] __hrtimer_run_queues+0x10e/0x270 [90377.892956] [] hrtimer_interrupt+0xaf/0x1d0 [90377.892957] [] local_apic_timer_interrupt+0x3b/0x60 [90377.892959] [] smp_apic_timer_interrupt+0x43/0x60 [90377.892960] [] apic_timer_interrupt+0x16a/0x170 [90377.892962] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [90377.892964] [] queued_spin_lock_slowpath+0xb/0xf [90377.892965] [] _raw_spin_lock+0x20/0x30 [90377.892983] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.893004] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.893022] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.893043] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.893049] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.893067] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893086] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893092] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.893110] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893133] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893156] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893158] [] kthread+0xd1/0xe0 [90377.893160] [] ? insert_kthread_work+0x40/0x40 [90377.893161] [] ret_from_fork_nospec_begin+0x7/0x21 [90377.893163] [] ? insert_kthread_work+0x40/0x40 [90377.893163] Task dump for CPU 4: [90377.893165] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.893165] Call Trace: [90377.893166] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.893168] [] ? _raw_spin_lock+0x20/0x30 [90377.893186] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.893206] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.893227] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.893248] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.893254] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.893274] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893293] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893298] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.893318] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893339] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893362] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893364] [] ? kthread+0xd1/0xe0 [90377.893365] [] ? insert_kthread_work+0x40/0x40 [90377.893367] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.893368] [] ? insert_kthread_work+0x40/0x40 [90377.893368] Task dump for CPU 5: [90377.893370] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.893370] Call Trace: [90377.893371] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.893373] [] ? _raw_spin_lock+0x20/0x30 [90377.893391] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.893410] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.893428] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.893447] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.893452] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.893470] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893489] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893494] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.893515] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893536] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893538] [] ? wake_up_state+0x20/0x20 [90377.893560] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893562] [] ? kthread+0xd1/0xe0 [90377.893563] [] ? insert_kthread_work+0x40/0x40 [90377.893565] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.893566] [] ? insert_kthread_work+0x40/0x40 [90377.893566] Task dump for CPU 6: [90377.893568] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.893568] Call Trace: [90377.893588] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.893607] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.893627] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893629] [] ? wake_up_state+0x20/0x20 [90377.893652] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893653] [] ? kthread+0xd1/0xe0 [90377.893655] [] ? insert_kthread_work+0x40/0x40 [90377.893656] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.893657] [] ? insert_kthread_work+0x40/0x40 [90377.893658] Task dump for CPU 7: [90377.893659] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.893659] Call Trace: [90377.893680] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893703] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893705] [] ? wake_up_state+0x20/0x20 [90377.893727] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893728] [] ? kthread+0xd1/0xe0 [90377.893730] [] ? insert_kthread_work+0x40/0x40 [90377.893731] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.893732] [] ? insert_kthread_work+0x40/0x40 [90377.893733] Task dump for CPU 8: [90377.893734] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.893734] Call Trace: [90377.893755] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893776] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.893778] [] ? wake_up_state+0x20/0x20 [90377.893799] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.893800] [] ? kthread+0xd1/0xe0 [90377.893802] [] ? insert_kthread_work+0x40/0x40 [90377.893803] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.893804] [] ? insert_kthread_work+0x40/0x40 [90377.893805] Task dump for CPU 10: [90377.893806] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.893806] Call Trace: [90377.893808] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.893809] [] ? _raw_spin_lock+0x20/0x30 [90377.893827] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.893847] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.893868] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.893889] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.893894] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.893912] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893931] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.893936] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.893957] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.893979] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894002] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894004] [] ? kthread+0xd1/0xe0 [90377.894005] [] ? insert_kthread_work+0x40/0x40 [90377.894007] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894008] [] ? insert_kthread_work+0x40/0x40 [90377.894009] Task dump for CPU 12: [90377.894010] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.894010] Call Trace: [90377.894029] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894050] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894052] [] ? wake_up_state+0x20/0x20 [90377.894074] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894076] [] ? kthread+0xd1/0xe0 [90377.894077] [] ? insert_kthread_work+0x40/0x40 [90377.894079] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894080] [] ? insert_kthread_work+0x40/0x40 [90377.894081] Task dump for CPU 13: [90377.894082] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.894082] Call Trace: [90377.894083] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.894085] [] ? _raw_spin_lock+0x20/0x30 [90377.894103] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.894123] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.894142] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.894163] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.894168] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.894186] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894205] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894210] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.894228] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894251] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894274] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894275] [] ? kthread+0xd1/0xe0 [90377.894277] [] ? insert_kthread_work+0x40/0x40 [90377.894278] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894279] [] ? insert_kthread_work+0x40/0x40 [90377.894280] Task dump for CPU 16: [90377.894283] 0 [90377.894284] ldlm_bl_14 R [90377.894285] 1 [90377.894285] running task [90377.894287] 0 28508 2 0x00000088 [90377.894288] Call Trace: [90377.894291] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.894294] [] ? _raw_spin_lock+0x20/0x30 [90377.894313] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.894334] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.894356] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.894378] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.894384] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.894402] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894421] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894426] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.894444] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894465] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894486] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894488] [] ? kthread+0xd1/0xe0 [90377.894489] [] ? insert_kthread_work+0x40/0x40 [90377.894491] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894492] [] ? insert_kthread_work+0x40/0x40 [90377.894493] Task dump for CPU 17: [90377.894493] 2 3 4 5 6 7 8 10 12 13 [90377.894494] ldlm_bl_11 R running task 0 28505 2 0x00000088 [90377.894494] Call Trace: [90377.894496] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.894497] [] ? _raw_spin_lock+0x20/0x30 [90377.894515] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.894535] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.894554] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.894575] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.894580] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.894598] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894617] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.894622] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.894643] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894665] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894689] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894690] [] ? kthread+0xd1/0xe0 [90377.894692] [] ? insert_kthread_work+0x40/0x40 [90377.894693] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894695] [] ? insert_kthread_work+0x40/0x40 [90377.894695] Task dump for CPU 19: [90377.894696] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.894696] Call Trace: [90377.894717] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894740] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894742] [] ? wake_up_state+0x20/0x20 [90377.894764] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894766] [] ? kthread+0xd1/0xe0 [90377.894767] [] ? insert_kthread_work+0x40/0x40 [90377.894769] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894770] [] ? insert_kthread_work+0x40/0x40 [90377.894771] Task dump for CPU 21: [90377.894772] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.894772] Call Trace: [90377.894793] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.894816] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.894818] [] ? wake_up_state+0x20/0x20 [90377.894840] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.894841] [] ? kthread+0xd1/0xe0 [90377.894843] [] ? insert_kthread_work+0x40/0x40 [90377.894844] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.894846] [] ? insert_kthread_work+0x40/0x40 [90377.894860] 16 17 19 21} (detected by 14, t=60002 jiffies, g=678150, c=678149, q=2698) [90377.894861] Task dump for CPU 0: [90377.894862] ldlm_bl_18 R running task 0 28512 2 0x00000088 [90377.894863] Call Trace: [90377.894867] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.894870] [] ? _raw_spin_lock+0x20/0x30 [90377.894900] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.894919] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.894940] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.894961] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.894972] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.894990] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895008] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895015] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.895035] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895059] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895082] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895084] [] ? kthread+0xd1/0xe0 [90377.895085] [] ? insert_kthread_work+0x40/0x40 [90377.895088] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895089] [] ? insert_kthread_work+0x40/0x40 [90377.895089] Task dump for CPU 1: [90377.895091] ldlm_bl_15 R running task 0 28509 2 0x00000088 [90377.895091] Call Trace: [90377.895092] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.895094] [] ? _raw_spin_lock+0x20/0x30 [90377.895112] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.895130] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.895149] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.895170] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.895175] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.895193] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895212] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895217] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.895238] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895261] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895282] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895283] [] ? kthread+0xd1/0xe0 [90377.895285] [] ? insert_kthread_work+0x40/0x40 [90377.895286] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895288] [] ? insert_kthread_work+0x40/0x40 [90377.895288] Task dump for CPU 2: [90377.895289] ldlm_bl_07 R running task 0 27292 2 0x00000088 [90377.895289] Call Trace: [90377.895311] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895331] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895333] [] ? wake_up_state+0x20/0x20 [90377.895356] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895357] [] ? kthread+0xd1/0xe0 [90377.895359] [] ? insert_kthread_work+0x40/0x40 [90377.895360] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895362] [] ? insert_kthread_work+0x40/0x40 [90377.895362] Task dump for CPU 3: [90377.895363] ldlm_bl_17 R running task 0 28511 2 0x00000088 [90377.895364] Call Trace: [90377.895365] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.895367] [] ? _raw_spin_lock+0x20/0x30 [90377.895385] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.895405] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.895424] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.895445] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.895450] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.895468] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895487] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895492] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.895510] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895531] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895554] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895555] [] ? kthread+0xd1/0xe0 [90377.895557] [] ? insert_kthread_work+0x40/0x40 [90377.895558] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895560] [] ? insert_kthread_work+0x40/0x40 [90377.895560] Task dump for CPU 4: [90377.895561] ldlm_bl_13 R running task 0 28507 2 0x00000088 [90377.895561] Call Trace: [90377.895563] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.895564] [] ? _raw_spin_lock+0x20/0x30 [90377.895582] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.895603] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.895621] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.895642] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.895647] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.895665] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895684] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895689] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.895710] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895732] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895755] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895757] [] ? kthread+0xd1/0xe0 [90377.895758] [] ? insert_kthread_work+0x40/0x40 [90377.895760] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895761] [] ? insert_kthread_work+0x40/0x40 [90377.895762] Task dump for CPU 5: [90377.895763] ldlm_bl_02 R running task 0 3841 2 0x00000088 [90377.895764] Call Trace: [90377.895765] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.895767] [] ? _raw_spin_lock+0x20/0x30 [90377.895784] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.895804] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.895823] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.895844] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.895849] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.895867] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895886] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.895891] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.895912] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.895933] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.895934] [] ? wake_up_state+0x20/0x20 [90377.895955] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.895957] [] ? kthread+0xd1/0xe0 [90377.895958] [] ? insert_kthread_work+0x40/0x40 [90377.895960] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.895961] [] ? insert_kthread_work+0x40/0x40 [90377.895962] Task dump for CPU 6: [90377.895963] ldlm_bl_04 R running task 0 22908 2 0x00000088 [90377.895963] Call Trace: [90377.895984] [] ? ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90377.896003] [] ? ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90377.896024] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896025] [] ? wake_up_state+0x20/0x20 [90377.896048] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896049] [] ? kthread+0xd1/0xe0 [90377.896051] [] ? insert_kthread_work+0x40/0x40 [90377.896052] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896054] [] ? insert_kthread_work+0x40/0x40 [90377.896054] Task dump for CPU 7: [90377.896055] ldlm_bl_01 R running task 0 3840 2 0x00000088 [90377.896056] Call Trace: [90377.896075] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896096] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896098] [] ? wake_up_state+0x20/0x20 [90377.896119] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896120] [] ? kthread+0xd1/0xe0 [90377.896121] [] ? insert_kthread_work+0x40/0x40 [90377.896123] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896124] [] ? insert_kthread_work+0x40/0x40 [90377.896125] Task dump for CPU 8: [90377.896126] ldlm_bl_03 R running task 0 22779 2 0x00000088 [90377.896126] Call Trace: [90377.896147] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896168] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896170] [] ? wake_up_state+0x20/0x20 [90377.896192] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896194] [] ? kthread+0xd1/0xe0 [90377.896195] [] ? insert_kthread_work+0x40/0x40 [90377.896197] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896198] [] ? insert_kthread_work+0x40/0x40 [90377.896199] Task dump for CPU 10: [90377.896200] ldlm_bl_12 R running task 0 28506 2 0x00000088 [90377.896200] Call Trace: [90377.896201] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.896203] [] ? _raw_spin_lock+0x20/0x30 [90377.896220] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.896241] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.896260] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.896281] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.896286] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.896304] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896323] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896328] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.896348] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896371] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896394] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896395] [] ? kthread+0xd1/0xe0 [90377.896397] [] ? insert_kthread_work+0x40/0x40 [90377.896398] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896400] [] ? insert_kthread_work+0x40/0x40 [90377.896400] Task dump for CPU 12: [90377.896402] ldlm_bl_08 R running task 0 27293 2 0x00000088 [90377.896402] Call Trace: [90377.896423] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896444] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896445] [] ? wake_up_state+0x20/0x20 [90377.896466] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896468] [] ? kthread+0xd1/0xe0 [90377.896469] [] ? insert_kthread_work+0x40/0x40 [90377.896471] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896472] [] ? insert_kthread_work+0x40/0x40 [90377.896473] Task dump for CPU 13: [90377.896474] ldlm_bl_16 R running task 0 28510 2 0x00000088 [90377.896474] Call Trace: [90377.896475] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.896477] [] ? _raw_spin_lock+0x20/0x30 [90377.896495] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.896516] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.896536] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.896557] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.896562] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.896580] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896599] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896605] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.896625] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896646] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896669] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896670] [] ? kthread+0xd1/0xe0 [90377.896671] [] ? insert_kthread_work+0x40/0x40 [90377.896673] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896674] [] ? insert_kthread_work+0x40/0x40 [90377.896675] Task dump for CPU 16: [90377.896676] ldlm_bl_14 R running task 0 28508 2 0x00000088 [90377.896676] Call Trace: [90377.896678] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.896679] [] ? _raw_spin_lock+0x20/0x30 [90377.896697] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.896716] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.896736] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.896757] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.896762] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.896781] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896800] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.896805] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.896826] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.896849] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.896872] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.896873] [] ? kthread+0xd1/0xe0 [90377.896874] [] ? insert_kthread_work+0x40/0x40 [90377.896876] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.896877] [] ? insert_kthread_work+0x40/0x40 [90377.896878] Task dump for CPU 17: [90377.896879] ldlm_bl_11 R running task 0 28505 2 0x00000088 [90377.896879] Call Trace: [90377.896881] [] ? queued_spin_lock_slowpath+0xb/0xf [90377.896882] [] ? _raw_spin_lock+0x20/0x30 [90377.896900] [] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [90377.896921] [] ? ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90377.896941] [] ? ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90377.896960] [] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90377.896966] [] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90377.896983] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.897002] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90377.897007] [] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90377.897028] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.897051] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.897072] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.897074] [] ? kthread+0xd1/0xe0 [90377.897075] [] ? insert_kthread_work+0x40/0x40 [90377.897077] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.897078] [] ? insert_kthread_work+0x40/0x40 [90377.897078] Task dump for CPU 19: [90377.897080] ldlm_bl_05 R running task 0 27290 2 0x00000088 [90377.897080] Call Trace: [90377.897101] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.897122] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.897124] [] ? wake_up_state+0x20/0x20 [90377.897146] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.897147] [] ? kthread+0xd1/0xe0 [90377.897149] [] ? insert_kthread_work+0x40/0x40 [90377.897150] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.897151] [] ? insert_kthread_work+0x40/0x40 [90377.897152] Task dump for CPU 21: [90377.897153] ldlm_bl_06 R running task 0 27291 2 0x00000088 [90377.897153] Call Trace: [90377.897174] [] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90377.897195] [] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90377.897197] [] ? wake_up_state+0x20/0x20 [90377.897218] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90377.897219] [] ? kthread+0xd1/0xe0 [90377.897221] [] ? insert_kthread_work+0x40/0x40 [90377.897222] [] ? ret_from_fork_nospec_begin+0x7/0x21 [90377.897223] [] ? insert_kthread_work+0x40/0x40 [90398.867399] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_21:28515] [90398.867418] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.867426] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.867428] CPU: 9 PID: 28515 Comm: ldlm_bl_21 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.867429] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.867429] task: ffff8a65cab9c200 ti: ffff8a65582c4000 task.ti: ffff8a65582c4000 [90398.867432] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.867432] RSP: 0018:ffff8a65582c7c10 EFLAGS: 00000246 [90398.867433] RAX: 0000000000000000 RBX: ffffd11580c1b340 RCX: 0000000000490000 [90398.867434] RDX: ffff8a657e45b8c0 RSI: 0000000000910001 RDI: ffff8a56673df99c [90398.867434] RBP: ffff8a65582c7c10 R08: ffff8a757db1b8c0 R09: 0000000000000000 [90398.867435] R10: ffff8a7568f2a458 R11: ffff8a7568f2a448 R12: ffff8a757db1acc0 [90398.867435] R13: ffffffffaa4e05ac R14: ffff8a65582c7b98 R15: ffffffffaa4e23f9 [90398.867436] FS: 0000000000000000(0000) GS:ffff8a757db00000(0000) knlGS:0000000000000000 [90398.867437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.867437] CR2: 00000000022f64b8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.867438] Call Trace: [90398.867439] [] queued_spin_lock_slowpath+0xb/0xf [90398.867441] [] _raw_spin_lock+0x20/0x30 [90398.867461] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.867480] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.867501] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.867503] [] ? __kmalloc+0x1eb/0x230 [90398.867524] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.867530] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.867548] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.867567] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.867573] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.867591] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.867613] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.867635] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.867636] [] kthread+0xd1/0xe0 [90398.867638] [] ? insert_kthread_work+0x40/0x40 [90398.867640] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.867641] [] ? insert_kthread_work+0x40/0x40 [90398.867655] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90398.876398] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [ldlm_bl_09:27294] [90398.876417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.876425] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.876427] CPU: 11 PID: 27294 Comm: ldlm_bl_09 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.876427] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.876428] task: ffff8a5642ba4200 ti: ffff8a6561074000 task.ti: ffff8a6561074000 [90398.876430] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.876431] RSP: 0018:ffff8a6561077c10 EFLAGS: 00000246 [90398.876431] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000590000 [90398.876432] RDX: ffff8a757dc1b8c0 RSI: 0000000000890001 RDI: ffff8a56673df99c [90398.876432] RBP: ffff8a6561077c10 R08: ffff8a757db5b8c0 R09: 0000000000000000 [90398.876433] R10: ffff8a7568f4e458 R11: ffff8a7568f4e448 R12: ffff8a6561077bc0 [90398.876434] R13: ffff8a56bd7520c0 R14: ffff8a6561077b80 R15: ffffffffc0d2a5f8 [90398.876434] FS: 0000000000000000(0000) GS:ffff8a757db40000(0000) knlGS:0000000000000000 [90398.876435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.876436] CR2: 000056004b843070 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.876436] Call Trace: [90398.876438] [] queued_spin_lock_slowpath+0xb/0xf [90398.876439] [] _raw_spin_lock+0x20/0x30 [90398.876457] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.876478] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.876497] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.876518] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.876524] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.876542] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.876561] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.876567] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.876585] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.876609] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.876610] [] ? wake_up_state+0x20/0x20 [90398.876632] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.876633] [] kthread+0xd1/0xe0 [90398.876635] [] ? insert_kthread_work+0x40/0x40 [90398.876636] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.876638] [] ? insert_kthread_work+0x40/0x40 [90398.876652] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90398.890398] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ldlm_bl_10:27295] [90398.890417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.890425] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.890427] CPU: 15 PID: 27295 Comm: ldlm_bl_10 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.890427] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.890428] task: ffff8a5664fa6300 ti: ffff8a56917dc000 task.ti: ffff8a56917dc000 [90398.890430] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90398.890431] RSP: 0018:ffff8a56917dfc10 EFLAGS: 00000246 [90398.890432] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000790000 [90398.890432] RDX: ffff8a757da5b8c0 RSI: 0000000000190001 RDI: ffff8a56673df99c [90398.890433] RBP: ffff8a56917dfc10 R08: ffff8a757dbdb8c0 R09: 0000000000000000 [90398.890434] R10: ffff8a7569378e58 R11: ffff8a7569378e48 R12: ffff8a56917dfbc0 [90398.890434] R13: ffff8a56bd7523c0 R14: ffff8a56917dfb80 R15: ffffffffc0d2a5f8 [90398.890435] FS: 0000000000000000(0000) GS:ffff8a757dbc0000(0000) knlGS:0000000000000000 [90398.890436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.890436] CR2: 00007f6df7e16871 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.890437] Call Trace: [90398.890438] [] queued_spin_lock_slowpath+0xb/0xf [90398.890440] [] _raw_spin_lock+0x20/0x30 [90398.890460] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.890481] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.890500] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.890521] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.890527] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.890545] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.890564] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.890569] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.890590] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.890613] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.890615] [] ? wake_up_state+0x20/0x20 [90398.890636] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.890638] [] kthread+0xd1/0xe0 [90398.890639] [] ? insert_kthread_work+0x40/0x40 [90398.890641] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.890643] [] ? insert_kthread_work+0x40/0x40 [90398.890656] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90398.899400] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [ldlm_bl_20:28514] [90398.899429] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.899440] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.899442] CPU: 18 PID: 28514 Comm: ldlm_bl_20 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.899443] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.899444] task: ffff8a65cab9d280 ti: ffff8a65582c0000 task.ti: ffff8a65582c0000 [90398.899450] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.899451] RSP: 0018:ffff8a65582c3c10 EFLAGS: 00000246 [90398.899452] RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000910000 [90398.899452] RDX: ffff8a657e3db8c0 RSI: 0000000000710001 RDI: ffff8a56673df99c [90398.899453] RBP: ffff8a65582c3c10 R08: ffff8a657e45b8c0 R09: 0000000000000000 [90398.899454] R10: ffff8a756f2bac58 R11: ffff8a756f2bac48 R12: ffff8a657e45acc0 [90398.899454] R13: ffffffffaa4e05ac R14: ffff8a65582c3b98 R15: ffffffffaa4e23f9 [90398.899455] FS: 0000000000000000(0000) GS:ffff8a657e440000(0000) knlGS:0000000000000000 [90398.899456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.899457] CR2: 0000000000ef1090 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.899458] Call Trace: [90398.899461] [] queued_spin_lock_slowpath+0xb/0xf [90398.899464] [] _raw_spin_lock+0x20/0x30 [90398.899498] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.899521] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.899544] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.899546] [] ? __kmalloc+0x1eb/0x230 [90398.899570] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.899583] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.899607] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.899634] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.899641] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.899664] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.899692] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.899718] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.899720] [] kthread+0xd1/0xe0 [90398.899722] [] ? insert_kthread_work+0x40/0x40 [90398.899724] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.899726] [] ? insert_kthread_work+0x40/0x40 [90398.899742] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90398.905399] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ldlm_bl_23:28517] [90398.905420] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.905429] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.905431] CPU: 20 PID: 28517 Comm: ldlm_bl_23 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.905432] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.905433] task: ffff8a65cab99080 ti: ffff8a65582d4000 task.ti: ffff8a65582d4000 [90398.905435] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.905436] RSP: 0018:ffff8a65582d7c10 EFLAGS: 00000246 [90398.905437] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a10000 [90398.905438] RDX: ffff8a757dcdb8c0 RSI: 0000000000b90001 RDI: ffff8a56673df99c [90398.905438] RBP: ffff8a65582d7c10 R08: ffff8a657e49b8c0 R09: 0000000000000000 [90398.905439] R10: ffff8a756e862258 R11: ffff8a756e862248 R12: ffff8a65582d7bc0 [90398.905439] R13: ffff8a756f2d2840 R14: ffff8a65582d7b80 R15: ffffffffc0d2a5f8 [90398.905440] FS: 0000000000000000(0000) GS:ffff8a657e480000(0000) knlGS:0000000000000000 [90398.905441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.905441] CR2: 00007f7ca3d3bbd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.905442] Call Trace: [90398.905444] [] queued_spin_lock_slowpath+0xb/0xf [90398.905446] [] _raw_spin_lock+0x20/0x30 [90398.905468] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.905487] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.905508] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.905529] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.905536] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.905554] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.905573] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.905579] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.905599] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.905621] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.905642] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.905644] [] kthread+0xd1/0xe0 [90398.905646] [] ? insert_kthread_work+0x40/0x40 [90398.905648] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.905650] [] ? insert_kthread_work+0x40/0x40 [90398.905664] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90398.911398] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [ldlm_bl_24:28518] [90398.911418] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.911426] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.911428] CPU: 22 PID: 28518 Comm: ldlm_bl_24 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.911429] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.911429] task: ffff8a65cab98000 ti: ffff8a656905c000 task.ti: ffff8a656905c000 [90398.911432] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.911432] RSP: 0018:ffff8a656905fc10 EFLAGS: 00000246 [90398.911433] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b10000 [90398.911434] RDX: ffff8a657e49b8c0 RSI: 0000000000a10001 RDI: ffff8a56673df99c [90398.911434] RBP: ffff8a656905fc10 R08: ffff8a657e4db8c0 R09: 0000000000000000 [90398.911435] R10: ffff8a6637350058 R11: ffff8a6637350048 R12: ffff8a656905fbc0 [90398.911436] R13: ffff8a756f2d2b40 R14: ffff8a656905fb80 R15: ffffffffc0d2a5f8 [90398.911436] FS: 0000000000000000(0000) GS:ffff8a657e4c0000(0000) knlGS:0000000000000000 [90398.911437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.911438] CR2: 00007efc6a1ca000 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.911438] Call Trace: [90398.911440] [] queued_spin_lock_slowpath+0xb/0xf [90398.911442] [] _raw_spin_lock+0x20/0x30 [90398.911463] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.911482] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.911501] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.911522] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.911529] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.911547] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.911566] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.911572] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.911590] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.911614] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.911635] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.911637] [] kthread+0xd1/0xe0 [90398.911639] [] ? insert_kthread_work+0x40/0x40 [90398.911640] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.911642] [] ? insert_kthread_work+0x40/0x40 [90398.911656] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90398.914398] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ldlm_bl_22:28516] [90398.914417] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90398.914425] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90398.914426] CPU: 23 PID: 28516 Comm: ldlm_bl_22 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90398.914427] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90398.914427] task: ffff8a65cab9a100 ti: ffff8a65582d0000 task.ti: ffff8a65582d0000 [90398.914429] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90398.914430] RSP: 0018:ffff8a65582d3c10 EFLAGS: 00000246 [90398.914430] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000b90000 [90398.914431] RDX: ffff8a757db1b8c0 RSI: 0000000000490001 RDI: ffff8a56673df99c [90398.914432] RBP: ffff8a65582d3c10 R08: ffff8a757dcdb8c0 R09: 0000000000000000 [90398.914432] R10: ffff8a65cbbf8458 R11: ffff8a65cbbf8448 R12: ffff8a65582d3bc0 [90398.914433] R13: ffff8a756f313e00 R14: ffff8a65582d3b80 R15: ffffffffc0d2a5f8 [90398.914434] FS: 0000000000000000(0000) GS:ffff8a757dcc0000(0000) knlGS:0000000000000000 [90398.914434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90398.914435] CR2: 00007f0a08c7a9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90398.914435] Call Trace: [90398.914437] [] queued_spin_lock_slowpath+0xb/0xf [90398.914438] [] _raw_spin_lock+0x20/0x30 [90398.914456] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90398.914474] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90398.914493] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90398.914514] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90398.914520] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90398.914538] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.914557] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90398.914562] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90398.914581] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90398.914604] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90398.914625] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90398.914627] [] kthread+0xd1/0xe0 [90398.914628] [] ? insert_kthread_work+0x40/0x40 [90398.914630] [] ret_from_fork_nospec_begin+0x7/0x21 [90398.914631] [] ? insert_kthread_work+0x40/0x40 [90398.914645] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.830328] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_15:28509] [90402.830348] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.830356] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.830357] CPU: 1 PID: 28509 Comm: ldlm_bl_15 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.830358] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.830359] task: ffff8a65c37ba100 ti: ffff8a65582ac000 task.ti: ffff8a65582ac000 [90402.830361] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.830361] RSP: 0018:ffff8a65582afc10 EFLAGS: 00000246 [90402.830362] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000090000 [90402.830363] RDX: ffff8a657e31b8c0 RSI: 0000000000410001 RDI: ffff8a56673df99c [90402.830363] RBP: ffff8a65582afc10 R08: ffff8a757da1b8c0 R09: 0000000000000000 [90402.830364] R10: ffff8a5650be3658 R11: ffff8a5650be3648 R12: ffff8a65582afbc0 [90402.830365] R13: ffff8a56673dfe00 R14: ffff8a65582afb80 R15: ffffffffc0d2a5f8 [90402.830365] FS: 0000000000000000(0000) GS:ffff8a757da00000(0000) knlGS:0000000000000000 [90402.830366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.830367] CR2: 00007efff91a9000 CR3: 000000010d78e000 CR4: 00000000001607e0 [90402.830367] Call Trace: [90402.830369] [] queued_spin_lock_slowpath+0xb/0xf [90402.830370] [] _raw_spin_lock+0x20/0x30 [90402.830389] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.830407] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.830428] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.830447] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.830453] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.830472] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.830490] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.830496] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.830515] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.830536] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.830558] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.830560] [] kthread+0xd1/0xe0 [90402.830561] [] ? insert_kthread_work+0x40/0x40 [90402.830563] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.830564] [] ? insert_kthread_work+0x40/0x40 [90402.830578] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.835330] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ldlm_bl_07:27292] [90402.835356] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.835367] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.835369] CPU: 2 PID: 27292 Comm: ldlm_bl_07 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.835370] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.835371] task: ffff8a5642ba2100 ti: ffff8a6560504000 task.ti: ffff8a6560504000 [90402.835376] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90402.835376] RSP: 0018:ffff8a6560507c10 EFLAGS: 00000246 [90402.835377] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000110000 [90402.835378] RDX: ffff8a657e41b8c0 RSI: 0000000000810001 RDI: ffff8a56673df99c [90402.835378] RBP: ffff8a6560507c10 R08: ffff8a657e25b8c0 R09: 0000000000000000 [90402.835379] R10: ffff8a56497f4458 R11: ffff8a56497f4448 R12: ffff8a6560507bc0 [90402.835379] R13: ffff8a756f2d2e40 R14: ffff8a6560507b80 R15: ffffffffc0d2a5f8 [90402.835380] FS: 0000000000000000(0000) GS:ffff8a657e240000(0000) knlGS:0000000000000000 [90402.835381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.835381] CR2: 00007fef15016140 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.835382] Call Trace: [90402.835385] [] queued_spin_lock_slowpath+0xb/0xf [90402.835388] [] _raw_spin_lock+0x20/0x30 [90402.835417] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.835436] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.835455] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.835476] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.835488] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.835506] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.835525] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.835531] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.835552] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.835575] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.835578] [] ? wake_up_state+0x20/0x20 [90402.835601] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.835603] [] kthread+0xd1/0xe0 [90402.835604] [] ? insert_kthread_work+0x40/0x40 [90402.835606] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.835608] [] ? insert_kthread_work+0x40/0x40 [90402.835622] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90402.840329] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_17:28511] [90402.840348] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.840356] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.840358] CPU: 3 PID: 28511 Comm: ldlm_bl_17 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.840359] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.840359] task: ffff8a56517ae300 ti: ffff8a65582b4000 task.ti: ffff8a65582b4000 [90402.840361] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.840362] RSP: 0018:ffff8a65582b7c10 EFLAGS: 00000246 [90402.840363] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000190000 [90402.840363] RDX: ffff8a657e25b8c0 RSI: 0000000000110001 RDI: ffff8a56673df99c [90402.840364] RBP: ffff8a65582b7c10 R08: ffff8a757da5b8c0 R09: 0000000000000000 [90402.840364] R10: ffff8a756cdedc58 R11: ffff8a756cdedc48 R12: ffff8a65582b7bc0 [90402.840365] R13: ffff8a56673de000 R14: ffff8a65582b7b80 R15: ffffffffc0d2a5f8 [90402.840366] FS: 0000000000000000(0000) GS:ffff8a757da40000(0000) knlGS:0000000000000000 [90402.840366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.840367] CR2: 00000000006dee20 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.840367] Call Trace: [90402.840369] [] queued_spin_lock_slowpath+0xb/0xf [90402.840371] [] _raw_spin_lock+0x20/0x30 [90402.840388] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.840407] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.840426] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.840447] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.840452] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.840471] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.840489] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.840495] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.840516] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.840537] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.840559] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.840560] [] kthread+0xd1/0xe0 [90402.840562] [] ? insert_kthread_work+0x40/0x40 [90402.840563] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.840565] [] ? insert_kthread_work+0x40/0x40 [90402.840578] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.844330] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ldlm_bl_13:28507] [90402.844354] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.844365] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.844367] CPU: 4 PID: 28507 Comm: ldlm_bl_13 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.844367] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.844369] task: ffff8a65c47a2100 ti: ffff8a6558294000 task.ti: ffff8a6558294000 [90402.844373] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90402.844374] RSP: 0018:ffff8a6558297c10 EFLAGS: 00000246 [90402.844375] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000210000 [90402.844376] RDX: ffff8a757dc5b8c0 RSI: 0000000000990001 RDI: ffff8a56673df99c [90402.844376] RBP: ffff8a6558297c10 R08: ffff8a657e29b8c0 R09: 0000000000000000 [90402.844377] R10: ffff8a6633765258 R11: ffff8a6633765248 R12: ffff8a6558297bc0 [90402.844377] R13: ffff8a756f313b00 R14: ffff8a6558297b80 R15: ffffffffc0d2a5f8 [90402.844378] FS: 0000000000000000(0000) GS:ffff8a657e280000(0000) knlGS:0000000000000000 [90402.844379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.844380] CR2: 00007fdec83450a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.844380] Call Trace: [90402.844383] [] queued_spin_lock_slowpath+0xb/0xf [90402.844386] [] _raw_spin_lock+0x20/0x30 [90402.844412] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.844431] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.844452] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.844471] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.844482] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.844500] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.844519] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.844525] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.844544] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.844566] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.844589] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.844591] [] kthread+0xd1/0xe0 [90402.844593] [] ? insert_kthread_work+0x40/0x40 [90402.844595] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.844596] [] ? insert_kthread_work+0x40/0x40 [90402.844610] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90402.849328] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [ldlm_bl_02:3841] [90402.849348] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.849356] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.849357] CPU: 5 PID: 3841 Comm: ldlm_bl_02 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.849358] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.849359] task: ffff8a6569fec200 ti: ffff8a6569ffc000 task.ti: ffff8a6569ffc000 [90402.849361] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.849361] RSP: 0018:ffff8a6569fffc10 EFLAGS: 00000246 [90402.849362] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000290000 [90402.849363] RDX: ffff8a657e39b8c0 RSI: 0000000000610001 RDI: ffff8a56673df99c [90402.849363] RBP: ffff8a6569fffc10 R08: ffff8a757da9b8c0 R09: 0000000000000000 [90402.849364] R10: ffff8a7568932058 R11: ffff8a7568932048 R12: ffff8a6569fffbc0 [90402.849365] R13: ffff8a56bd753d40 R14: ffff8a6569fffb80 R15: ffffffffc0d2a5f8 [90402.849365] FS: 0000000000000000(0000) GS:ffff8a757da80000(0000) knlGS:0000000000000000 [90402.849366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.849367] CR2: 00007f0a08ce1c70 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.849367] Call Trace: [90402.849369] [] queued_spin_lock_slowpath+0xb/0xf [90402.849370] [] _raw_spin_lock+0x20/0x30 [90402.849388] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.849407] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.849428] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.849447] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.849453] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.849472] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.849490] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.849496] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.849514] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.849536] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.849538] [] ? wake_up_state+0x20/0x20 [90402.849559] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.849561] [] kthread+0xd1/0xe0 [90402.849562] [] ? insert_kthread_work+0x40/0x40 [90402.849564] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.849565] [] ? insert_kthread_work+0x40/0x40 [90402.849579] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.853329] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [ldlm_bl_04:22908] [90402.853353] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.853364] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.853366] CPU: 6 PID: 22908 Comm: ldlm_bl_04 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.853367] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.853367] task: ffff8a5644bca100 ti: ffff8a6560430000 task.ti: ffff8a6560430000 [90402.853396] RIP: 0010:[] [] ldlm_process_plain_lock+0x491/0xab0 [ptlrpc] [90402.853396] RSP: 0018:ffff8a6560433b88 EFLAGS: 00000283 [90402.853397] RAX: ffff8a75636f1640 RBX: 0000000000000000 RCX: 0000000000000010 [90402.853397] RDX: 0000000000000000 RSI: ffff8a655a49fbc0 RDI: ffff8a756a714ec0 [90402.853398] RBP: ffff8a6560433bd0 R08: ffff8a6560433c28 R09: ffff8a657e39b8c0 [90402.853398] R10: ffff8a56633e4e58 R11: ffff8a655a49fbc0 R12: 0000000000000002 [90402.853399] R13: ffff8a7566cc06c0 R14: 0000000000000000 R15: 0000000000000000 [90402.853400] FS: 0000000000000000(0000) GS:ffff8a657e2c0000(0000) knlGS:0000000000000000 [90402.853400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.853401] CR2: 00007fff7e854fc8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.853402] Call Trace: [90402.853422] [] ? ldlm_errno2error+0x60/0x60 [ptlrpc] [90402.853441] [] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [90402.853460] [] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [90402.853481] [] ldlm_reprocess_res+0x28/0x30 [ptlrpc] [90402.853490] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.853508] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90402.853527] [] ? ldlm_lock_mode_downgrade+0x330/0x330 [ptlrpc] [90402.853533] [] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs] [90402.853554] [] ldlm_reprocess_recovery_done+0x3c/0x110 [ptlrpc] [90402.853575] [] ldlm_export_cancel_locks+0x16c/0x180 [ptlrpc] [90402.853598] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.853600] [] ? wake_up_state+0x20/0x20 [90402.853624] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.853626] [] kthread+0xd1/0xe0 [90402.853627] [] ? insert_kthread_work+0x40/0x40 [90402.853629] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.853630] [] ? insert_kthread_work+0x40/0x40 [90402.853644] Code: a0 0f 84 98 02 00 00 4c 39 fb ba 01 00 00 00 0f 84 fd 04 00 00 0f 1f 44 00 00 49 8b 87 08 02 00 00 8b 4d cc 4c 8b b0 60 fe ff ff <41> 8b 87 98 00 00 00 85 0c 85 40 70 37 c1 75 64 4d 85 e4 0f 84 [90402.858328] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ldlm_bl_01:3840] [90402.858348] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.858356] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.858358] CPU: 7 PID: 3840 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.858358] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.858359] task: ffff8a6569feb180 ti: ffff8a6569ff8000 task.ti: ffff8a6569ff8000 [90402.858361] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90402.858362] RSP: 0018:ffff8a6569ffbc10 EFLAGS: 00000246 [90402.858362] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000390000 [90402.858363] RDX: ffff8a657e29b8c0 RSI: 0000000000210001 RDI: ffff8a56673df99c [90402.858364] RBP: ffff8a6569ffbc10 R08: ffff8a757dadb8c0 R09: 0000000000000000 [90402.858364] R10: ffff8a75692b7658 R11: ffff8a75692b7648 R12: ffff8a6569ffbbc0 [90402.858365] R13: ffff8a756f2d3d40 R14: ffff8a6569ffbb80 R15: ffffffffc0d2a5f8 [90402.858366] FS: 0000000000000000(0000) GS:ffff8a757dac0000(0000) knlGS:0000000000000000 [90402.858366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.858367] CR2: 00007f0a09682978 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.858367] Call Trace: [90402.858369] [] queued_spin_lock_slowpath+0xb/0xf [90402.858370] [] _raw_spin_lock+0x20/0x30 [90402.858388] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.858410] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.858429] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.858449] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.858455] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.858473] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.858492] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.858498] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.858516] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.858540] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.858542] [] ? wake_up_state+0x20/0x20 [90402.858565] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.858566] [] kthread+0xd1/0xe0 [90402.858568] [] ? insert_kthread_work+0x40/0x40 [90402.858569] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.858571] [] ? insert_kthread_work+0x40/0x40 [90402.858584] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90402.862329] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [ldlm_bl_03:22779] [90402.862352] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.862362] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.862364] CPU: 8 PID: 22779 Comm: ldlm_bl_03 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.862365] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.862365] task: ffff8a65e97a9080 ti: ffff8a6561218000 task.ti: ffff8a6561218000 [90402.862369] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.862369] RSP: 0018:ffff8a656121bc10 EFLAGS: 00000246 [90402.862370] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000410000 [90402.862371] RDX: ffff8a757dc9b8c0 RSI: 0000000000a90001 RDI: ffff8a56673df99c [90402.862371] RBP: ffff8a656121bc10 R08: ffff8a657e31b8c0 R09: 0000000000000000 [90402.862372] R10: ffff8a564c3c6c58 R11: ffff8a564c3c6c48 R12: ffff8a656121bbc0 [90402.862372] R13: ffff8a756f2d2180 R14: ffff8a656121bb80 R15: ffffffffc0d2a5f8 [90402.862373] FS: 0000000000000000(0000) GS:ffff8a657e300000(0000) knlGS:0000000000000000 [90402.862374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.862374] CR2: 0000563d758c22e8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.862375] Call Trace: [90402.862378] [] queued_spin_lock_slowpath+0xb/0xf [90402.862380] [] _raw_spin_lock+0x20/0x30 [90402.862407] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.862426] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.862447] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.862468] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.862477] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.862496] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.862515] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.862521] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.862542] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.862565] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.862567] [] ? wake_up_state+0x20/0x20 [90402.862591] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.862593] [] kthread+0xd1/0xe0 [90402.862594] [] ? insert_kthread_work+0x40/0x40 [90402.862596] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.862598] [] ? insert_kthread_work+0x40/0x40 [90402.862611] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.871329] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [ldlm_bl_12:28506] [90402.871352] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.871363] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.871364] CPU: 10 PID: 28506 Comm: ldlm_bl_12 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.871365] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.871366] task: ffff8a56bdb63180 ti: ffff8a6558290000 task.ti: ffff8a6558290000 [90402.871369] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90402.871370] RSP: 0018:ffff8a6558293c10 EFLAGS: 00000246 [90402.871371] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000510000 [90402.871371] RDX: ffff8a757db5b8c0 RSI: 0000000000590001 RDI: ffff8a56673df99c [90402.871372] RBP: ffff8a6558293c10 R08: ffff8a657e35b8c0 R09: 0000000000000000 [90402.871372] R10: ffff8a56943b5e58 R11: ffff8a56943b5e48 R12: ffff8a6558293bc0 [90402.871373] R13: ffff8a756f313800 R14: ffff8a6558293b80 R15: ffffffffc0d2a5f8 [90402.871374] FS: 0000000000000000(0000) GS:ffff8a657e340000(0000) knlGS:0000000000000000 [90402.871374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.871375] CR2: 00007fef13b0c1e4 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.871376] Call Trace: [90402.871378] [] queued_spin_lock_slowpath+0xb/0xf [90402.871380] [] _raw_spin_lock+0x20/0x30 [90402.871406] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.871425] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.871444] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.871463] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.871471] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.871489] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.871508] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.871514] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.871535] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.871557] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.871580] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.871582] [] kthread+0xd1/0xe0 [90402.871584] [] ? insert_kthread_work+0x40/0x40 [90402.871586] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.871587] [] ? insert_kthread_work+0x40/0x40 [90402.871601] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90402.879328] NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [ldlm_bl_08:27293] [90402.879349] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.879358] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.879359] CPU: 12 PID: 27293 Comm: ldlm_bl_08 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.879360] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.879361] task: ffff8a5642ba3180 ti: ffff8a564e3c8000 task.ti: ffff8a564e3c8000 [90402.879364] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [90402.879365] RSP: 0018:ffff8a564e3cbb90 EFLAGS: 00000202 [90402.879365] RAX: 0000000000000001 RBX: ffff8a564e3cbbc8 RCX: 0000000000610000 [90402.879366] RDX: 0000000000b10001 RSI: 0000000000310000 RDI: ffff8a56673df99c [90402.879366] RBP: ffff8a564e3cbb90 R08: ffff8a657e39b8c0 R09: 0000000000000000 [90402.879367] R10: ffff8a6561009a58 R11: ffff8a6561009a48 R12: 0000000000000000 [90402.879367] R13: 0000000000000000 R14: 0000000000000002 R15: ffff8a7562d77980 [90402.879368] FS: 0000000000000000(0000) GS:ffff8a657e380000(0000) knlGS:0000000000000000 [90402.879369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.879369] CR2: 00007f1c986fc9e5 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.879370] Call Trace: [90402.879372] [] queued_spin_lock_slowpath+0xb/0xf [90402.879374] [] _raw_spin_lock+0x20/0x30 [90402.879396] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.879414] [] ldlm_cancel_callback+0x92/0x330 [ptlrpc] [90402.879438] [] ? class_handle_unhash_nolock+0xd8/0x150 [obdclass] [90402.879445] [] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs] [90402.879461] [] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [90402.879480] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.879497] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.879503] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.879520] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.879537] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.879543] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.879561] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.879582] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.879584] [] ? wake_up_state+0x20/0x20 [90402.879604] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.879606] [] kthread+0xd1/0xe0 [90402.879607] [] ? insert_kthread_work+0x40/0x40 [90402.879609] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.879610] [] ? insert_kthread_work+0x40/0x40 [90402.879622] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [90402.884328] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [ldlm_bl_16:28510] [90402.884347] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.884354] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.884356] CPU: 13 PID: 28510 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.884356] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.884357] task: ffff8a56517ad280 ti: ffff8a65582b0000 task.ti: ffff8a65582b0000 [90402.884359] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.884360] RSP: 0018:ffff8a65582b3c10 EFLAGS: 00000246 [90402.884360] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000690000 [90402.884361] RDX: ffff8a757da9b8c0 RSI: 0000000000290001 RDI: ffff8a56673df99c [90402.884362] RBP: ffff8a65582b3c10 R08: ffff8a757db9b8c0 R09: 0000000000000000 [90402.884362] R10: ffff8a65c3382858 R11: ffff8a65c3382848 R12: ffff8a65582b3bc0 [90402.884363] R13: ffff8a56673dfa40 R14: ffff8a65582b3b80 R15: ffffffffc0d2a5f8 [90402.884364] FS: 0000000000000000(0000) GS:ffff8a757db80000(0000) knlGS:0000000000000000 [90402.884364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.884365] CR2: 00007f2096ed4bd0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.884365] Call Trace: [90402.884367] [] queued_spin_lock_slowpath+0xb/0xf [90402.884368] [] _raw_spin_lock+0x20/0x30 [90402.884388] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.884407] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.884426] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.884447] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.884453] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.884471] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.884489] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.884495] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.884513] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.884537] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.884559] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.884560] [] kthread+0xd1/0xe0 [90402.884562] [] ? insert_kthread_work+0x40/0x40 [90402.884563] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.884565] [] ? insert_kthread_work+0x40/0x40 [90402.884578] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.887328] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [ldlm_bl_19:28513] [90402.887352] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.887362] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.887364] CPU: 14 PID: 28513 Comm: ldlm_bl_19 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.887365] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.887365] task: ffff8a65cab9e300 ti: ffff8a65582bc000 task.ti: ffff8a65582bc000 [90402.887369] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.887369] RSP: 0018:ffff8a65582bfc10 EFLAGS: 00000246 [90402.887370] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000710000 [90402.887371] RDX: ffff8a757dadb8c0 RSI: 0000000000390001 RDI: ffff8a56673df99c [90402.887371] RBP: ffff8a65582bfc10 R08: ffff8a657e3db8c0 R09: 0000000000000000 [90402.887372] R10: ffff8a65cb3bca58 R11: ffff8a65cb3bca48 R12: ffff8a65582bfbc0 [90402.887372] R13: ffff8a756f313980 R14: ffff8a65582bfb80 R15: ffffffffc0d2a5f8 [90402.887373] FS: 0000000000000000(0000) GS:ffff8a657e3c0000(0000) knlGS:0000000000000000 [90402.887374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.887374] CR2: 00007fefa676b810 CR3: 000000101bf80000 CR4: 00000000001607e0 [90402.887375] Call Trace: [90402.887377] [] queued_spin_lock_slowpath+0xb/0xf [90402.887380] [] _raw_spin_lock+0x20/0x30 [90402.887403] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.887424] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.887445] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.887466] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.887473] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.887491] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.887510] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.887516] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.887537] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.887558] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.887582] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.887583] [] kthread+0xd1/0xe0 [90402.887585] [] ? insert_kthread_work+0x40/0x40 [90402.887587] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.887588] [] ? insert_kthread_work+0x40/0x40 [90402.887602] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.893328] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [ldlm_bl_14:28508] [90402.893348] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.893356] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.893358] CPU: 16 PID: 28508 Comm: ldlm_bl_14 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.893359] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.893359] task: ffff8a65c37bb180 ti: ffff8a65582a8000 task.ti: ffff8a65582a8000 [90402.893362] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [90402.893363] RSP: 0018:ffff8a65582abc10 EFLAGS: 00000246 [90402.893363] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000810000 [90402.893364] RDX: ffff8a657e21b8c0 RSI: 0000000000010001 RDI: ffff8a56673df99c [90402.893365] RBP: ffff8a65582abc10 R08: ffff8a657e41b8c0 R09: 0000000000000000 [90402.893365] R10: ffff8a756f2f5c58 R11: ffff8a756f2f5c48 R12: ffff8a65582abbc0 [90402.893366] R13: ffff8a756f3120c0 R14: ffff8a65582abb80 R15: ffffffffc0d2a5f8 [90402.893366] FS: 0000000000000000(0000) GS:ffff8a657e400000(0000) knlGS:0000000000000000 [90402.893367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.893368] CR2: 00007f266f2d3daa CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.893368] Call Trace: [90402.893370] [] queued_spin_lock_slowpath+0xb/0xf [90402.893371] [] _raw_spin_lock+0x20/0x30 [90402.893393] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.893414] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.893435] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.893456] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.893462] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.893480] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.893499] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.893505] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.893525] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.893547] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.893568] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.893570] [] kthread+0xd1/0xe0 [90402.893572] [] ? insert_kthread_work+0x40/0x40 [90402.893574] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.893575] [] ? insert_kthread_work+0x40/0x40 [90402.893589] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [90402.896327] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [ldlm_bl_11:28505] [90402.896346] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.896354] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.896356] CPU: 17 PID: 28505 Comm: ldlm_bl_11 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.896356] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.896357] task: ffff8a56bdb60000 ti: ffff8a655829c000 task.ti: ffff8a655829c000 [90402.896359] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.896360] RSP: 0018:ffff8a655829fc10 EFLAGS: 00000246 [90402.896360] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000890000 [90402.896361] RDX: ffff8a757da1b8c0 RSI: 0000000000090001 RDI: ffff8a56673df99c [90402.896362] RBP: ffff8a655829fc10 R08: ffff8a757dc1b8c0 R09: 0000000000000000 [90402.896362] R10: ffff8a661a792e58 R11: ffff8a661a792e48 R12: ffff8a655829fbc0 [90402.896363] R13: ffff8a756f312f00 R14: ffff8a655829fb80 R15: ffffffffc0d2a5f8 [90402.896363] FS: 0000000000000000(0000) GS:ffff8a757dc00000(0000) knlGS:0000000000000000 [90402.896364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.896365] CR2: 0000000000412515 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.896365] Call Trace: [90402.896366] [] queued_spin_lock_slowpath+0xb/0xf [90402.896368] [] _raw_spin_lock+0x20/0x30 [90402.896386] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.896405] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.896425] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.896444] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.896450] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.896468] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.896487] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.896493] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.896511] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.896532] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.896554] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.896555] [] kthread+0xd1/0xe0 [90402.896557] [] ? insert_kthread_work+0x40/0x40 [90402.896559] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.896560] [] ? insert_kthread_work+0x40/0x40 [90402.896574] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.902327] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [ldlm_bl_05:27290] [90402.902346] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.902354] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.902355] CPU: 19 PID: 27290 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.902356] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.902357] task: ffff8a5642ba0000 ti: ffff8a65693d0000 task.ti: ffff8a65693d0000 [90402.902359] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.902359] RSP: 0018:ffff8a65693d3c10 EFLAGS: 00000246 [90402.902360] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000990000 [90402.902360] RDX: ffff8a757dbdb8c0 RSI: 0000000000790001 RDI: ffff8a56673df99c [90402.902361] RBP: ffff8a65693d3c10 R08: ffff8a757dc5b8c0 R09: 0000000000000000 [90402.902362] R10: ffff8a7568900c58 R11: ffff8a7568900c48 R12: ffff8a65693d3bc0 [90402.902362] R13: ffff8a756f2d3140 R14: ffff8a65693d3b80 R15: ffffffffc0d2a5f8 [90402.902363] FS: 0000000000000000(0000) GS:ffff8a757dc40000(0000) knlGS:0000000000000000 [90402.902363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.902364] CR2: 00007f99a2b321a0 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.902364] Call Trace: [90402.902366] [] queued_spin_lock_slowpath+0xb/0xf [90402.902368] [] _raw_spin_lock+0x20/0x30 [90402.902386] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.902407] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.902425] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.902446] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.902452] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.902470] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.902489] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.902494] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.902515] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.902537] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.902538] [] ? wake_up_state+0x20/0x20 [90402.902561] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.902563] [] kthread+0xd1/0xe0 [90402.902564] [] ? insert_kthread_work+0x40/0x40 [90402.902566] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.902567] [] ? insert_kthread_work+0x40/0x40 [90402.902581] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90402.908327] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 23s! [ldlm_bl_06:27291] [90402.908346] Modules linked in: mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) mpt2sas mptctl mptbase ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp mxm_wmi intel_rapl iosf_mbi kvm_intel dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_round_robin lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter sunrpc dm_multipath dm_mod [90402.908354] ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) ahci fb_sys_fops mlxfw(OE) ttm libahci devlink mlx_compat(OE) mpt3sas(OE) drm libata crct10dif_pclmul tg3 crct10dif_common raid_class crc32c_intel ptp megaraid_sas scsi_transport_sas drm_panel_orientation_quirks pps_core [90402.908356] CPU: 21 PID: 27291 Comm: ldlm_bl_06 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90402.908356] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90402.908357] task: ffff8a5642ba5280 ti: ffff8a564f3a4000 task.ti: ffff8a564f3a4000 [90402.908359] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [90402.908360] RSP: 0018:ffff8a564f3a7c10 EFLAGS: 00000246 [90402.908360] RAX: 0000000000000000 RBX: 00000001c1261db2 RCX: 0000000000a90000 [90402.908361] RDX: ffff8a757db9b8c0 RSI: 0000000000690001 RDI: ffff8a56673df99c [90402.908361] RBP: ffff8a564f3a7c10 R08: ffff8a757dc9b8c0 R09: 0000000000000000 [90402.908362] R10: ffff8a663db72258 R11: ffff8a663db72248 R12: ffff8a564f3a7bc0 [90402.908363] R13: ffff8a56bd752480 R14: ffff8a564f3a7b80 R15: ffffffffc0d2a5f8 [90402.908363] FS: 0000000000000000(0000) GS:ffff8a757dc80000(0000) knlGS:0000000000000000 [90402.908364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90402.908365] CR2: 00000000020ca6d8 CR3: 0000001edb210000 CR4: 00000000001607e0 [90402.908365] Call Trace: [90402.908366] [] queued_spin_lock_slowpath+0xb/0xf [90402.908368] [] _raw_spin_lock+0x20/0x30 [90402.908386] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90402.908405] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90402.908426] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90402.908444] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90402.908450] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90402.908469] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.908488] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90402.908493] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90402.908512] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90402.908533] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90402.908535] [] ? wake_up_state+0x20/0x20 [90402.908557] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90402.908558] [] kthread+0xd1/0xe0 [90402.908560] [] ? insert_kthread_work+0x40/0x40 [90402.908561] [] ret_from_fork_nospec_begin+0x7/0x21 [90402.908563] [] ? insert_kthread_work+0x40/0x40 [90402.908576] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 13 15 ab 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [90414.507577] Uhhuh. NMI received for unknown reason 39 on CPU 0. [90414.507578] Do you have a strange power saving mode enabled? [90414.507579] Kernel panic - not syncing: NMI: Not continuing [90414.507581] CPU: 0 PID: 28512 Comm: ldlm_bl_18 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 [90414.507581] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.11.0 11/02/2019 [90414.507582] Call Trace: [90414.507589] [] dump_stack+0x19/0x1b [90414.507591] [] panic+0xe8/0x21f [90414.507595] [] nmi_panic+0x3f/0x40 [90414.507597] [] do_nmi+0x3f6/0x460 [90414.507598] [] end_repeat_nmi+0x1e/0x81 [90414.507603] [] ? mem32_serial_out+0x20/0x20 [90414.507605] [] ? io_serial_in+0x15/0x20 [90414.507607] [] ? io_serial_in+0x15/0x20 [90414.507608] [] ? io_serial_in+0x15/0x20 [90414.507611] [] wait_for_xmitr+0x46/0xb0 [90414.507613] [] serial8250_console_putchar+0x1c/0x40 [90414.507614] [] ? serial8250_console_write+0x160/0x160 [90414.507616] [] uart_console_write+0x3c/0x80 [90414.507618] [] serial8250_console_write+0xbf/0x160 [90414.507620] [] call_console_drivers.constprop.19+0x93/0xf0 [90414.507621] [] console_unlock+0x468/0x4b0 [90414.507622] [] vprintk_emit+0x3c4/0x510 [90414.507624] [] vprintk_default+0x29/0x40 [90414.507625] [] printk+0x60/0x77 [90414.507628] [] watchdog_timer_fn+0x1dc/0x240 [90414.507629] [] ? watchdog+0x40/0x40 [90414.507631] [] __hrtimer_run_queues+0x10e/0x270 [90414.507632] [] hrtimer_interrupt+0xaf/0x1d0 [90414.507634] [] local_apic_timer_interrupt+0x3b/0x60 [90414.507635] [] smp_apic_timer_interrupt+0x43/0x60 [90414.507638] [] apic_timer_interrupt+0x16a/0x170 [90414.507641] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [90414.507642] [] queued_spin_lock_slowpath+0xb/0xf [90414.507645] [] _raw_spin_lock+0x20/0x30 [90414.507678] [] lock_res_and_lock+0x2c/0x50 [ptlrpc] [90414.507698] [] ldlm_lock_cancel+0x2d/0x1f0 [ptlrpc] [90414.507719] [] ldlm_cancel_lock_for_export.isra.26+0x74/0x360 [ptlrpc] [90414.507738] [] ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [90414.507752] [] cfs_hash_for_each_relax+0x250/0x450 [libcfs] [90414.507771] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90414.507791] [] ? ldlm_cancel_lock_for_export.isra.26+0x360/0x360 [ptlrpc] [90414.507798] [] cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [90414.507817] [] ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [90414.507840] [] ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [90414.507863] [] ? ldlm_handle_bl_callback+0x420/0x420 [ptlrpc] [90414.507865] [] kthread+0xd1/0xe0 [90414.507866] [] ? insert_kthread_work+0x40/0x40 [90414.507868] [] ret_from_fork_nospec_begin+0x7/0x21 [90414.507869] [] ? insert_kthread_work+0x40/0x40