[ 0.000000] microcode: microcode updated early to revision 0xb00003e, date = 2021-02-06 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-1160.53.1.1chaos.ch6.x86_64 (mockbuild@builder1-x86.buildfarm.internal) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jan 25 12:06:24 PST 2022 [ 0.000000] Command line: initrd=initramfs ip=dhcp root=LABEL=/ netroot=iscsi:192.168.64.82::811:0:iqn.2006-04.gov.llnl:lc.pascal:compute-3.7-17-root-32273-gc906036.x86-64 intel_pstate=disable processor.ignore_ppc=1 console=ttyS1,115200n8 crashkernel=320M audit=1 intel_idle.max_cstate=0 rd.plymouth=0 plymouth.enable=0 BOOT_IMAGE=vmlinuz BOOTIF=01-e0-d5-5e-19-0c-29 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000787e8fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000787e9000-0x0000000079ac1fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000079ac2000-0x0000000079f1afff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000079f1b000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0 present. [ 0.000000] DMI: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x4080000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000080000000 mask 3FFF80000000 uncachable [ 0.000000] 1 base 010000000000 mask 3F8000000000 uncachable [ 0.000000] 2 base 013000000000 mask 3FF800000000 write-through [ 0.000000] 3 base 013800000000 mask 3FFC00000000 write-through [ 0.000000] 4 base 013C00000000 mask 3FFFFC000000 write-through [ 0.000000] 5 base 013C04000000 mask 3FFFFE000000 write-through [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0x787e9 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fcfa0-0x000fcfaf] mapped at [ffffffffff200fa0] [ 0.000000] Base memory trampoline at [ffff8f2880094000] 94000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x1a35c73000, 0x1a35c73fff] PGTABLE [ 0.000000] BRK [0x1a35c74000, 0x1a35c74fff] PGTABLE [ 0.000000] BRK [0x1a35c75000, 0x1a35c75fff] PGTABLE [ 0.000000] BRK [0x1a35c76000, 0x1a35c76fff] PGTABLE [ 0.000000] BRK [0x1a35c77000, 0x1a35c77fff] PGTABLE [ 0.000000] BRK [0x1a35c78000, 0x1a35c78fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x7050e000-0x787c7fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f05b0 00024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 0000000079b4a0a0 000C4 (v01 GBT GBTUACPI 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 0000000079b7cb10 0010C (v05 GBT GBTUACPI 01072009 AMI 00010013) [ 0.000000] ACPI: DSDT 0000000079b4a200 3290E (v02 GBT GBTUACPI 01072009 INTL 20091013) [ 0.000000] ACPI: FACS 0000000079f19f80 00040 [ 0.000000] ACPI: APIC 0000000079b7cc20 00454 (v03 GBT GBTUACPI 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 0000000079b7d078 00044 (v01 GBT GBTUACPI 01072009 AMI 00010013) [ 0.000000] ACPI: FIDT 0000000079b7d0c0 0009C (v01 GBT GBTUACPI 01072009 AMI 00010013) [ 0.000000] ACPI: SPMI 0000000079b7d160 00041 (v05 GBT GBTUACPI 00000000 AMI. 00000000) [ 0.000000] ACPI: MCFG 0000000079b7d1a8 0003C (v01 GBT GBTUACPI 01072009 MSFT 00000097) [ 0.000000] ACPI: UEFI 0000000079b7d1e8 00042 (v01 GBT GBTUACPI 01072009 00000000) [ 0.000000] ACPI: HPET 0000000079b7d230 00038 (v01 GBT GBTUACPI 00000001 INTL 20091013) [ 0.000000] ACPI: MSCT 0000000079b7d268 00090 (v01 GBT GBTUACPI 00000001 INTL 20091013) [ 0.000000] ACPI: SLIT 0000000079b7d2f8 00030 (v01 GBT GBTUACPI 00000001 INTL 20091013) [ 0.000000] ACPI: SRAT 0000000079b7d328 01158 (v03 GBT GBTUACPI 00000001 INTL 20091013) [ 0.000000] ACPI: WDDT 0000000079b7e480 00040 (v01 GBT GBTUACPI 00000000 INTL 20091013) [ 0.000000] ACPI: SSDT 0000000079b7e4c0 16C1B (v02 GBT GBTUACPI 00000001 INTL 20120913) [ 0.000000] ACPI: SSDT 0000000079b950e0 0264A (v02 GBT GBTUACPI 00000002 INTL 20120913) [ 0.000000] ACPI: SSDT 0000000079b97730 00064 (v02 GBT GBTUACPI 00000002 INTL 20120913) [ 0.000000] ACPI: PRAD 0000000079b97798 00102 (v02 ALASKA A M I 00000002 INTL 20120913) [ 0.000000] ACPI: HEST 0000000079b978a0 000A8 (v01 GBT GBTUACPI 00000001 INTL 00000001) [ 0.000000] ACPI: BERT 0000000079b97948 00030 (v01 GBT GBTUACPI 00000001 INTL 00000001) [ 0.000000] ACPI: ERST 0000000079b97978 00230 (v01 GBT GBTUACPI 00000001 INTL 00000001) [ 0.000000] ACPI: EINJ 0000000079b97ba8 00130 (v01 GBT GBTUACPI 00000001 INTL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x10 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x12 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x14 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x16 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x20 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x22 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x24 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x26 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x28 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x30 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x32 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x34 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x36 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x40 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x42 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x44 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x46 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x48 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x50 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x52 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x54 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x56 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x60 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x62 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x64 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x66 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x68 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x70 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x72 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x74 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x76 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x11 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x13 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x15 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x17 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x21 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x23 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x25 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x27 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x29 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x31 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x33 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x35 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x37 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x41 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x43 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x45 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x47 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x49 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x51 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x53 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x55 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x57 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x61 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x63 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x65 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x67 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x69 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x71 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x73 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x75 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x77 -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x207fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x2080000000-0x407fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x207fffffff] -> [mem 0x00000000-0x207fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x207ffd9000-0x207fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x407ffd8000-0x407fffefff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 262023MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x407fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009afff] [ 0.000000] node 0: [mem 0x00100000-0x787e8fff] [ 0.000000] node 0: [mem 0x100000000-0x207fffffff] [ 0.000000] node 1: [mem 0x2080000000-0x407fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x207fffffff] [ 0.000000] On node 0 totalpages: 33523587 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 22 pages reserved [ 0.000000] DMA zone: 3994 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 7648 pages used for memmap [ 0.000000] DMA32 zone: 489449 pages, LIFO batch:31 [ 0.000000] Normal zone: 516096 pages used for memmap [ 0.000000] Normal zone: 33030144 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x2080000000-0x407fffffff] [ 0.000000] On node 1 totalpages: 33554432 [ 0.000000] Normal zone: 524288 pages used for memmap [ 0.000000] Normal zone: 33554432 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x26] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x30] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x32] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x34] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x36] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0x40] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0x42] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0x44] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0x46] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0x48] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0x50] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0x52] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0x54] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0x56] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0x60] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0x62] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0x64] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0x66] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0x68] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0x70] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0x72] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0x74] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0x76] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x27] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x31] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x33] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x35] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x37] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0x41] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0x43] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0x45] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0x47] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0x49] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0x51] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0x53] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0x55] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0x57] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0x61] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0x63] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0x65] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0x67] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0x69] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0x71] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0x73] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0x75] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0x77] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x24] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x26] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x28] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x30] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x32] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x34] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x36] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x40] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x42] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x44] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x46] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x48] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x50] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x52] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x54] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x56] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x60] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x62] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x64] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x66] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x68] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x70] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x72] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x74] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x76] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x23] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x25] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x27] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x29] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x31] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x33] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x35] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x37] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x41] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x43] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x45] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x47] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x49] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x51] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x53] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x55] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x57] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x61] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x63] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x65] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x67] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x69] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x71] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x73] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x75] high edge lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x77] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec01000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x03] address[0xfec40000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 72 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009b000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x787e9000-0x79ac1fff] [ 0.000000] PM: Registered nosave memory: [mem 0x79ac2000-0x79f1afff] [ 0.000000] PM: Registered nosave memory: [mem 0x79f1b000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed44fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed45000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:72 nr_cpu_ids:72 nr_node_ids:2 [ 0.000000] percpu: Embedded 38 pages/cpu s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15 [ 0.000000] pcpu-alloc: [0] 16 17 36 37 38 39 40 41 [0] 42 43 44 45 46 47 48 49 [ 0.000000] pcpu-alloc: [0] 50 51 52 53 -- -- -- -- [1] 18 19 20 21 22 23 24 25 [ 0.000000] pcpu-alloc: [1] 26 27 28 29 30 31 32 33 [1] 34 35 54 55 56 57 58 59 [ 0.000000] pcpu-alloc: [1] 60 61 62 63 64 65 66 67 [1] 68 69 70 71 -- -- -- -- [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 66029901 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: initrd=initramfs ip=dhcp root=LABEL=/ netroot=iscsi:192.168.64.82::811:0:iqn.2006-04.gov.llnl:lc.pascal:compute-3.7-17-root-32273-gc906036.x86-64 intel_pstate=disable processor.ignore_ppc=1 console=ttyS1,115200n8 crashkernel=320M audit=1 intel_idle.max_cstate=0 rd.plymouth=0 plymouth.enable=0 BOOT_IMAGE=vmlinuz BOOTIF=01-e0-d5-5e-19-0c-29 [ 0.000000] audit: enabled (after initialization) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 5640876k/270532608k available (7984k kernel code, 2220532k absent, 4752684k reserved, 5759k data, 1976k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=72, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=72. [ 0.000000] NR_IRQS:327936 nr_irqs:1816 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 2147483648 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2100.023 MHz processor [ 0.000049] Calibrating delay loop (skipped), value calculated using timer frequency.. 4200.04 BogoMIPS (lpj=2100023) [ 0.011102] pid_max: default: 73728 minimum: 576 [ 0.016294] Security Framework initialized [ 0.020833] SELinux: Initializing. [ 0.024794] SELinux: Starting in permissive mode [ 0.024795] Yama: becoming mindful. [ 0.048945] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.108257] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.137937] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.145763] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.155389] Initializing cgroup subsys memory [ 0.160199] Initializing cgroup subsys devices [ 0.165079] Initializing cgroup subsys freezer [ 0.169957] Initializing cgroup subsys net_cls [ 0.174836] Initializing cgroup subsys blkio [ 0.179542] Initializing cgroup subsys perf_event [ 0.184683] Initializing cgroup subsys hugetlb [ 0.189560] Initializing cgroup subsys pids [ 0.194179] Initializing cgroup subsys net_prio [ 0.199201] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.205636] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.213876] CPU0: Thermal monitoring enabled (TM1) [ 0.219154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.224893] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 [ 0.230726] tlb_flushall_shift: 6 [ 0.234510] FEATURE SPEC_CTRL Present [ 0.238603] FEATURE IBPB_SUPPORT Present [ 0.242963] Spectre V1 : Mitigation: Load fences, usercopy/swapgs barriers and __user pointer sanitization [ 0.253043] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.259929] Spectre V2 : Mitigation: Full retpoline [ 0.265254] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.275262] TAA: Mitigation: Clear CPU buffers [ 0.280142] MDS: Mitigation: Clear CPU buffers [ 0.286091] Freeing SMP alternatives: 28k freed [ 0.292751] ACPI: Core revision 20130517 [ 0.322453] ACPI: All ACPI Tables successfully acquired [ 0.328519] ftrace: allocating 29710 entries in 117 pages [ 0.359894] IRQ remapping doesn't support X2APIC mode, disable x2apic. [ 0.366923] Switched APIC routing to physical flat. [ 0.372840] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.389274] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (fam: 06, model: 4f, stepping: 01) [ 0.399138] TSC deadline timer enabled [ 0.399200] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 0.409971] ... version: 3 [ 0.414406] ... bit width: 48 [ 0.418929] ... generic registers: 4 [ 0.423368] ... value mask: 0000ffffffffffff [ 0.429104] ... max period: 00007fffffffffff [ 0.434841] ... fixed-purpose events: 3 [ 0.439279] ... event mask: 000000070000000f [ 0.459861] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.446950] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 OK [ 0.580937] smpboot: CPU 18 Converting physical 0 to logical die 1 [ 0.568147] smpboot: Booting Node 1, Processors #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 OK [ 0.776490] smpboot: Booting Node 0, Processors #36 [ 0.785034] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 0.799368] TAA CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html for more details. [ 0.814645] #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 OK [ 0.870493] smpboot: Booting Node 1, Processors #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 OK [ 0.933725] Brought up 72 CPUs [ 0.937209] smpboot: Max logical packages: 2 [ 0.941906] smpboot: Total of 72 processors activated (302585.97 BogoMIPS) [ 1.769658] node 0 initialised, 31975332 pages in 550ms [ 1.775220] node 1 initialised, 32504297 pages in 546ms [ 1.781341] devtmpfs: initialized [ 1.785163] x86/mm: Memory block size: 2048MB [ 1.792539] EVM: security.selinux [ 1.796284] EVM: security.ima [ 1.799679] EVM: security.capability [ 1.803794] PM: Registering ACPI NVS region [mem 0x79ac2000-0x79f1afff] (4558848 bytes) [ 1.813811] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.821113] pinctrl core: initialized pinctrl subsystem [ 1.826828] RTC time: 18:31:26, date: 04/22/22 [ 1.831835] NET: Registered protocol family 16 [ 1.837095] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.845086] ACPI: bus type PCI registered [ 1.849522] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.856494] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.866223] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.873440] PCI: Using configuration type 1 for base access [ 1.888813] ACPI: Added _OSI(Module Device) [ 1.893427] ACPI: Added _OSI(Processor Device) [ 1.898297] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.903425] ACPI: Added _OSI(Processor Aggregator Device) [ 1.909250] ACPI: Added _OSI(Linux-Dell-Video) [ 1.920733] ACPI: EC: Look up EC in DSDT [ 1.939918] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 1.995284] ACPI: Dynamic OEM Table Load: [ 1.999760] ACPI: PRAD (null) 00102 (v02 ALASKA A M I 00000002 INTL 20120913) [ 2.036000] ACPI: Interpreter enabled [ 2.040108] ACPI: (supports S0 S4 S5) [ 2.044204] ACPI: Using IOAPIC for interrupt routing [ 2.049639] HEST: Table parsing has been initialized. [ 2.055116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 2.064696] ACPI: GPE 0x24 active on init [ 2.069175] ACPI: Enabled 5 GPEs in block 00 to 3F [ 2.108856] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff]) [ 2.115203] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.125718] acpi PNP0A03:02: _OSC: platform does not support [SHPCHotplug] [ 2.133695] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 2.142127] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.150763] PCI host bridge to bus 0000:ff [ 2.155290] pci_bus 0000:ff: root bus resource [bus ff] [ 2.160949] pci 0000:ff:08.0: [8086:6f80] type 00 class 0x088000 [ 2.161008] pci 0000:ff:08.2: [8086:6f32] type 00 class 0x110100 [ 2.161061] pci 0000:ff:08.3: [8086:6f83] type 00 class 0x088000 [ 2.161124] pci 0000:ff:09.0: [8086:6f90] type 00 class 0x088000 [ 2.161171] pci 0000:ff:09.2: [8086:6f33] type 00 class 0x110100 [ 2.161219] pci 0000:ff:09.3: [8086:6f93] type 00 class 0x088000 [ 2.161282] pci 0000:ff:0b.0: [8086:6f81] type 00 class 0x088000 [ 2.161327] pci 0000:ff:0b.1: [8086:6f36] type 00 class 0x110100 [ 2.161371] pci 0000:ff:0b.2: [8086:6f37] type 00 class 0x110100 [ 2.161415] pci 0000:ff:0b.3: [8086:6f76] type 00 class 0x088000 [ 2.161462] pci 0000:ff:0c.0: [8086:6fe0] type 00 class 0x088000 [ 2.161506] pci 0000:ff:0c.1: [8086:6fe1] type 00 class 0x088000 [ 2.161551] pci 0000:ff:0c.2: [8086:6fe2] type 00 class 0x088000 [ 2.161595] pci 0000:ff:0c.3: [8086:6fe3] type 00 class 0x088000 [ 2.161639] pci 0000:ff:0c.4: [8086:6fe4] type 00 class 0x088000 [ 2.161685] pci 0000:ff:0c.5: [8086:6fe5] type 00 class 0x088000 [ 2.161729] pci 0000:ff:0c.6: [8086:6fe6] type 00 class 0x088000 [ 2.161773] pci 0000:ff:0c.7: [8086:6fe7] type 00 class 0x088000 [ 2.161819] pci 0000:ff:0d.0: [8086:6fe8] type 00 class 0x088000 [ 2.161863] pci 0000:ff:0d.1: [8086:6fe9] type 00 class 0x088000 [ 2.161908] pci 0000:ff:0d.2: [8086:6fea] type 00 class 0x088000 [ 2.161954] pci 0000:ff:0d.3: [8086:6feb] type 00 class 0x088000 [ 2.161999] pci 0000:ff:0d.4: [8086:6fec] type 00 class 0x088000 [ 2.162044] pci 0000:ff:0d.5: [8086:6fed] type 00 class 0x088000 [ 2.162089] pci 0000:ff:0d.6: [8086:6fee] type 00 class 0x088000 [ 2.162133] pci 0000:ff:0d.7: [8086:6fef] type 00 class 0x088000 [ 2.162178] pci 0000:ff:0e.0: [8086:6ff0] type 00 class 0x088000 [ 2.162222] pci 0000:ff:0e.1: [8086:6ff1] type 00 class 0x088000 [ 2.162270] pci 0000:ff:0f.0: [8086:6ff8] type 00 class 0x088000 [ 2.162313] pci 0000:ff:0f.1: [8086:6ff9] type 00 class 0x088000 [ 2.162358] pci 0000:ff:0f.2: [8086:6ffa] type 00 class 0x088000 [ 2.162402] pci 0000:ff:0f.3: [8086:6ffb] type 00 class 0x088000 [ 2.162446] pci 0000:ff:0f.4: [8086:6ffc] type 00 class 0x088000 [ 2.162491] pci 0000:ff:0f.5: [8086:6ffd] type 00 class 0x088000 [ 2.162536] pci 0000:ff:0f.6: [8086:6ffe] type 00 class 0x088000 [ 2.162582] pci 0000:ff:10.0: [8086:6f1d] type 00 class 0x088000 [ 2.162626] pci 0000:ff:10.1: [8086:6f34] type 00 class 0x110100 [ 2.162673] pci 0000:ff:10.5: [8086:6f1e] type 00 class 0x088000 [ 2.162717] pci 0000:ff:10.6: [8086:6f7d] type 00 class 0x110100 [ 2.162762] pci 0000:ff:10.7: [8086:6f1f] type 00 class 0x088000 [ 2.162807] pci 0000:ff:12.0: [8086:6fa0] type 00 class 0x088000 [ 2.162840] pci 0000:ff:12.1: [8086:6f30] type 00 class 0x110100 [ 2.162889] pci 0000:ff:12.4: [8086:6f60] type 00 class 0x088000 [ 2.162921] pci 0000:ff:12.5: [8086:6f38] type 00 class 0x110100 [ 2.162975] pci 0000:ff:13.0: [8086:6fa8] type 00 class 0x088000 [ 2.163061] pci 0000:ff:13.1: [8086:6f71] type 00 class 0x088000 [ 2.163121] pci 0000:ff:13.2: [8086:6faa] type 00 class 0x088000 [ 2.163180] pci 0000:ff:13.3: [8086:6fab] type 00 class 0x088000 [ 2.163240] pci 0000:ff:13.6: [8086:6fae] type 00 class 0x088000 [ 2.163286] pci 0000:ff:13.7: [8086:6faf] type 00 class 0x088000 [ 2.163337] pci 0000:ff:14.0: [8086:6fb0] type 00 class 0x088000 [ 2.163397] pci 0000:ff:14.1: [8086:6fb1] type 00 class 0x088000 [ 2.163459] pci 0000:ff:14.2: [8086:6fb2] type 00 class 0x088000 [ 2.163519] pci 0000:ff:14.3: [8086:6fb3] type 00 class 0x088000 [ 2.163580] pci 0000:ff:14.4: [8086:6fbc] type 00 class 0x088000 [ 2.163627] pci 0000:ff:14.5: [8086:6fbd] type 00 class 0x088000 [ 2.163673] pci 0000:ff:14.6: [8086:6fbe] type 00 class 0x088000 [ 2.163721] pci 0000:ff:14.7: [8086:6fbf] type 00 class 0x088000 [ 2.163772] pci 0000:ff:16.0: [8086:6f68] type 00 class 0x088000 [ 2.163857] pci 0000:ff:16.1: [8086:6f79] type 00 class 0x088000 [ 2.163941] pci 0000:ff:16.2: [8086:6f6a] type 00 class 0x088000 [ 2.164004] pci 0000:ff:16.3: [8086:6f6b] type 00 class 0x088000 [ 2.164064] pci 0000:ff:16.6: [8086:6f6e] type 00 class 0x088000 [ 2.164111] pci 0000:ff:16.7: [8086:6f6f] type 00 class 0x088000 [ 2.164160] pci 0000:ff:17.0: [8086:6fd0] type 00 class 0x088000 [ 2.164246] pci 0000:ff:17.1: [8086:6fd1] type 00 class 0x088000 [ 2.164308] pci 0000:ff:17.2: [8086:6fd2] type 00 class 0x088000 [ 2.164368] pci 0000:ff:17.3: [8086:6fd3] type 00 class 0x088000 [ 2.164428] pci 0000:ff:17.4: [8086:6fb8] type 00 class 0x088000 [ 2.164476] pci 0000:ff:17.5: [8086:6fb9] type 00 class 0x088000 [ 2.164524] pci 0000:ff:17.6: [8086:6fba] type 00 class 0x088000 [ 2.164572] pci 0000:ff:17.7: [8086:6fbb] type 00 class 0x088000 [ 2.164632] pci 0000:ff:1e.0: [8086:6f98] type 00 class 0x088000 [ 2.164679] pci 0000:ff:1e.1: [8086:6f99] type 00 class 0x088000 [ 2.164726] pci 0000:ff:1e.2: [8086:6f9a] type 00 class 0x088000 [ 2.164773] pci 0000:ff:1e.3: [8086:6fc0] type 00 class 0x088000 [ 2.164807] pci 0000:ff:1e.4: [8086:6f9c] type 00 class 0x088000 [ 2.164860] pci 0000:ff:1f.0: [8086:6f88] type 00 class 0x088000 [ 2.164909] pci 0000:ff:1f.2: [8086:6f8a] type 00 class 0x088000 [ 2.165019] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f]) [ 2.171369] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.242854] acpi PNP0A03:03: _OSC: platform does not support [SHPCHotplug] [ 2.250833] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 2.259262] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.267895] PCI host bridge to bus 0000:7f [ 2.272425] pci_bus 0000:7f: root bus resource [bus 7f] [ 2.278082] pci 0000:7f:08.0: [8086:6f80] type 00 class 0x088000 [ 2.278132] pci 0000:7f:08.2: [8086:6f32] type 00 class 0x110100 [ 2.278177] pci 0000:7f:08.3: [8086:6f83] type 00 class 0x088000 [ 2.278239] pci 0000:7f:09.0: [8086:6f90] type 00 class 0x088000 [ 2.278284] pci 0000:7f:09.2: [8086:6f33] type 00 class 0x110100 [ 2.278330] pci 0000:7f:09.3: [8086:6f93] type 00 class 0x088000 [ 2.278387] pci 0000:7f:0b.0: [8086:6f81] type 00 class 0x088000 [ 2.278429] pci 0000:7f:0b.1: [8086:6f36] type 00 class 0x110100 [ 2.278470] pci 0000:7f:0b.2: [8086:6f37] type 00 class 0x110100 [ 2.278512] pci 0000:7f:0b.3: [8086:6f76] type 00 class 0x088000 [ 2.278555] pci 0000:7f:0c.0: [8086:6fe0] type 00 class 0x088000 [ 2.278597] pci 0000:7f:0c.1: [8086:6fe1] type 00 class 0x088000 [ 2.278639] pci 0000:7f:0c.2: [8086:6fe2] type 00 class 0x088000 [ 2.278680] pci 0000:7f:0c.3: [8086:6fe3] type 00 class 0x088000 [ 2.278722] pci 0000:7f:0c.4: [8086:6fe4] type 00 class 0x088000 [ 2.278763] pci 0000:7f:0c.5: [8086:6fe5] type 00 class 0x088000 [ 2.278804] pci 0000:7f:0c.6: [8086:6fe6] type 00 class 0x088000 [ 2.278845] pci 0000:7f:0c.7: [8086:6fe7] type 00 class 0x088000 [ 2.278887] pci 0000:7f:0d.0: [8086:6fe8] type 00 class 0x088000 [ 2.278932] pci 0000:7f:0d.1: [8086:6fe9] type 00 class 0x088000 [ 2.278974] pci 0000:7f:0d.2: [8086:6fea] type 00 class 0x088000 [ 2.279016] pci 0000:7f:0d.3: [8086:6feb] type 00 class 0x088000 [ 2.279058] pci 0000:7f:0d.4: [8086:6fec] type 00 class 0x088000 [ 2.279104] pci 0000:7f:0d.5: [8086:6fed] type 00 class 0x088000 [ 2.279146] pci 0000:7f:0d.6: [8086:6fee] type 00 class 0x088000 [ 2.279188] pci 0000:7f:0d.7: [8086:6fef] type 00 class 0x088000 [ 2.279231] pci 0000:7f:0e.0: [8086:6ff0] type 00 class 0x088000 [ 2.279275] pci 0000:7f:0e.1: [8086:6ff1] type 00 class 0x088000 [ 2.279322] pci 0000:7f:0f.0: [8086:6ff8] type 00 class 0x088000 [ 2.279365] pci 0000:7f:0f.1: [8086:6ff9] type 00 class 0x088000 [ 2.279408] pci 0000:7f:0f.2: [8086:6ffa] type 00 class 0x088000 [ 2.279450] pci 0000:7f:0f.3: [8086:6ffb] type 00 class 0x088000 [ 2.279493] pci 0000:7f:0f.4: [8086:6ffc] type 00 class 0x088000 [ 2.279536] pci 0000:7f:0f.5: [8086:6ffd] type 00 class 0x088000 [ 2.279578] pci 0000:7f:0f.6: [8086:6ffe] type 00 class 0x088000 [ 2.279623] pci 0000:7f:10.0: [8086:6f1d] type 00 class 0x088000 [ 2.279665] pci 0000:7f:10.1: [8086:6f34] type 00 class 0x110100 [ 2.279710] pci 0000:7f:10.5: [8086:6f1e] type 00 class 0x088000 [ 2.279752] pci 0000:7f:10.6: [8086:6f7d] type 00 class 0x110100 [ 2.279793] pci 0000:7f:10.7: [8086:6f1f] type 00 class 0x088000 [ 2.279836] pci 0000:7f:12.0: [8086:6fa0] type 00 class 0x088000 [ 2.279867] pci 0000:7f:12.1: [8086:6f30] type 00 class 0x110100 [ 2.279913] pci 0000:7f:12.4: [8086:6f60] type 00 class 0x088000 [ 2.279946] pci 0000:7f:12.5: [8086:6f38] type 00 class 0x110100 [ 2.279994] pci 0000:7f:13.0: [8086:6fa8] type 00 class 0x088000 [ 2.280076] pci 0000:7f:13.1: [8086:6f71] type 00 class 0x088000 [ 2.280134] pci 0000:7f:13.2: [8086:6faa] type 00 class 0x088000 [ 2.280192] pci 0000:7f:13.3: [8086:6fab] type 00 class 0x088000 [ 2.280252] pci 0000:7f:13.6: [8086:6fae] type 00 class 0x088000 [ 2.280297] pci 0000:7f:13.7: [8086:6faf] type 00 class 0x088000 [ 2.280344] pci 0000:7f:14.0: [8086:6fb0] type 00 class 0x088000 [ 2.280401] pci 0000:7f:14.1: [8086:6fb1] type 00 class 0x088000 [ 2.280458] pci 0000:7f:14.2: [8086:6fb2] type 00 class 0x088000 [ 2.280515] pci 0000:7f:14.3: [8086:6fb3] type 00 class 0x088000 [ 2.280570] pci 0000:7f:14.4: [8086:6fbc] type 00 class 0x088000 [ 2.280615] pci 0000:7f:14.5: [8086:6fbd] type 00 class 0x088000 [ 2.280660] pci 0000:7f:14.6: [8086:6fbe] type 00 class 0x088000 [ 2.280706] pci 0000:7f:14.7: [8086:6fbf] type 00 class 0x088000 [ 2.280753] pci 0000:7f:16.0: [8086:6f68] type 00 class 0x088000 [ 2.280835] pci 0000:7f:16.1: [8086:6f79] type 00 class 0x088000 [ 2.280917] pci 0000:7f:16.2: [8086:6f6a] type 00 class 0x088000 [ 2.280998] pci 0000:7f:16.3: [8086:6f6b] type 00 class 0x088000 [ 2.281081] pci 0000:7f:16.6: [8086:6f6e] type 00 class 0x088000 [ 2.281126] pci 0000:7f:16.7: [8086:6f6f] type 00 class 0x088000 [ 2.281172] pci 0000:7f:17.0: [8086:6fd0] type 00 class 0x088000 [ 2.281253] pci 0000:7f:17.1: [8086:6fd1] type 00 class 0x088000 [ 2.281310] pci 0000:7f:17.2: [8086:6fd2] type 00 class 0x088000 [ 2.281367] pci 0000:7f:17.3: [8086:6fd3] type 00 class 0x088000 [ 2.281424] pci 0000:7f:17.4: [8086:6fb8] type 00 class 0x088000 [ 2.281470] pci 0000:7f:17.5: [8086:6fb9] type 00 class 0x088000 [ 2.281516] pci 0000:7f:17.6: [8086:6fba] type 00 class 0x088000 [ 2.281562] pci 0000:7f:17.7: [8086:6fbb] type 00 class 0x088000 [ 2.281620] pci 0000:7f:1e.0: [8086:6f98] type 00 class 0x088000 [ 2.281665] pci 0000:7f:1e.1: [8086:6f99] type 00 class 0x088000 [ 2.281709] pci 0000:7f:1e.2: [8086:6f9a] type 00 class 0x088000 [ 2.281754] pci 0000:7f:1e.3: [8086:6fc0] type 00 class 0x088000 [ 2.281787] pci 0000:7f:1e.4: [8086:6f9c] type 00 class 0x088000 [ 2.281838] pci 0000:7f:1f.0: [8086:6f88] type 00 class 0x088000 [ 2.281884] pci 0000:7f:1f.2: [8086:6f8a] type 00 class 0x088000 [ 2.296602] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 2.303212] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.312132] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] [ 2.319903] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 2.328333] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.337177] PCI host bridge to bus 0000:00 [ 2.341707] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 2.348924] pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window] [ 2.356134] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.364037] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window] [ 2.371941] pci_bus 0000:00: root bus resource [mem 0x10000000000-0x13fffffffff window] [ 2.380364] pci_bus 0000:00: root bus resource [bus 00-7e] [ 2.386282] pci 0000:00:00.0: [8086:6f00] type 00 class 0x060000 [ 2.386407] pci 0000:00:01.0: [8086:6f02] type 01 class 0x060400 [ 2.386457] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 2.386536] pci 0000:00:01.0: System wakeup disabled by ACPI [ 2.392657] pci 0000:00:03.0: [8086:6f08] type 01 class 0x060400 [ 2.392707] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 2.392785] pci 0000:00:03.0: System wakeup disabled by ACPI [ 2.398906] pci 0000:00:05.0: [8086:6f28] type 00 class 0x088000 [ 2.399009] pci 0000:00:05.1: [8086:6f29] type 00 class 0x088000 [ 2.399125] pci 0000:00:05.2: [8086:6f2a] type 00 class 0x088000 [ 2.399227] pci 0000:00:05.4: [8086:6f2c] type 00 class 0x080020 [ 2.399235] pci 0000:00:05.4: reg 0x10: [mem 0xc7205000-0xc7205fff] [ 2.399352] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 [ 2.399532] pci 0000:00:14.0: [8086:8d31] type 00 class 0x0c0330 [ 2.399549] pci 0000:00:14.0: reg 0x10: [mem 0x13ffff00000-0x13ffff0ffff 64bit] [ 2.399607] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 2.399671] pci 0000:00:14.0: System wakeup disabled by ACPI [ 2.405796] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 [ 2.405813] pci 0000:00:16.0: reg 0x10: [mem 0xc7204000-0xc720400f 64bit] [ 2.405872] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 2.405956] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 [ 2.405973] pci 0000:00:16.1: reg 0x10: [mem 0xc7203000-0xc720300f 64bit] [ 2.406031] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 2.406125] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 [ 2.406142] pci 0000:00:1a.0: reg 0x10: [mem 0xc7201000-0xc72013ff] [ 2.406220] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 2.406286] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 2.412407] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 [ 2.412477] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 2.412536] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 2.418657] pci 0000:00:1c.2: [8086:8d14] type 01 class 0x060400 [ 2.418725] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold [ 2.418786] pci 0000:00:1c.2: System wakeup disabled by ACPI [ 2.424905] pci 0000:00:1c.4: [8086:8d18] type 01 class 0x060400 [ 2.424973] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold [ 2.425032] pci 0000:00:1c.4: System wakeup disabled by ACPI [ 2.431156] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 [ 2.431173] pci 0000:00:1d.0: reg 0x10: [mem 0xc7200000-0xc72003ff] [ 2.431252] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 2.431321] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 2.437443] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 [ 2.437616] pci 0000:00:1f.3: [8086:8d22] type 00 class 0x0c0500 [ 2.437630] pci 0000:00:1f.3: reg 0x10: [mem 0x13ffff11000-0x13ffff110ff 64bit] [ 2.437650] pci 0000:00:1f.3: reg 0x20: [io 0x0580-0x059f] [ 2.437744] pci 0000:00:1f.6: [8086:8d24] type 00 class 0x118000 [ 2.437762] pci 0000:00:1f.6: reg 0x10: [mem 0x13ffff10000-0x13ffff10fff 64bit] [ 2.438015] acpiphp: Slot [2] registered [ 2.442382] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.447911] pci 0000:02:00.0: [10b5:8796] type 01 class 0x060400 [ 2.447926] pci 0000:02:00.0: reg 0x10: [mem 0xc5100000-0xc513ffff] [ 2.447992] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 2.448069] pci 0000:00:03.0: PCI bridge to [bus 02-08] [ 2.453721] pci 0000:00:03.0: bridge window [mem 0xc3000000-0xc51fffff] [ 2.453726] pci 0000:00:03.0: bridge window [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 2.453770] pci 0000:03:04.0: [10b5:8796] type 01 class 0x060400 [ 2.453851] pci 0000:03:04.0: PME# supported from D0 D3hot D3cold [ 2.453914] pci 0000:03:08.0: [10b5:8796] type 01 class 0x060400 [ 2.453951] pci 0000:03:08.0: Max Payload Size set to 256 (was 128, max 2048) [ 2.461554] pci 0000:03:08.0: PME# supported from D0 D3hot D3cold [ 2.461615] pci 0000:03:0c.0: [10b5:8796] type 01 class 0x060400 [ 2.461653] pci 0000:03:0c.0: Max Payload Size set to 256 (was 128, max 2048) [ 2.469259] pci 0000:03:0c.0: PME# supported from D0 D3hot D3cold [ 2.469322] pci 0000:03:10.0: [10b5:8796] type 01 class 0x060400 [ 2.469403] pci 0000:03:10.0: PME# supported from D0 D3hot D3cold [ 2.469464] pci 0000:03:14.0: [10b5:8796] type 01 class 0x060400 [ 2.469546] pci 0000:03:14.0: PME# supported from D0 D3hot D3cold [ 2.469607] pci 0000:02:00.0: PCI bridge to [bus 03-08] [ 2.475266] pci 0000:02:00.0: bridge window [mem 0xc3000000-0xc50fffff] [ 2.475271] pci 0000:02:00.0: bridge window [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 2.475314] pci 0000:04:00.0: [10de:15f8] type 00 class 0x030200 [ 2.475332] pci 0000:04:00.0: reg 0x10: [mem 0xc4000000-0xc4ffffff] [ 2.475343] pci 0000:04:00.0: reg 0x14: [mem 0x13800000000-0x13bffffffff 64bit pref] [ 2.475354] pci 0000:04:00.0: reg 0x1c: [mem 0x13c00000000-0x13c01ffffff 64bit pref] [ 2.475483] pci 0000:03:04.0: PCI bridge to [bus 04] [ 2.480882] pci 0000:03:04.0: bridge window [mem 0xc4000000-0xc4ffffff] [ 2.480887] pci 0000:03:04.0: bridge window [mem 0x13800000000-0x13c01ffffff 64bit pref] [ 2.480922] pci 0000:03:08.0: PCI bridge to [bus 05] [ 2.486350] pci 0000:03:0c.0: PCI bridge to [bus 06] [ 2.491791] pci 0000:07:00.0: [10de:15f8] type 00 class 0x030200 [ 2.491810] pci 0000:07:00.0: reg 0x10: [mem 0xc3000000-0xc3ffffff] [ 2.491821] pci 0000:07:00.0: reg 0x14: [mem 0x13000000000-0x133ffffffff 64bit pref] [ 2.491832] pci 0000:07:00.0: reg 0x1c: [mem 0x13400000000-0x13401ffffff 64bit pref] [ 2.491967] pci 0000:03:10.0: PCI bridge to [bus 07] [ 2.497366] pci 0000:03:10.0: bridge window [mem 0xc3000000-0xc3ffffff] [ 2.497371] pci 0000:03:10.0: bridge window [mem 0x13000000000-0x13401ffffff 64bit pref] [ 2.497484] pci 0000:08:00.0: [15b3:1017] type 00 class 0x020700 [ 2.497685] pci 0000:08:00.0: reg 0x10: [mem 0x13c04000000-0x13c05ffffff 64bit pref] [ 2.498073] pci 0000:08:00.0: reg 0x30: [mem 0xc5000000-0xc50fffff pref] [ 2.498671] pci 0000:08:00.0: PME# supported from D3cold [ 2.499008] pci 0000:03:14.0: PCI bridge to [bus 08] [ 2.504404] pci 0000:03:14.0: bridge window [mem 0xc5000000-0xc50fffff] [ 2.504409] pci 0000:03:14.0: bridge window [mem 0x13c04000000-0x13c05ffffff 64bit pref] [ 2.504467] pci 0000:00:1c.0: PCI bridge to [bus 09] [ 2.509924] pci 0000:0a:00.0: [1a03:1150] type 01 class 0x060400 [ 2.510056] pci 0000:0a:00.0: supports D1 D2 [ 2.510058] pci 0000:0a:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 2.510132] pci 0000:00:1c.2: PCI bridge to [bus 0a-0b] [ 2.515788] pci 0000:00:1c.2: bridge window [io 0x6000-0x6fff] [ 2.515792] pci 0000:00:1c.2: bridge window [mem 0xc6000000-0xc70fffff] [ 2.515866] pci 0000:0b:00.0: [1a03:2000] type 00 class 0x030000 [ 2.515891] pci 0000:0b:00.0: reg 0x10: [mem 0xc6000000-0xc6ffffff] [ 2.515904] pci 0000:0b:00.0: reg 0x14: [mem 0xc7000000-0xc701ffff] [ 2.515918] pci 0000:0b:00.0: reg 0x18: [io 0x6000-0x607f] [ 2.516022] pci 0000:0b:00.0: supports D1 D2 [ 2.516024] pci 0000:0b:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 2.516131] pci 0000:0a:00.0: PCI bridge to [bus 0b] [ 2.521530] pci 0000:0a:00.0: bridge window [io 0x6000-0x6fff] [ 2.521535] pci 0000:0a:00.0: bridge window [mem 0xc6000000-0xc70fffff] [ 2.521612] pci 0000:0c:00.0: [8086:1521] type 00 class 0x020000 [ 2.521638] pci 0000:0c:00.0: reg 0x10: [mem 0xc7120000-0xc713ffff] [ 2.521659] pci 0000:0c:00.0: reg 0x18: [io 0x5020-0x503f] [ 2.521670] pci 0000:0c:00.0: reg 0x1c: [mem 0xc7144000-0xc7147fff] [ 2.521780] pci 0000:0c:00.0: PME# supported from D0 D3hot D3cold [ 2.521819] pci 0000:0c:00.0: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 2.521822] pci 0000:0c:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs) [ 2.532443] pci 0000:0c:00.0: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 2.532445] pci 0000:0c:00.0: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs) [ 2.543100] pci 0000:0c:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:00:1c.4 (capable of 31.504 Gb/s with 8 GT/s x4 link) [ 2.557386] pci 0000:0c:00.1: [8086:1521] type 00 class 0x020000 [ 2.557411] pci 0000:0c:00.1: reg 0x10: [mem 0xc7100000-0xc711ffff] [ 2.557431] pci 0000:0c:00.1: reg 0x18: [io 0x5000-0x501f] [ 2.557442] pci 0000:0c:00.1: reg 0x1c: [mem 0xc7140000-0xc7143fff] [ 2.557549] pci 0000:0c:00.1: PME# supported from D0 D3hot D3cold [ 2.557585] pci 0000:0c:00.1: reg 0x184: [mem 0x00000000-0x00003fff 64bit pref] [ 2.557588] pci 0000:0c:00.1: VF(n) BAR0 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR0 for 8 VFs) [ 2.568206] pci 0000:0c:00.1: reg 0x190: [mem 0x00000000-0x00003fff 64bit pref] [ 2.568209] pci 0000:0c:00.1: VF(n) BAR3 space: [mem 0x00000000-0x0001ffff 64bit pref] (contains BAR3 for 8 VFs) [ 2.578905] pci 0000:00:1c.4: PCI bridge to [bus 0c] [ 2.584295] pci 0000:00:1c.4: bridge window [io 0x5000-0x5fff] [ 2.584299] pci 0000:00:1c.4: bridge window [mem 0xc7100000-0xc71fffff] [ 2.584305] pci 0000:00:1c.4: bridge has subordinate 0c but max busn 0d [ 2.591361] pci_bus 0000:00: on NUMA node 0 [ 2.592018] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 2.598624] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.607538] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] [ 2.615298] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 2.623727] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.632471] PCI host bridge to bus 0000:80 [ 2.636997] pci_bus 0000:80: root bus resource [io 0x8000-0xffff window] [ 2.644207] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window] [ 2.652110] pci_bus 0000:80: root bus resource [mem 0x14000000000-0x17fffffffff window] [ 2.660535] pci_bus 0000:80: root bus resource [bus 80-fe] [ 2.666453] pci 0000:80:05.0: [8086:6f28] type 00 class 0x088000 [ 2.666544] pci 0000:80:05.1: [8086:6f29] type 00 class 0x088000 [ 2.666648] pci 0000:80:05.2: [8086:6f2a] type 00 class 0x088000 [ 2.666732] pci 0000:80:05.4: [8086:6f2c] type 00 class 0x080020 [ 2.666740] pci 0000:80:05.4: reg 0x10: [mem 0xfbf00000-0xfbf00fff] [ 2.666838] pci_bus 0000:80: on NUMA node 1 [ 2.667081] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15) [ 2.674804] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15) [ 2.682525] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 *11 12 14 15) [ 2.690054] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 *10 11 12 14 15) [ 2.697586] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. [ 2.706469] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. [ 2.715351] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. [ 2.724234] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled. [ 2.733362] vgaarb: device added: PCI:0000:0b:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.741879] vgaarb: loaded [ 2.745021] vgaarb: bridge control possible 0000:0b:00.0 [ 2.750873] SCSI subsystem initialized [ 2.755075] ACPI: bus type USB registered [ 2.759528] usbcore: registered new interface driver usbfs [ 2.765447] usbcore: registered new interface driver hub [ 2.771362] usbcore: registered new device driver usb [ 2.777130] EDAC MC: Ver: 3.0.0 [ 2.781018] PCI: Using ACPI for IRQ routing [ 2.789625] PCI: pci_cache_line_size set to 64 bytes [ 2.789834] e820: reserve RAM buffer [mem 0x0009b000-0x0009ffff] [ 2.789836] e820: reserve RAM buffer [mem 0x787e9000-0x7bffffff] [ 2.789975] NetLabel: Initializing [ 2.793805] NetLabel: domain hash size = 128 [ 2.798588] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.803994] NetLabel: unlabeled traffic allowed by default [ 2.810169] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.816926] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.825227] amd_nb: Cannot enumerate AMD northbridges [ 2.830844] Switched to clocksource hpet [ 2.841064] pnp: PnP ACPI init [ 2.844574] ACPI: bus type PNP registered [ 2.849363] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.849524] system 00:01: [io 0x0500-0x057f] has been reserved [ 2.855881] system 00:01: [io 0x0400-0x047f] could not be reserved [ 2.862578] system 00:01: [io 0x0580-0x059f] has been reserved [ 2.868928] system 00:01: [io 0x0600-0x061f] has been reserved [ 2.875281] system 00:01: [io 0x0880-0x0883] has been reserved [ 2.881634] system 00:01: [io 0x0800-0x081f] has been reserved [ 2.887988] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved [ 2.895033] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved [ 2.902077] system 00:01: [mem 0xff000000-0xffffffff] has been reserved [ 2.909114] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved [ 2.916161] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved [ 2.923208] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved [ 2.930251] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved [ 2.937291] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.937385] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.937561] pnp 00:03: [dma 0 disabled] [ 2.937619] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.937778] pnp 00:04: [dma 0 disabled] [ 2.937844] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.938007] system 00:05: [io 0x0a00-0x0a1f] has been reserved [ 2.944354] system 00:05: [io 0x0a20-0x0a2f] has been reserved [ 2.950704] system 00:05: [io 0x0a30-0x0a3f] has been reserved [ 2.957059] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.957679] pnp: PnP ACPI: found 6 devices [ 2.962205] ACPI: bus type PNP unregistered [ 2.973175] pci 0000:00:1c.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000 [ 2.973179] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 09] add_size 200000 add_align 100000 [ 2.973182] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff] to [bus 09] add_size 200000 add_align 100000 [ 2.973186] pci 0000:00:1c.4: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 0c] add_size 100000 add_align 100000 [ 2.973193] pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000 [ 2.973195] pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000 [ 2.973198] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000 [ 2.973200] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000 [ 2.973203] pci 0000:00:1c.4: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 100000 min_align 100000 [ 2.973205] pci 0000:00:1c.4: res[15]=[mem 0x00100000-0x001fffff 64bit pref] res_to_dev_res add_size 100000 min_align 100000 [ 2.973207] pci 0000:00:1c.0: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000 [ 2.973210] pci 0000:00:1c.0: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000 [ 2.973215] pci 0000:00:1c.0: BAR 14: assigned [mem 0x90000000-0x901fffff] [ 2.980522] pci 0000:00:1c.0: BAR 15: assigned [mem 0x10000000000-0x100001fffff 64bit pref] [ 2.989297] pci 0000:00:1c.4: BAR 15: assigned [mem 0x10000200000-0x100002fffff 64bit pref] [ 2.998076] pci 0000:00:1c.0: BAR 13: assigned [io 0x1000-0x1fff] [ 3.004690] pci 0000:00:01.0: PCI bridge to [bus 01] [ 3.010093] pci 0000:03:04.0: PCI bridge to [bus 04] [ 3.015489] pci 0000:03:04.0: bridge window [mem 0xc4000000-0xc4ffffff] [ 3.022706] pci 0000:03:04.0: bridge window [mem 0x13800000000-0x13c01ffffff 64bit pref] [ 3.031401] pci 0000:03:08.0: PCI bridge to [bus 05] [ 3.036808] pci 0000:03:0c.0: PCI bridge to [bus 06] [ 3.042213] pci 0000:03:10.0: PCI bridge to [bus 07] [ 3.047606] pci 0000:03:10.0: bridge window [mem 0xc3000000-0xc3ffffff] [ 3.054824] pci 0000:03:10.0: bridge window [mem 0x13000000000-0x13401ffffff 64bit pref] [ 3.063518] pci 0000:03:14.0: PCI bridge to [bus 08] [ 3.068919] pci 0000:03:14.0: bridge window [mem 0xc5000000-0xc50fffff] [ 3.076136] pci 0000:03:14.0: bridge window [mem 0x13c04000000-0x13c05ffffff 64bit pref] [ 3.084831] pci 0000:02:00.0: PCI bridge to [bus 03-08] [ 3.090489] pci 0000:02:00.0: bridge window [mem 0xc3000000-0xc50fffff] [ 3.097707] pci 0000:02:00.0: bridge window [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 3.106400] pci 0000:00:03.0: PCI bridge to [bus 02-08] [ 3.112059] pci 0000:00:03.0: bridge window [mem 0xc3000000-0xc51fffff] [ 3.119277] pci 0000:00:03.0: bridge window [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 3.127970] pci 0000:00:1c.0: PCI bridge to [bus 09] [ 3.133368] pci 0000:00:1c.0: bridge window [io 0x1000-0x1fff] [ 3.139894] pci 0000:00:1c.0: bridge window [mem 0x90000000-0x901fffff] [ 3.147115] pci 0000:00:1c.0: bridge window [mem 0x10000000000-0x100001fffff 64bit pref] [ 3.155811] pci 0000:0a:00.0: PCI bridge to [bus 0b] [ 3.161206] pci 0000:0a:00.0: bridge window [io 0x6000-0x6fff] [ 3.167733] pci 0000:0a:00.0: bridge window [mem 0xc6000000-0xc70fffff] [ 3.174957] pci 0000:00:1c.2: PCI bridge to [bus 0a-0b] [ 3.180617] pci 0000:00:1c.2: bridge window [io 0x6000-0x6fff] [ 3.187145] pci 0000:00:1c.2: bridge window [mem 0xc6000000-0xc70fffff] [ 3.194370] pci 0000:0c:00.0: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 20000 min_align 0 [ 3.194372] pci 0000:0c:00.0: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 20000 min_align 0 [ 3.194375] pci 0000:0c:00.1: res[7]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 20000 min_align 0 [ 3.194377] pci 0000:0c:00.1: res[10]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 20000 min_align 0 [ 3.194380] pci 0000:0c:00.0: BAR 7: assigned [mem 0x10000200000-0x1000021ffff 64bit pref] [ 3.203082] pci 0000:0c:00.0: BAR 10: assigned [mem 0x10000220000-0x1000023ffff 64bit pref] [ 3.211868] pci 0000:0c:00.1: BAR 7: assigned [mem 0x10000240000-0x1000025ffff 64bit pref] [ 3.220571] pci 0000:0c:00.1: BAR 10: assigned [mem 0x10000260000-0x1000027ffff 64bit pref] [ 3.229358] pci 0000:00:1c.4: PCI bridge to [bus 0c] [ 3.234756] pci 0000:00:1c.4: bridge window [io 0x5000-0x5fff] [ 3.241285] pci 0000:00:1c.4: bridge window [mem 0xc7100000-0xc71fffff] [ 3.248503] pci 0000:00:1c.4: bridge window [mem 0x10000200000-0x100002fffff 64bit pref] [ 3.257198] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 3.257200] pci_bus 0000:00: resource 5 [io 0x1000-0x7fff window] [ 3.257203] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 3.257204] pci_bus 0000:00: resource 7 [mem 0x90000000-0xc7ffbfff window] [ 3.257206] pci_bus 0000:00: resource 8 [mem 0x10000000000-0x13fffffffff window] [ 3.257209] pci_bus 0000:02: resource 1 [mem 0xc3000000-0xc51fffff] [ 3.257211] pci_bus 0000:02: resource 2 [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 3.257213] pci_bus 0000:03: resource 1 [mem 0xc3000000-0xc50fffff] [ 3.257214] pci_bus 0000:03: resource 2 [mem 0x13000000000-0x13c05ffffff 64bit pref] [ 3.257217] pci_bus 0000:04: resource 1 [mem 0xc4000000-0xc4ffffff] [ 3.257219] pci_bus 0000:04: resource 2 [mem 0x13800000000-0x13c01ffffff 64bit pref] [ 3.257221] pci_bus 0000:07: resource 1 [mem 0xc3000000-0xc3ffffff] [ 3.257223] pci_bus 0000:07: resource 2 [mem 0x13000000000-0x13401ffffff 64bit pref] [ 3.257225] pci_bus 0000:08: resource 1 [mem 0xc5000000-0xc50fffff] [ 3.257227] pci_bus 0000:08: resource 2 [mem 0x13c04000000-0x13c05ffffff 64bit pref] [ 3.257229] pci_bus 0000:09: resource 0 [io 0x1000-0x1fff] [ 3.257231] pci_bus 0000:09: resource 1 [mem 0x90000000-0x901fffff] [ 3.257233] pci_bus 0000:09: resource 2 [mem 0x10000000000-0x100001fffff 64bit pref] [ 3.257235] pci_bus 0000:0a: resource 0 [io 0x6000-0x6fff] [ 3.257237] pci_bus 0000:0a: resource 1 [mem 0xc6000000-0xc70fffff] [ 3.257238] pci_bus 0000:0b: resource 0 [io 0x6000-0x6fff] [ 3.257240] pci_bus 0000:0b: resource 1 [mem 0xc6000000-0xc70fffff] [ 3.257242] pci_bus 0000:0c: resource 0 [io 0x5000-0x5fff] [ 3.257244] pci_bus 0000:0c: resource 1 [mem 0xc7100000-0xc71fffff] [ 3.257246] pci_bus 0000:0c: resource 2 [mem 0x10000200000-0x100002fffff 64bit pref] [ 3.257250] pci_bus 0000:80: resource 4 [io 0x8000-0xffff window] [ 3.257252] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window] [ 3.257254] pci_bus 0000:80: resource 6 [mem 0x14000000000-0x17fffffffff window] [ 3.257328] NET: Registered protocol family 2 [ 3.262949] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 3.271595] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 3.278904] TCP: Hash tables configured (established 524288 bind 65536) [ 3.286006] TCP: reno registered [ 3.289821] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.296975] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.304701] NET: Registered protocol family 1 [ 3.341945] PCI: CLS mismatch (64 != 32), using 64 bytes [ 3.341971] pci 0000:0b:00.0: Boot video device [ 3.342024] Unpacking initramfs... [ 4.482274] Freeing initrd memory: 133864k freed [ 4.531623] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.538501] software IO TLB [mem 0x6c50e000-0x7050e000] (64MB) mapped at [ffff8f28ec50e000-ffff8f28f050dfff] [ 4.548967] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer [ 4.557310] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules [ 4.563480] RAPL PMU: hw unit of domain package 2^-14 Joules [ 4.569563] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 4.583176] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 4.589760] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 4.597757] futex hash table entries: 32768 (order: 9, 2097152 bytes) [ 4.604995] Initialise system trusted keyring [ 4.609825] audit: initializing netlink socket (enabled) [ 4.615578] type=2000 audit(1650652285.885:1): initialized [ 4.646303] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.653086] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.661272] zpool: loaded [ 4.664325] zbud: loaded [ 4.667666] VFS: Disk quotas dquot_6.5.2 [ 4.672083] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.679288] Key type big_key registered [ 4.683558] SELinux: Registering netfilter hooks [ 4.685154] NET: Registered protocol family 38 [ 4.690039] Key type asymmetric registered [ 4.694573] Asymmetric key parser 'x509' registered [ 4.699918] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 4.707883] io scheduler noop registered [ 4.712245] io scheduler deadline registered (default) [ 4.717850] io scheduler cfq registered [ 4.722122] io scheduler mq-deadline registered [ 4.727088] io scheduler kyber registered [ 4.732092] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 4.732294] pcieport 0000:00:03.0: irq 27 for MSI/MSI-X [ 4.732461] pcieport 0000:00:1c.0: irq 28 for MSI/MSI-X [ 4.732631] pcieport 0000:00:1c.2: irq 29 for MSI/MSI-X [ 4.732778] pcieport 0000:00:1c.4: irq 30 for MSI/MSI-X [ 4.732909] pcieport 0000:02:00.0: irq 31 for MSI/MSI-X [ 4.733049] pcieport 0000:03:04.0: irq 32 for MSI/MSI-X [ 4.733188] pcieport 0000:03:08.0: irq 33 for MSI/MSI-X [ 4.733336] pcieport 0000:03:0c.0: irq 34 for MSI/MSI-X [ 4.733481] pcieport 0000:03:10.0: irq 35 for MSI/MSI-X [ 4.733631] pcieport 0000:03:14.0: irq 36 for MSI/MSI-X [ 4.733815] aer 0000:00:01.0:pcie002: service driver aer loaded [ 4.733858] aer 0000:00:03.0:pcie002: service driver aer loaded [ 4.733878] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 4.741267] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded [ 4.741279] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 4.748668] pcieport 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 4.756057] pcieport 0000:03:04.0: Signaling PME through PCIe PME interrupt [ 4.763443] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 4.770400] pcieport 0000:03:08.0: Signaling PME through PCIe PME interrupt [ 4.777784] pcieport 0000:03:0c.0: Signaling PME through PCIe PME interrupt [ 4.785177] pcieport 0000:03:10.0: Signaling PME through PCIe PME interrupt [ 4.792570] pci 0000:07:00.0: Signaling PME through PCIe PME interrupt [ 4.799528] pcieport 0000:03:14.0: Signaling PME through PCIe PME interrupt [ 4.806911] pci 0000:08:00.0: Signaling PME through PCIe PME interrupt [ 4.813864] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded [ 4.813882] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 4.821274] pcie_pme 0000:00:1c.0:pcie001: service driver pcie_pme loaded [ 4.821292] pcieport 0000:00:1c.2: Signaling PME through PCIe PME interrupt [ 4.828683] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt [ 4.835642] pci 0000:0b:00.0: Signaling PME through PCIe PME interrupt [ 4.842604] pcie_pme 0000:00:1c.2:pcie001: service driver pcie_pme loaded [ 4.842620] pcieport 0000:00:1c.4: Signaling PME through PCIe PME interrupt [ 4.850010] pci 0000:0c:00.0: Signaling PME through PCIe PME interrupt [ 4.856971] pci 0000:0c:00.1: Signaling PME through PCIe PME interrupt [ 4.863930] pcie_pme 0000:00:1c.4:pcie001: service driver pcie_pme loaded [ 4.864002] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.870021] pciehp 0000:00:1c.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ LLActRep+ [ 4.882809] pciehp 0000:00:1c.0:pcie004: service driver pciehp loaded [ 4.882816] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.889930] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.897156] intel_idle: disabled [ 4.897447] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 [ 4.906053] ACPI: Power Button [PWRB] [ 4.910183] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 4.918008] ACPI: Power Button [PWRF] [ 4.922148] ACPI: Requesting acpi_cpufreq [ 4.927182] Monitor-Mwait will be used to enter C-1 state [ 4.927197] Monitor-Mwait will be used to enter C-2 state [ 4.927209] ACPI: acpi_idle registered with cpuidle [ 4.944331] ERST: Error Record Serialization Table (ERST) support is initialized. [ 4.952244] pstore: Registered erst as persistent store backend [ 4.959058] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 4.966967] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.994315] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 5.020994] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 5.027506] Non-volatile memory driver v1.3 [ 5.032152] Linux agpgart interface v0.103 [ 5.037040] crash memory driver: version 1.1 [ 5.042311] rdac: device handler registered [ 5.046967] hp_sw: device handler registered [ 5.051674] emc: device handler registered [ 5.056348] alua: device handler registered [ 5.061044] libphy: Fixed MDIO Bus: probed [ 5.065611] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.072570] ehci-pci: EHCI PCI platform driver [ 5.077847] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 5.083591] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 5.091428] ehci-pci 0000:00:1a.0: debug port 2 [ 5.100291] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 5.100306] ehci-pci 0000:00:1a.0: irq 18, io mem 0xc7201000 [ 5.111798] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 5.118021] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.126716] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.134368] usb usb1: Product: EHCI Host Controller [ 5.139681] usb usb1: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 ehci_hcd [ 5.147940] usb usb1: SerialNumber: 0000:00:1a.0 [ 5.153102] hub 1-0:1.0: USB hub found [ 5.157288] hub 1-0:1.0: 2 ports detected [ 5.161985] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 5.167687] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 5.175515] ehci-pci 0000:00:1d.0: debug port 2 [ 5.184393] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 5.184398] ehci-pci 0000:00:1d.0: irq 18, io mem 0xc7200000 [ 5.195799] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 5.202009] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.210702] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.218354] usb usb2: Product: EHCI Host Controller [ 5.223666] usb usb2: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 ehci_hcd [ 5.231924] usb usb2: SerialNumber: 0000:00:1d.0 [ 5.237078] hub 2-0:1.0: USB hub found [ 5.241264] hub 2-0:1.0: 2 ports detected [ 5.245863] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.252481] ohci-pci: OHCI PCI platform driver [ 5.257406] uhci_hcd: USB Universal Host Controller Interface driver [ 5.264411] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 5.270118] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 [ 5.279022] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810 [ 5.288582] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported [ 5.288605] xhci_hcd 0000:00:14.0: irq 37 for MSI/MSI-X [ 5.288741] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 3.10 [ 5.297426] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.305080] usb usb3: Product: xHCI Host Controller [ 5.310392] usb usb3: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 xhci-hcd [ 5.318648] usb usb3: SerialNumber: 0000:00:14.0 [ 5.323805] hub 3-0:1.0: USB hub found [ 5.327997] hub 3-0:1.0: 15 ports detected [ 5.333562] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 5.339262] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 [ 5.347089] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed [ 5.353913] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 3.10 [ 5.362605] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 5.370257] usb usb4: Product: xHCI Host Controller [ 5.375562] usb usb4: Manufacturer: Linux 3.10.0-1160.53.1.1chaos.ch6.x86_64 xhci-hcd [ 5.383819] usb usb4: SerialNumber: 0000:00:14.0 [ 5.388979] hub 4-0:1.0: USB hub found [ 5.393181] hub 4-0:1.0: 6 ports detected [ 5.400338] usbcore: registered new interface driver usbserial_generic [ 5.407306] usbserial: USB Serial support registered for generic [ 5.413783] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 5.579796] tsc: Refined TSC clocksource calibration: 2099.998 MHz [ 5.580799] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 6.457635] i8042: No controller found [ 6.461864] Switched to clocksource tsc [ 6.461900] mousedev: PS/2 mouse device common for all mice [ 6.462049] rtc_cmos 00:00: RTC can wake from S4 [ 6.462184] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 6.462242] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 6.462283] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 6.463220] cpuidle: using governor menu [ 6.463501] hidraw: raw HID events driver (C) Jiri Kosina [ 6.463599] usbcore: registered new interface driver usbhid [ 6.463600] usbhid: USB HID core driver [ 6.463780] drop_monitor: Initializing network drop monitor service [ 6.463860] Netfilter messages via NETLINK v0.30. [ 6.463912] TCP: cubic registered [ 6.463915] Initializing XFRM netlink socket [ 6.464072] NET: Registered protocol family 10 [ 6.464583] NET: Registered protocol family 17 [ 6.464588] mpls_gso: MPLS GSO support [ 6.473401] intel_rdt: Intel RDT L3 allocation detected [ 6.473401] intel_rdt: Intel RDT L3DATA allocation detected [ 6.473402] intel_rdt: Intel RDT L3CODE allocation detected [ 6.473402] intel_rdt: Intel RDT L3 monitoring detected [ 6.473405] mce: Using 22 MCE banks [ 6.473457] microcode: sig=0x406f1, pf=0x1, revision=0xb00003e [ 6.477406] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 6.477514] PM: Hibernation image not present or could not be loaded. [ 6.477517] Loading compiled-in X.509 certificates [ 6.477949] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' [ 6.478356] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' [ 6.478758] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: a1775f904d8c469b10aaacf6a86d40e66f3447a1' [ 6.478779] registered taskstats version 1 [ 6.478797] page_owner is disabled [ 6.480873] Key type trusted registered [ 6.482741] Key type encrypted registered [ 6.482769] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 6.483223] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 6.483303] Magic number: 2:30:545 [ 6.483429] acpi device:131: hash matches [ 6.484162] rtc_cmos 00:00: setting system clock to 2022-04-22 18:31:31 UTC (1650652291) [ 6.512826] usb 3-10: new high-speed USB device number 2 using xhci_hcd [ 6.525172] usb 1-1: New USB device found, idVendor=8087, idProduct=800a, bcdDevice= 0.05 [ 6.525176] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 6.525467] hub 1-1:1.0: USB hub found [ 6.525541] hub 1-1:1.0: 6 ports detected [ 6.586162] usb 2-1: New USB device found, idVendor=8087, idProduct=8002, bcdDevice= 0.05 [ 6.586163] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 6.586306] hub 2-1:1.0: USB hub found [ 6.586410] hub 2-1:1.0: 8 ports detected [ 6.587660] random: fast init done [ 6.636414] usb 3-10: New USB device found, idVendor=0624, idProduct=0248, bcdDevice= 0.00 [ 6.636416] usb 3-10: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 6.636417] usb 3-10: Product: Gadget USB HUB [ 6.636418] usb 3-10: Manufacturer: no manufacturer [ 6.636419] usb 3-10: SerialNumber: 0123456789 [ 6.636802] hub 3-10:1.0: USB hub found [ 6.636879] hub 3-10:1.0: 5 ports detected [ 6.788341] Freeing unused kernel memory: 1976k freed [ 6.794475] Write protecting the kernel read-only data: 12288k [ 6.801479] Freeing unused kernel memory: 196k freed [ 6.807954] Freeing unused kernel memory: 524k freed [ 6.818826] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 6.837343] systemd[1]: Detected architecture x86-64. [ 6.842824] systemd[1]: Running in initial RAM disk. [ 6.857843] systemd[1]: No hostname configured. [ 6.862825] systemd[1]: Set hostname to . [ 6.868248] systemd[1]: Initializing machine ID from random generator. [ 6.913012] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ 6.923649] systemd[1]: Reached target Local Encrypted Volumes. [ 6.925832] usb 3-10.1: new high-speed USB device number 3 using xhci_hcd [ 6.942920] systemd[1]: Reached target Local File Systems. [ 6.954892] systemd[1]: Reached target Swap. [ 6.963889] systemd[1]: Reached target Timers. [ 6.973140] systemd[1]: Created slice Root Slice. [ 6.983030] systemd[1]: Created slice System Slice. [ 6.992882] systemd[1]: Listening on udev Control Socket. [ 7.005002] systemd[1]: Listening on Journal Socket. [ 7.011381] usb 3-10.1: New USB device found, idVendor=0624, idProduct=0249, bcdDevice= 0.00 [ 7.020749] usb 3-10.1: New USB device strings: Mfr=4, Product=5, SerialNumber=6 [ 7.029960] usb 3-10.1: Product: Keyboard/Mouse Function [ 7.035732] usb 3-10.1: Manufacturer: Avocent [ 7.041866] usb 3-10.1: SerialNumber: 20121018 [ 7.047402] systemd[1]: Starting Journal Service... [ 7.047949] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:14.0/usb3/3-10/3-10.1/3-10.1:1.0/input/input2 [ 7.069344] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 7.088629] systemd[1]: Starting Load Kernel Modules... [ 7.097604] fuse init (API version 7.23) [ 7.102912] hid-generic 0003:0624:0249.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse Function] on usb-0000:00:14.0-10.1/input0 [ 7.103308] systemd[1]: Starting Setup Virtual Console... [ 7.123022] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:14.0/usb3/3-10/3-10.1/3-10.1:1.1/input/input3 [ 7.135988] hid-generic 0003:0624:0249.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:14.0-10.1/input1 [ 7.151264] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:14.0/usb3/3-10/3-10.1/3-10.1:1.2/input/input4 [ 7.163767] hid-generic 0003:0624:0249.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:14.0-10.1/input2 [ 7.164291] systemd[1]: Starting dracut cmdline hook... [ 7.186837] systemd[1]: Reached target Slices. [ 7.196830] systemd[1]: Listening on udev Kernel Socket. [ 7.208389] systemd[1]: Started Journal Service. [ 7.218974] type=1130 audit(1650652292.234:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.247812] type=1130 audit(1650652292.263:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.272861] type=1130 audit(1650652292.288:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.293081] Loading iSCSI transport class v2.0-870. [ 7.303839] type=1130 audit(1650652292.319:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.325985] iscsi: registered transport (tcp) [ 7.347818] type=1130 audit(1650652292.363:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.374808] type=1130 audit(1650652292.390:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.400830] type=1130 audit(1650652292.416:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.443048] device-mapper: uevent: version 1.0.3 [ 7.448194] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7.500450] RPC: Registered named UNIX socket transport module. [ 7.506805] RPC: Registered udp transport module. [ 7.511944] RPC: Registered tcp transport module. [ 7.517082] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 7.604879] type=1130 audit(1650652292.620:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.637905] type=1130 audit(1650652292.653:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 7.819589] pps_core: LinuxPPS API ver. 1 registered [ 7.826032] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 7.836520] cryptd: max_cpu_qlen set to 1000 [ 7.843586] PTP clock support registered [ 7.851343] dca service started, version 1.12.1 [ 7.867020] AVX2 version of gcm_enc/dec engaged. [ 7.872797] AES CTR mode by8 optimization enabled [ 7.879870] nvidia: loading out-of-tree module taints kernel. [ 7.879874] nvidia: loading out-of-tree module taints kernel. [ 7.879881] nvidia: module license 'NVIDIA' taints kernel. [ 7.879882] Disabling lock debugging due to kernel taint [ 7.881996] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 7.882041] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 7.920257] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k [ 7.928357] igb: Copyright (c) 2007-2014 Intel Corporation. [ 7.936237] igb 0000:0c:00.0: irq 38 for MSI/MSI-X [ 7.936287] igb 0000:0c:00.0: irq 38 for MSI/MSI-X [ 7.936297] igb 0000:0c:00.0: irq 39 for MSI/MSI-X [ 7.936306] igb 0000:0c:00.0: irq 40 for MSI/MSI-X [ 7.936315] igb 0000:0c:00.0: irq 41 for MSI/MSI-X [ 7.936324] igb 0000:0c:00.0: irq 42 for MSI/MSI-X [ 7.936331] igb 0000:0c:00.0: irq 43 for MSI/MSI-X [ 7.936339] igb 0000:0c:00.0: irq 44 for MSI/MSI-X [ 7.936346] igb 0000:0c:00.0: irq 45 for MSI/MSI-X [ 7.936354] igb 0000:0c:00.0: irq 46 for MSI/MSI-X [ 7.946342] mlx5_core 0000:08:00.0: firmware version: 16.29.2002 [ 7.952819] mlx5_core 0000:08:00.0: 126.016 Gb/s available PCIe bandwidth (8 GT/s x16 link) [ 7.970937] [drm] Using P2A bridge for configuration [ 7.977791] [drm] AST 2400 detected [ 7.981745] [drm] Analog VGA only [ 7.985582] [drm] dram MCLK=408 Mhz type=6 bus_width=16 size=01000000 [ 7.991734] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 8.002387] [TTM] Zone kernel: Available graphics memory: 131847990 kiB [ 8.008890] igb 0000:0c:00.0: added PHC on eth0 [ 8.008892] igb 0000:0c:00.0: Intel(R) Gigabit Ethernet Network Connection [ 8.008893] igb 0000:0c:00.0: eth0: (PCIe:5.0Gb/s:Width x4) e0:d5:5e:19:0c:29 [ 8.008969] igb 0000:0c:00.0: eth0: PBA No: 140422-008 [ 8.008970] igb 0000:0c:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 8.009396] igb 0000:0c:00.1: irq 47 for MSI/MSI-X [ 8.009456] igb 0000:0c:00.1: irq 47 for MSI/MSI-X [ 8.009464] igb 0000:0c:00.1: irq 48 for MSI/MSI-X [ 8.009472] igb 0000:0c:00.1: irq 49 for MSI/MSI-X [ 8.009479] igb 0000:0c:00.1: irq 50 for MSI/MSI-X [ 8.009489] igb 0000:0c:00.1: irq 51 for MSI/MSI-X [ 8.009497] igb 0000:0c:00.1: irq 52 for MSI/MSI-X [ 8.009505] igb 0000:0c:00.1: irq 53 for MSI/MSI-X [ 8.009512] igb 0000:0c:00.1: irq 54 for MSI/MSI-X [ 8.009520] igb 0000:0c:00.1: irq 55 for MSI/MSI-X [ 8.042916] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 8.051262] [TTM] Initializing pool allocator [ 8.057460] [TTM] Initializing DMA pool allocator [ 8.069434] fbcon: astdrmfb (fb0) is primary device [ 8.069898] 8021q: 802.1Q VLAN Support v1.8 [ 8.072916] igb 0000:0c:00.1: added PHC on eth1 [ 8.072917] igb 0000:0c:00.1: Intel(R) Gigabit Ethernet Network Connection [ 8.072920] igb 0000:0c:00.1: eth1: (PCIe:5.0Gb/s:Width x4) e0:d5:5e:19:0c:2a [ 8.073004] igb 0000:0c:00.1: eth1: PBA No: 140422-008 [ 8.073005] igb 0000:0c:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 8.081015] nvidia-nvlink: Nvlink Core is being initialized, major device number 241 [ 8.105102] iscsi: registered transport (qla4xxx) [ 8.105140] QLogic iSCSI HBA Driver [ 8.111430] libcxgbi:libcxgbi_init_module: Chelsio iSCSI driver library libcxgbi v0.9.1-ko (Apr. 2015) [ 8.114380] Console: switching to colour frame buffer device 128x48 [ 8.125819] Chelsio T3 iSCSI Driver cxgb3i v2.0.1-ko (Apr. 2015) [ 8.125835] iscsi: registered transport (cxgb3i) [ 8.156748] ast 0000:0b:00.0: fb0: astdrmfb frame buffer device [ 8.232534] Chelsio T4-T6 iSCSI Driver cxgb4i v0.9.5-ko (Apr. 2015) [ 8.239246] iscsi: registered transport (cxgb4i) [ 8.244629] mlx5_core 0000:08:00.0: irq 56 for MSI/MSI-X [ 8.244640] mlx5_core 0000:08:00.0: irq 57 for MSI/MSI-X [ 8.244654] mlx5_core 0000:08:00.0: irq 58 for MSI/MSI-X [ 8.244664] mlx5_core 0000:08:00.0: irq 59 for MSI/MSI-X [ 8.244679] mlx5_core 0000:08:00.0: irq 60 for MSI/MSI-X [ 8.244688] mlx5_core 0000:08:00.0: irq 61 for MSI/MSI-X [ 8.244696] mlx5_core 0000:08:00.0: irq 62 for MSI/MSI-X [ 8.244720] mlx5_core 0000:08:00.0: irq 63 for MSI/MSI-X [ 8.244735] mlx5_core 0000:08:00.0: irq 64 for MSI/MSI-X [ 8.244749] mlx5_core 0000:08:00.0: irq 65 for MSI/MSI-X [ 8.244757] mlx5_core 0000:08:00.0: irq 66 for MSI/MSI-X [ 8.244783] mlx5_core 0000:08:00.0: irq 67 for MSI/MSI-X [ 8.244799] mlx5_core 0000:08:00.0: irq 68 for MSI/MSI-X [ 8.244808] mlx5_core 0000:08:00.0: irq 69 for MSI/MSI-X [ 8.244816] mlx5_core 0000:08:00.0: irq 70 for MSI/MSI-X [ 8.244825] mlx5_core 0000:08:00.0: irq 71 for MSI/MSI-X [ 8.244833] mlx5_core 0000:08:00.0: irq 72 for MSI/MSI-X [ 8.244841] mlx5_core 0000:08:00.0: irq 73 for MSI/MSI-X [ 8.244850] mlx5_core 0000:08:00.0: irq 74 for MSI/MSI-X [ 8.244859] mlx5_core 0000:08:00.0: irq 75 for MSI/MSI-X [ 8.244868] mlx5_core 0000:08:00.0: irq 76 for MSI/MSI-X [ 8.244876] mlx5_core 0000:08:00.0: irq 77 for MSI/MSI-X [ 8.244885] mlx5_core 0000:08:00.0: irq 78 for MSI/MSI-X [ 8.244893] mlx5_core 0000:08:00.0: irq 79 for MSI/MSI-X [ 8.244903] mlx5_core 0000:08:00.0: irq 80 for MSI/MSI-X [ 8.244911] mlx5_core 0000:08:00.0: irq 81 for MSI/MSI-X [ 8.244920] mlx5_core 0000:08:00.0: irq 82 for MSI/MSI-X [ 8.244929] mlx5_core 0000:08:00.0: irq 83 for MSI/MSI-X [ 8.244947] mlx5_core 0000:08:00.0: irq 84 for MSI/MSI-X [ 8.244955] mlx5_core 0000:08:00.0: irq 85 for MSI/MSI-X [ 8.244964] mlx5_core 0000:08:00.0: irq 86 for MSI/MSI-X [ 8.244973] mlx5_core 0000:08:00.0: irq 87 for MSI/MSI-X [ 8.244982] mlx5_core 0000:08:00.0: irq 88 for MSI/MSI-X [ 8.244991] mlx5_core 0000:08:00.0: irq 89 for MSI/MSI-X [ 8.244999] mlx5_core 0000:08:00.0: irq 90 for MSI/MSI-X [ 8.245009] mlx5_core 0000:08:00.0: irq 91 for MSI/MSI-X [ 8.245018] mlx5_core 0000:08:00.0: irq 92 for MSI/MSI-X [ 8.245028] mlx5_core 0000:08:00.0: irq 93 for MSI/MSI-X [ 8.245037] mlx5_core 0000:08:00.0: irq 94 for MSI/MSI-X [ 8.245046] mlx5_core 0000:08:00.0: irq 95 for MSI/MSI-X [ 8.245054] mlx5_core 0000:08:00.0: irq 96 for MSI/MSI-X [ 8.245079] mlx5_core 0000:08:00.0: irq 97 for MSI/MSI-X [ 8.245087] mlx5_core 0000:08:00.0: irq 98 for MSI/MSI-X [ 8.245112] mlx5_core 0000:08:00.0: irq 99 for MSI/MSI-X [ 8.245129] mlx5_core 0000:08:00.0: irq 100 for MSI/MSI-X [ 8.245137] mlx5_core 0000:08:00.0: irq 101 for MSI/MSI-X [ 8.245145] mlx5_core 0000:08:00.0: irq 102 for MSI/MSI-X [ 8.245152] mlx5_core 0000:08:00.0: irq 103 for MSI/MSI-X [ 8.245160] mlx5_core 0000:08:00.0: irq 104 for MSI/MSI-X [ 8.245168] mlx5_core 0000:08:00.0: irq 105 for MSI/MSI-X [ 8.245176] mlx5_core 0000:08:00.0: irq 106 for MSI/MSI-X [ 8.245185] mlx5_core 0000:08:00.0: irq 107 for MSI/MSI-X [ 8.245193] mlx5_core 0000:08:00.0: irq 108 for MSI/MSI-X [ 8.245201] mlx5_core 0000:08:00.0: irq 109 for MSI/MSI-X [ 8.245208] mlx5_core 0000:08:00.0: irq 110 for MSI/MSI-X [ 8.245216] mlx5_core 0000:08:00.0: irq 111 for MSI/MSI-X [ 8.245224] mlx5_core 0000:08:00.0: irq 112 for MSI/MSI-X [ 8.245232] mlx5_core 0000:08:00.0: irq 113 for MSI/MSI-X [ 8.245239] mlx5_core 0000:08:00.0: irq 114 for MSI/MSI-X [ 8.245248] mlx5_core 0000:08:00.0: irq 115 for MSI/MSI-X [ 8.245264] mlx5_core 0000:08:00.0: irq 116 for MSI/MSI-X [ 8.245272] mlx5_core 0000:08:00.0: irq 117 for MSI/MSI-X [ 8.245280] mlx5_core 0000:08:00.0: irq 118 for MSI/MSI-X [ 8.245287] mlx5_core 0000:08:00.0: irq 119 for MSI/MSI-X [ 8.245870] [drm] Initialized ast 0.1.0 20120228 for 0000:0b:00.0 on minor 0 [ 8.257912] cnic: QLogic cnicDriver v2.5.22 (July 20, 2015) [ 8.268728] QLogic NetXtreme II iSCSI Driver bnx2i v2.7.10.1 (Jul 16, 2014) [ 8.276132] iscsi: registered transport (bnx2i) [ 8.292067] iscsi: registered transport (be2iscsi) [ 8.297287] In beiscsi_module_init, tt=ffffffffc0694120 [ 8.308885] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 510.39.01 Fri Dec 31 11:03:22 UTC 2021 [ 8.318651] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0 [ 8.347786] nvidia-uvm: Loaded the UVM driver, major device number 237. [ 8.393605] IPv6: ADDRCONF(NETDEV_UP): enp12s0f0: link is not ready [ 8.400308] 8021q: adding VLAN 0 to HW filter on device enp12s0f0 [ 8.574851] random: crng init done [ 11.840138] igb 0000:0c:00.0 enp12s0f0: igb: enp12s0f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 11.850641] IPv6: ADDRCONF(NETDEV_CHANGE): enp12s0f0: link becomes ready [ 21.124717] audit_printk_skb: 13 callbacks suppressed [ 21.130205] type=1131 audit(1650652306.140:15): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 21.158828] type=1130 audit(1650652306.174:16): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 23.195738] scsi host0: iSCSI Initiator over TCP/IP [ 23.210710] scsi 0:0:0:0: Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5 [ 23.231670] scsi 0:0:0:0: alua: supports implicit and explicit TPGS [ 23.238373] scsi 0:0:0:0: alua: device naa.600140549f9c1bec4d835f7f70072314 port group 0 rel port 1 [ 23.247841] scsi 0:0:0:0: alua: Attached [ 23.252741] scsi 0:0:0:0: alua: transition timeout set to 60 seconds [ 23.259531] scsi 0:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA [ 23.273328] sd 0:0:0:0: [sda] 73203712 512-byte logical blocks: (37.4 GB/34.9 GiB) [ 23.283335] sd 0:0:0:0: [sda] Write Protect is on [ 23.288751] sd 0:0:0:0: [sda] Mode Sense: 43 00 80 08 [ 23.288821] type=1130 audit(1650652308.304:17): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=iscsistart_192.168.64.82::811:0:iqn.2006\x2d04.gov.llnl:lc.pascal:compute\x2d3.7\x2d17\x2droot\x2d32273\x2dgc906036.x86\x2d64 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 23.318156] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 23.328025] sd 0:0:0:0: [sda] Optimal transfer size 4194304 bytes [ 23.337402] sd 0:0:0:0: [sda] Attached SCSI disk [ 23.393696] type=1130 audit(1650652308.409:18): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 24.228655] type=1130 audit(1650652309.244:19): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 24.271656] type=1130 audit(1650652309.287:20): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 24.355663] EXT4-fs (sda): mounted filesystem without journal. Opts: (null) [ 24.444617] type=1130 audit(1650652309.460:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 24.464327] type=1131 audit(1650652309.479:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-parse-etc comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 24.519099] TECH PREVIEW: Overlay filesystem may not be fully supported. Please review provided documentation for limitations. [ 24.550245] FS-Cache: Loaded [ 24.581715] FS-Cache: Netfs 'nfs' registered for caching [ 24.590337] Key type dns_resolver registered [ 24.621979] NFS: Registering the id_resolver key type [ 24.627470] Key type id_resolver registered [ 24.632084] Key type id_legacy registered [ 31.624051] type=1305 audit(1650652316.639:23): audit_backlog_limit=320 old=64 auid=4294967295 ses=4294967295 subj=kernel res=1 [ 31.640538] type=1305 audit(1650652316.656:24): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 31.659511] type=1305 audit(1650652316.675:25): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 31.676510] type=1305 audit(1650652316.692:26): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 31.693533] type=1305 audit(1650652316.709:27): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 31.710534] type=1305 audit(1650652316.726:28): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=1 [ 31.722836] type=1305 audit(1650652316.738:29): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=0 [ 31.735148] type=1305 audit(1650652316.750:30): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="shared-fs" list=4 res=0 [ 31.751511] type=1305 audit(1650652316.767:31): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="tmpdir" list=4 res=1 [ 31.768532] type=1305 audit(1650652316.784:32): auid=4294967295 ses=4294967295 subj=kernel op=add_rule key="tmpdir" list=4 res=1 [ 53.703508] audit_printk_skb: 5913 callbacks suppressed [ 53.709165] type=1300 audit(1650652338.718:465): arch=c000003e syscall=59 success=yes exit=0 a0=7f2c2c7dc009 a1=7fffe0dd2890 a2=7fffe0dd3108 a3=8 items=2 ppid=2328 pid=2335 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sh" exe="/usr/bin/bash" subj=kernel key="root_commands" [ 53.738828] type=1309 audit(1650652338.718:465): argc=3 a0="sh" a1="-c" a2=726D202D66202F746D702F75666F72746966792E6C6F636B [ 53.750381] type=1307 audit(1650652338.718:465): cwd="/admin/scripts/atlassian" [ 53.758208] type=1302 audit(1650652338.718:465): item=0 name="/bin/sh" inode=77269 dev=00:26 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 53.778772] type=1302 audit(1650652338.718:465): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=77234 dev=00:26 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 53.801070] type=1327 audit(1650652338.718:465): proctitle=7368002D6300726D202D66202F746D702F75666F72746966792E6C6F636B [ 53.813559] type=1300 audit(1650652338.829:466): arch=c000003e syscall=59 success=yes exit=0 a0=23ad160 a1=23adbc0 a2=23ad6b0 a3=7ffdaf194ae0 items=2 ppid=2328 pid=2335 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rm" exe="/usr/bin/rm" subj=kernel key="root_commands" [ 53.842708] type=1309 audit(1650652338.829:466): argc=3 a0="rm" a1="-f" a2="/tmp/ufortify.lock" [ 53.851831] type=1307 audit(1650652338.829:466): cwd="/admin/scripts/atlassian" [ 53.859657] type=1302 audit(1650652338.829:466): item=0 name="/usr/bin/rm" inode=172136 dev=00:26 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 [ 54.837163] systemd-journald[528]: Received SIGTERM from PID 1 (systemd). [ 54.904903] SELinux: Disabled at runtime. [ 54.909460] SELinux: Unregistering netfilter hooks [ 54.974382] ip_tables: (C) 2000-2006 Netfilter Core Team [ 54.981273] systemd[1]: Inserted module 'ip_tables' [ 55.766001] systemd-journald[2489]: Received request to flush runtime journal from PID 1 [ 56.835947] nvidia 0000:04:00.0: irq 120 for MSI/MSI-X [ 56.844872] IPMI message handler: version 39.2 [ 56.853227] ipmi device interface [ 56.865689] ipmi_si: IPMI System Interface driver [ 56.870845] ipmi_si dmi-ipmi-si.0: probing via SMBIOS [ 56.876329] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 56.883977] ipmi_si: Adding SMBIOS-specified kcs state machine [ 56.890262] ipmi_si IPI0001:00: probing via ACPI [ 56.895327] ipmi_si IPI0001:00: [io 0x0ca2] regsize 1 spacing 1 irq 0 [ 56.902283] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI [ 56.915449] power_meter ACPI000D:00: Found ACPI power meter. [ 56.921758] power_meter ACPI000D:00: Ignoring unsafe software power cap! [ 57.325228] ipmi_si: Adding ACPI-specified kcs state machine [ 57.331404] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 57.377943] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [ 57.459213] ipmi_si IPI0001:00: Found new BMC (man_id: 0x003c0a, prod_id: 0x0118, dev_id: 0x20) [ 57.513330] pmd_set_huge: Cannot satisfy [mem 0x13c00000000-0x13c00200000] with a huge-page mapping due to MTRR override. [ 57.818310] ipmi_si IPI0001:00: IPMI kcs interface initialized [ 57.902781] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt [ 57.906718] nvidia 0000:07:00.0: irq 121 for MSI/MSI-X [ 57.939425] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 57.946695] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 58.044499] intel_rapl: Found RAPL domain package [ 58.049650] intel_rapl: Found RAPL domain dram [ 58.054529] intel_rapl: DRAM domain energy unit 15300pj [ 58.060205] intel_rapl: Found RAPL domain package [ 58.065345] intel_rapl: Found RAPL domain dram [ 58.070218] intel_rapl: DRAM domain energy unit 15300pj [ 58.094794] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 58.100466] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 58.106139] EDAC sbridge: Seeking for: PCI ID 8086:6fa0 [ 58.111812] EDAC sbridge: Seeking for: PCI ID 8086:6f60 [ 58.117470] EDAC sbridge: Seeking for: PCI ID 8086:6f60 [ 58.123132] EDAC sbridge: Seeking for: PCI ID 8086:6f60 [ 58.128806] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 58.134464] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 58.140126] EDAC sbridge: Seeking for: PCI ID 8086:6fa8 [ 58.145800] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 58.151460] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 58.157146] EDAC sbridge: Seeking for: PCI ID 8086:6f71 [ 58.162822] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 58.168501] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 58.174193] EDAC sbridge: Seeking for: PCI ID 8086:6faa [ 58.179850] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 58.185509] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 58.191206] EDAC sbridge: Seeking for: PCI ID 8086:6fab [ 58.196862] EDAC sbridge: Seeking for: PCI ID 8086:6fac [ 58.202547] EDAC sbridge: Seeking for: PCI ID 8086:6fad [ 58.208234] EDAC sbridge: Seeking for: PCI ID 8086:6f68 [ 58.213893] EDAC sbridge: Seeking for: PCI ID 8086:6f68 [ 58.219554] EDAC sbridge: Seeking for: PCI ID 8086:6f68 [ 58.225211] EDAC sbridge: Seeking for: PCI ID 8086:6f79 [ 58.230870] EDAC sbridge: Seeking for: PCI ID 8086:6f79 [ 58.236530] EDAC sbridge: Seeking for: PCI ID 8086:6f79 [ 58.242194] EDAC sbridge: Seeking for: PCI ID 8086:6f6a [ 58.247855] EDAC sbridge: Seeking for: PCI ID 8086:6f6a [ 58.253515] EDAC sbridge: Seeking for: PCI ID 8086:6f6a [ 58.259179] EDAC sbridge: Seeking for: PCI ID 8086:6f6b [ 58.259187] EDAC sbridge: Seeking for: PCI ID 8086:6f6b [ 58.259196] EDAC sbridge: Seeking for: PCI ID 8086:6f6b [ 58.259202] EDAC sbridge: Seeking for: PCI ID 8086:6f6c [ 58.259219] EDAC sbridge: Seeking for: PCI ID 8086:6f6d [ 58.259236] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 58.259242] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 58.259252] EDAC sbridge: Seeking for: PCI ID 8086:6ffc [ 58.259260] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 58.259266] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 58.259275] EDAC sbridge: Seeking for: PCI ID 8086:6ffd [ 58.259283] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 58.259290] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 58.259299] EDAC sbridge: Seeking for: PCI ID 8086:6faf [ 58.260300] EDAC MC0: Giving out device to 'sb_edac.c' 'Broadwell SrcID#1_Ha#0': DEV 0000:ff:12.0 [ 58.260416] EDAC MC1: Giving out device to 'sb_edac.c' 'Broadwell SrcID#0_Ha#0': DEV 0000:7f:12.0 [ 58.260519] EDAC MC2: Giving out device to 'sb_edac.c' 'Broadwell SrcID#1_Ha#1': DEV 0000:ff:12.4 [ 58.260619] EDAC MC3: Giving out device to 'sb_edac.c' 'Broadwell SrcID#0_Ha#1': DEV 0000:7f:12.4 [ 58.260619] EDAC sbridge: Ver: 1.1.2 [ 58.796181] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 510.39.01 Fri Dec 31 10:52:52 UTC 2021 [ 58.823071] NVRM: Persistence mode is deprecated and will be removed in a future release. Please use nvidia-persistenced instead. [ 58.854929] iscsi: registered transport (iser) [ 58.866673] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver [ 58.874288] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 1 [ 58.884115] [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver [ 58.891671] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:07:00.0 on minor 2 [ 58.902707] RPC: Registered rdma transport module. [ 58.909277] RPC: Registered rdma backchannel transport module. [ 59.557435] iTCO_vendor_support: vendor-support=0 [ 59.565100] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 59.571171] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 63.200149] gdrdrv:device registered with major number 230 [ 64.993144] ghes_read_estatus: 8866 callbacks suppressed [ 64.998899] GHES: ghes_read_estatus: 0, 0x7999a018 0 [ 88.659482] IPv6: ADDRCONF(NETDEV_CHANGE): lo: link becomes ready [ 89.151386] hsi0: enabling connected mode will cause multicast packet drops [ 89.158827] hsi0: mtu > 4092 will cause multicast packet drops. [ 89.182979] IPv6: ADDRCONF(NETDEV_UP): hsi0: link is not ready [ 89.192336] IPv6: ADDRCONF(NETDEV_CHANGE): hsi0: link becomes ready [ 153.821497] LNet: HW NUMA nodes: 2, HW CPU cores: 72, npartitions: 2 [ 153.830428] alg: No test for adler32 (adler32-zlib) [ 154.626465] LNet: 16043:0:(config.c:1641:lnet_inet_enumerate()) lnet: Ignoring interface enp12s0f1: it's down [ 154.637043] LNet: Using FastReg for registration [ 154.714888] LNet: Added LNI 192.168.128.128@o2ib35 [8/256/0/180] [ 778.394110] Lustre: Lustre: Build Version: 2.12.8_6.llnl [ 779.711384] Lustre: Mounted aspls2-client [ 780.359081] Lustre: Mounted aspls3-client [ 780.363532] Lustre: Skipped 2 previous similar messages [ 2069.746279] Lustre: 16806:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654248/real 1650654248] req@ffff8f484df2a880 x1730835187226048/t0(0) o400->lsh-OST0019-osc-ffff8f687d1f4800@172.19.3.42@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654354 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2069.746282] Lustre: 16807:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654248/real 1650654248] req@ffff8f48490d9f80 x1730835187219200/t0(0) o400->aspls2-OST000c-osc-ffff8f686c694000@172.19.3.205@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654354 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2069.746297] Lustre: aspls2-OST000c-osc-ffff8f686c694000: Connection to aspls2-OST000c (at 172.19.3.205@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 2069.824186] Lustre: 16806:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 2085.849866] Lustre: 16814:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654265/real 1650654265] req@ffff8f4849c1ec00 x1730835187228864/t0(0) o17->aspls2-OST000a-osc-ffff8f686c694000@172.19.3.203@o2ib600:28/4 lens 456/432 e 0 to 1 dl 1650654371 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 2085.849874] Lustre: aspls2-OST0014-osc-ffff8f686c694000: Connection to aspls2-OST0014 (at 172.19.3.213@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 2085.849876] Lustre: Skipped 36 previous similar messages [ 2085.903584] Lustre: 16814:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [ 2093.929633] Lustre: 16806:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654273/real 1650654273] req@ffff8f48490e3f00 x1730835187237568/t0(0) o400->lsh-OST0011-osc-ffff8f687d1f4800@172.19.3.34@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654379 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2093.929669] Lustre: lsh-OST0012-osc-ffff8f687d1f4800: Connection to lsh-OST0012 (at 172.19.3.35@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 2093.929672] Lustre: Skipped 8 previous similar messages [ 2093.982491] Lustre: 16806:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 54 previous similar messages [ 2118.944952] Lustre: 16826:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654298/real 1650654298] req@ffff8f484911f080 x1730835187248064/t0(0) o400->lsh-OST0019-osc-ffff8f687d1f4800@172.19.3.42@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654404 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2118.974705] Lustre: 16826:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 53 previous similar messages [ 2143.944295] Lustre: 16812:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654323/real 1650654323] req@ffff8f484dd3e780 x1730835187258688/t0(0) o400->lsh-OST0023-osc-ffff8f687d1f4800@172.19.3.51@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2143.974049] Lustre: 16812:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 53 previous similar messages [ 2168.943648] Lustre: 16816:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654348/real 1650654348] req@ffff8f484e960d80 x1730835187268160/t0(0) o400->lsh-OST001b-osc-ffff8f687d1f4800@172.19.3.44@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2168.973407] Lustre: 16816:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 44 previous similar messages [ 2191.943081] Lustre: 16811:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654371/real 1650654371] req@ffff8f484914cc80 x1730835187307840/t0(0) o400->lsh-OST0023-osc-ffff8f687d1f4800@172.19.3.51@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2191.972840] Lustre: 16811:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 83 previous similar messages [ 2224.942308] Lustre: 16801:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650654355/real 1650654355] req@ffff8f484ebc1200 x1730835187285824/t0(0) o400->lsh-OST0010-osc-ffff8f687d1f4800@172.19.3.34@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650654510 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2224.972072] Lustre: 16801:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 2417.208417] Lustre: aspls2-OST0016-osc-ffff8f686c694000: Connection restored to 172.19.3.215@o2ib600 (at 172.19.3.215@o2ib600) [ 2430.049554] Lustre: lsh-OST0017-osc-ffff8f687d1f4800: Connection restored to 172.19.3.40@o2ib600 (at 172.19.3.40@o2ib600) [ 2432.520987] Lustre: lsh-OST0007-osc-ffff8f687d1f4800: Connection restored to 172.19.3.24@o2ib600 (at 172.19.3.24@o2ib600) [ 2432.532376] Lustre: Skipped 1 previous similar message [ 2434.829398] Lustre: lsh-OST0005-osc-ffff8f687d1f4800: Connection restored to 172.19.3.22@o2ib600 (at 172.19.3.22@o2ib600) [ 2434.840797] Lustre: Skipped 5 previous similar messages [ 2435.131300] Lustre: Evicted from lsh-OST0010_UUID (at 172.19.3.33@o2ib600) after server handle changed from 0xce53bce990fc4c36 to 0xce53bce99112f775 [ 2435.145030] LustreError: 167-0: lsh-OST0010-osc-ffff8f687d1f4800: This client was evicted by lsh-OST0010; in progress operations using this service will fail. [ 2436.057606] Lustre: Evicted from aspls2-OST0004_UUID (at 172.19.3.197@o2ib600) after server handle changed from 0x6f184c7a599e7b18 to 0x6f184c7a59ba96bd [ 2436.071677] LustreError: 167-0: aspls2-OST0004-osc-ffff8f686c694000: This client was evicted by aspls2-OST0004; in progress operations using this service will fail. [ 2447.999188] Lustre: lsh-MDT0003-mdc-ffff8f687d1f4800: Connection restored to 172.19.3.4@o2ib600 (at 172.19.3.4@o2ib600) [ 2448.010408] Lustre: Skipped 51 previous similar messages [ 5131.431799] Lustre: 16834:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650657310/real 1650657310] req@ffff8f67f2bbd580 x1730835189353024/t0(0) o400->lsh-OST0023-osc-ffff8f687d1f4800@172.19.3.52@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650657416 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5131.431819] Lustre: lsh-OST0007-osc-ffff8f687d1f4800: Connection to lsh-OST0007 (at 172.19.3.24@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5131.431821] Lustre: Skipped 10 previous similar messages [ 5131.484740] Lustre: 16834:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 61 previous similar messages [ 5132.494818] Lustre: lsh-OST0021-osc-ffff8f687d1f4800: Connection to lsh-OST0021 (at 172.19.3.50@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5132.512276] Lustre: Skipped 47 previous similar messages [ 5156.521185] Lustre: 16853:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650657335/real 1650657335] req@ffff8f67f18dd100 x1730835189361088/t0(0) o400->lsh-OST0005-osc-ffff8f687d1f4800@172.19.3.22@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650657441 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5156.521215] Lustre: lsh-OST000e-osc-ffff8f687d1f4800: Connection to lsh-OST000e (at 172.19.3.31@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5156.521216] Lustre: Skipped 1 previous similar message [ 5156.521234] LustreError: 166-1: MGC172.19.3.1@o2ib600: Connection to MGS (at 172.19.3.1@o2ib600) was lost; in progress operations using this service will fail [ 5156.588525] Lustre: 16853:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 55 previous similar messages [ 5182.521531] Lustre: 16840:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650657360/real 1650657360] req@ffff8f67f0acb180 x1730835189371776/t0(0) o400->lsh-OST0010-osc-ffff8f687d1f4800@172.19.3.34@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650657466 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5182.521546] Lustre: lsh-OST0014-osc-ffff8f687d1f4800: Connection to lsh-OST0014 (at 172.19.3.37@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5182.521548] Lustre: Skipped 8 previous similar messages [ 5182.574378] Lustre: 16840:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 53 previous similar messages [ 5206.536048] Lustre: aspls2-MDT0002-mdc-ffff8f686c694000: Connection to aspls2-MDT0002 (at 172.19.3.183@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5206.554700] Lustre: aspls2-MDT0002-mdc-ffff8f686c694000: Connection restored to 172.19.3.183@o2ib600 (at 172.19.3.183@o2ib600) [ 5217.557738] Lustre: 16850:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650657385/real 1650657385] req@ffff8f67f1e44800 x1730835189381632/t0(0) o400->lsh-OST000e-osc-ffff8f687d1f4800@172.19.3.31@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650657502 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5217.587501] Lustre: 16850:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 50 previous similar messages [ 5282.566185] Lustre: 16846:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650657410/real 1650657410] req@ffff8f67f08d5100 x1730835189392896/t0(0) o400->lsh-OST0022-osc-ffff8f687d1f4800@172.19.3.52@o2ib600:28/4 lens 224/224 e 0 to 1 dl 1650657567 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5282.595939] Lustre: 16846:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 137 previous similar messages [ 5338.076049] Lustre: lsh-OST0014-osc-ffff8f687d1f4800: Connection restored to 172.19.3.37@o2ib600 (at 172.19.3.37@o2ib600) [ 5340.114977] Lustre: lsh-OST0004-osc-ffff8f687d1f4800: Connection restored to 172.19.3.21@o2ib600 (at 172.19.3.21@o2ib600) [ 5340.126361] Lustre: Skipped 17 previous similar messages [ 5341.993941] LustreError: 167-0: lsh-OST0022-osc-ffff8f687d1f4800: This client was evicted by lsh-OST0022; in progress operations using this service will fail. [ 5342.008534] LustreError: Skipped 1 previous similar message [ 5344.120139] Lustre: aspls2-OST000f-osc-ffff8f686c694000: Connection restored to 172.19.3.208@o2ib600 (at 172.19.3.208@o2ib600) [ 5344.131959] Lustre: Skipped 33 previous similar messages [ 5959.187704] Lustre: 16857:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650658138/real 1650658138] req@ffff8f67f13d4380 x1730835190629568/t0(0) o400->lsh-MDT0001-mdc-ffff8f687d1f4800@172.19.3.2@o2ib600:12/10 lens 224/224 e 0 to 1 dl 1650658244 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5959.217464] Lustre: 16857:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 5959.227728] Lustre: lsh-MDT0001-mdc-ffff8f687d1f4800: Connection to lsh-MDT0001 (at 172.19.3.2@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 5960.392015] Lustre: lsh-MDT0001-mdc-ffff8f687d1f4800: Connection restored to 172.19.3.2@o2ib600 (at 172.19.3.2@o2ib600) [ 5960.403235] Lustre: Skipped 10 previous similar messages [ 6034.369885] Lustre: lsh-OST000d-osc-ffff8f687d1f4800: Connection to lsh-OST000d (at 172.19.3.30@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 6034.369887] Lustre: lsh-OST0004-osc-ffff8f687d1f4800: Connection to lsh-OST0004 (at 172.19.3.21@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 6034.404781] Lustre: Skipped 56 previous similar messages [ 6050.425499] Lustre: lsh-OST000e-osc-ffff8f687d1f4800: Connection to lsh-OST000e (at 172.19.3.31@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 6498.593908] Lustre: aspls2-OST0016-osc-ffff8f686c694000: Connection restored to 172.19.3.215@o2ib600 (at 172.19.3.215@o2ib600) [ 7416.875840] bdev: Dropped 220.888M of page cache [ 7416.985893] sda: Dropped 749.588M of page cache [ 7417.003957] 0:49: Dropped 12K of page cache [ 7417.008584] 0:52: Dropped 8K of page cache [ 7417.014752] 0:53: Dropped 11.244M of page cache [ 7417.025401] lustre: Dropped 776K of page cache [ 7418.404562] epilog.real (3755): drop_caches: 3 [ 7419.675715] epilog.real (3509): drop_caches: 2 [ 7905.452898] bdev: Dropped 26.040M of page cache [ 7905.493451] sda: Dropped 311.760M of page cache [ 7905.503532] 0:49: Dropped 12K of page cache [ 7905.508167] 0:52: Dropped 8K of page cache [ 7905.514120] 0:53: Dropped 11.244M of page cache [ 7905.754798] lustre: Dropped 42.664M of page cache [ 7959.036921] epilog.real (7546): drop_caches: 3 [ 7975.173547] epilog.real (7392): drop_caches: 2 [ 8025.753950] Lustre: 16809:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650660205/real 1650660222] req@ffff8f47266ade80 x1730835605588992/t0(0) o103->aspls2-OST000c-osc-ffff8f686c694000@172.19.3.205@o2ib600:17/18 lens 328/224 e 0 to 1 dl 1650660311 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 8025.754956] Lustre: lsh-OST000f-osc-ffff8f687d1f4800: Connection to lsh-OST000f (at 172.19.3.32@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 8025.754957] Lustre: Skipped 1 previous similar message [ 8025.807062] Lustre: 16809:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 793 previous similar messages [ 8026.095995] Lustre: lsh-OST0001-osc-ffff8f687d1f4800: Connection restored to 172.19.3.18@o2ib600 (at 172.19.3.18@o2ib600) [ 8026.107381] Lustre: Skipped 60 previous similar messages [ 8027.439825] Lustre: lsh-OST0022-osc-ffff8f687d1f4800: Connection to lsh-OST0022 (at 172.19.3.51@o2ib600) was lost; in progress operations using this service will wait for recovery to complete [ 8027.457279] Lustre: Skipped 59 previous similar messages [ 8033.423917] sched: RT throttling activated [ 8044.117938] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4302700885, last ping 4302700864, now 4302710885 [ 8044.130293] connection1:0: detected conn error (1022) [ 8052.750725] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_00_34:16832] [ 63.206068] gdrdrv:dbg traces disabled, info traces disabled [ 8052.759159] Modules linked in: [ 8052.762662] mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic nvidia_uvm(OE) [ 8052.833258] mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.884356] CPU: 1 PID: 16832 Comm: ptlrpcd_00_34 Kdump: loaded Tainted: P OE ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.897457] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.906314] task: ffff8f484f838000 ti: ffff8f484f834000 task.ti: ffff8f484f834000 [ 8052.914219] RIP: 0010:[] [] lnet_res_lh_lookup+0x48/0x70 [lnet] [ 8052.923805] RSP: 0018:ffff8f484f837ba0 EFLAGS: 00000206 [ 8052.926721] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [ptlrpcd_01_12:16847] [ 8052.926769] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.926805] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.926810] CPU: 18 PID: 16847 Comm: ptlrpcd_01_12 Kdump: loaded Tainted: P OE ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.926811] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.926812] task: ffff8f484fb98000 ti: ffff8f484fb94000 task.ti: ffff8f484fb94000 [ 8052.926821] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.926823] RSP: 0018:ffff8f484fb97b58 EFLAGS: 00000246 [ 8052.926824] RAX: 0000000000000000 RBX: ffff8f6601635580 RCX: 0000000000910000 [ 8052.926824] RDX: ffff8f487f61b8c0 RSI: 0000000000410001 RDI: ffff8f686e2b6b40 [ 8052.926825] RBP: ffff8f484fb97b58 R08: ffff8f687ec1b8c0 R09: 0000000000000000 [ 8052.926826] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.926827] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e233c8b2 [ 8052.926828] FS: 0000000000000000(0000) GS:ffff8f687ec00000(0000) knlGS:0000000000000000 [ 8052.926829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.926830] CR2: 00002aaaac3f92a0 CR3: 0000003ff8218000 CR4: 00000000003607e0 [ 8052.926832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.926832] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.926833] Call Trace: [ 8052.926841] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.926846] [] _raw_spin_lock+0x30/0x40 [ 8052.926861] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.926880] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.926941] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.926977] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.927008] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.927059] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.927064] [] ? wake_up_state+0x20/0x20 [ 8052.927115] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.927119] [] kthread+0xd1/0xe0 [ 8052.927121] [] ? insert_kthread_work+0x40/0x40 [ 8052.927124] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.927126] [] ? insert_kthread_work+0x40/0x40 [ 8052.927147] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.932719] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [ptlrpcd_01_31:16866] [ 8052.932749] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.932772] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.932775] CPU: 19 PID: 16866 Comm: ptlrpcd_01_31 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.932776] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.932777] task: ffff8f484fbed280 ti: ffff8f484d288000 task.ti: ffff8f484d288000 [ 8052.932781] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8052.932783] RSP: 0018:ffff8f484d28bb58 EFLAGS: 00000246 [ 8052.932784] RAX: 0000000000000000 RBX: ffff8f68429b9680 RCX: 0000000000990000 [ 8052.932784] RDX: ffff8f687eedb8c0 RSI: 0000000000e90001 RDI: ffff8f686e2b6b40 [ 8052.932785] RBP: ffff8f484d28bb58 R08: ffff8f687ec5b8c0 R09: 0000000000000000 [ 8052.932786] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.932787] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000c64e37e3 [ 8052.932788] FS: 0000000000000000(0000) GS:ffff8f687ec40000(0000) knlGS:0000000000000000 [ 8052.932789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.932790] CR2: 00002aaaad64527d CR3: 0000003e066a4000 CR4: 00000000003607e0 [ 8052.932791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.932792] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.932792] Call Trace: [ 8052.932796] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.932799] [] _raw_spin_lock+0x30/0x40 [ 8052.932809] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.932820] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.932861] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.932866] [] ? del_timer_sync+0x52/0x60 [ 8052.932900] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.932931] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.932966] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.932969] [] ? wake_up_state+0x20/0x20 [ 8052.933003] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.933006] [] kthread+0xd1/0xe0 [ 8052.933008] [] ? insert_kthread_work+0x40/0x40 [ 8052.933010] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.933012] [] ? insert_kthread_work+0x40/0x40 [ 8052.933033] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8052.938719] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ptlrpcd_01_19:16854] [ 8052.938749] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.938771] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.938774] CPU: 20 PID: 16854 Comm: ptlrpcd_01_19 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.938775] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.938776] task: ffff8f484fbc0000 ti: ffff8f484fbc8000 task.ti: ffff8f484fbc8000 [ 8052.938780] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8052.938781] RSP: 0018:ffff8f484fbcbb58 EFLAGS: 00000246 [ 8052.938782] RAX: 0000000000000000 RBX: ffff8f683fa59680 RCX: 0000000000a10000 [ 8052.938783] RDX: ffff8f687f0db8c0 RSI: 0000000001b90001 RDI: ffff8f686e2b6b40 [ 8052.938784] RBP: ffff8f484fbcbb58 R08: ffff8f687ec9b8c0 R09: 0000000000000000 [ 8052.938785] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.938785] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d52ecab8 [ 8052.938787] FS: 0000000000000000(0000) GS:ffff8f687ec80000(0000) knlGS:0000000000000000 [ 8052.938788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.938789] CR2: 00002aaaad64527d CR3: 0000003f50bc4000 CR4: 00000000003607e0 [ 8052.938790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.938790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.938791] Call Trace: [ 8052.938794] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.938796] [] _raw_spin_lock+0x30/0x40 [ 8052.938805] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.938815] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.938848] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.938880] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.938910] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.938945] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.938947] [] ? wake_up_state+0x20/0x20 [ 8052.938981] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.938983] [] kthread+0xd1/0xe0 [ 8052.938985] [] ? insert_kthread_work+0x40/0x40 [ 8052.938987] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.938989] [] ? insert_kthread_work+0x40/0x40 [ 8052.939010] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8052.944718] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [ptlrpcd_01_27:16862] [ 8052.944749] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.944771] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.944774] CPU: 21 PID: 16862 Comm: ptlrpcd_01_27 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.944775] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.944776] task: ffff8f484fbe9080 ti: ffff8f484fbf0000 task.ti: ffff8f484fbf0000 [ 8052.944780] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.944781] RSP: 0018:ffff8f484fbf3b58 EFLAGS: 00000246 [ 8052.944782] RAX: 0000000000000000 RBX: ffff8f666b0dcc80 RCX: 0000000000a90000 [ 8052.944783] RDX: ffff8f487f5db8c0 RSI: 0000000000390001 RDI: ffff8f686e2b6b40 [ 8052.944784] RBP: ffff8f484fbf3b58 R08: ffff8f687ecdb8c0 R09: 0000000000000000 [ 8052.944784] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.944785] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003fe4b1a6 [ 8052.944787] FS: 0000000000000000(0000) GS:ffff8f687ecc0000(0000) knlGS:0000000000000000 [ 8052.944788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.944789] CR2: 00002aaaab176a00 CR3: 0000003ff8218000 CR4: 00000000003607e0 [ 8052.944790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.944790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.944791] Call Trace: [ 8052.944794] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.944796] [] _raw_spin_lock+0x30/0x40 [ 8052.944805] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.944815] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.944848] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.944895] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.944943] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.944993] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.944996] [] ? wake_up_state+0x20/0x20 [ 8052.945047] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.945050] [] kthread+0xd1/0xe0 [ 8052.945052] [] ? insert_kthread_work+0x40/0x40 [ 8052.945054] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.945056] [] ? insert_kthread_work+0x40/0x40 [ 8052.945077] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.950718] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [ptlrpcd_01_03:16838] [ 8052.950748] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.950770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.950773] CPU: 22 PID: 16838 Comm: ptlrpcd_01_03 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.950774] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.950775] task: ffff8f484f83d280 ti: ffff8f484fb20000 task.ti: ffff8f484fb20000 [ 8052.950779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.950780] RSP: 0018:ffff8f484fb23b58 EFLAGS: 00000246 [ 8052.950781] RAX: 0000000000000000 RBX: ffff8f65fdb85100 RCX: 0000000000b10000 [ 8052.950782] RDX: ffff8f687ec9b8c0 RSI: 0000000000a10001 RDI: ffff8f686e2b6b40 [ 8052.950783] RBP: ffff8f484fb23b58 R08: ffff8f687ed1b8c0 R09: 0000000000000000 [ 8052.950783] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.950784] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000027c33393 [ 8052.950786] FS: 0000000000000000(0000) GS:ffff8f687ed00000(0000) knlGS:0000000000000000 [ 8052.950787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.950787] CR2: 00002aaaad64527d CR3: 0000003ff0bda000 CR4: 00000000003607e0 [ 8052.950788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.950789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.950790] Call Trace: [ 8052.950793] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.950795] [] _raw_spin_lock+0x30/0x40 [ 8052.950804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.950813] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.950846] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.950879] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.950911] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.950946] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.950949] [] ? wake_up_state+0x20/0x20 [ 8052.950982] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.950985] [] kthread+0xd1/0xe0 [ 8052.950987] [] ? insert_kthread_work+0x40/0x40 [ 8052.950989] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.950991] [] ? insert_kthread_work+0x40/0x40 [ 8052.951012] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.969718] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [ptlrpcd_01_02:16837] [ 8052.969748] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.969771] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.969773] CPU: 25 PID: 16837 Comm: ptlrpcd_01_02 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.969774] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.969776] task: ffff8f484f83c200 ti: ffff8f484fb14000 task.ti: ffff8f484fb14000 [ 8052.969780] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.969781] RSP: 0018:ffff8f484fb17b58 EFLAGS: 00000246 [ 8052.969782] RAX: 0000000000000000 RBX: ffff8f65fc4bad00 RCX: 0000000000c90000 [ 8052.969783] RDX: ffff8f687f01b8c0 RSI: 0000000001110001 RDI: ffff8f686e2b6b40 [ 8052.969784] RBP: ffff8f484fb17b58 R08: ffff8f687eddb8c0 R09: 0000000000000000 [ 8052.969784] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.969785] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000924c28ea [ 8052.969787] FS: 0000000000000000(0000) GS:ffff8f687edc0000(0000) knlGS:0000000000000000 [ 8052.969787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.969788] CR2: 00007ffff7ff7000 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8052.969789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.969790] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.969791] Call Trace: [ 8052.969794] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.969796] [] _raw_spin_lock+0x30/0x40 [ 8052.969804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.969814] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.969847] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.969879] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.969909] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.969944] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.969947] [] ? wake_up_state+0x20/0x20 [ 8052.969980] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.969983] [] kthread+0xd1/0xe0 [ 8052.969985] [] ? insert_kthread_work+0x40/0x40 [ 8052.969987] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.969989] [] ? insert_kthread_work+0x40/0x40 [ 8052.970010] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.975718] NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [ptlrpcd_01_06:16841] [ 8052.975748] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.975770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.975773] CPU: 26 PID: 16841 Comm: ptlrpcd_01_06 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.975774] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.975775] task: ffff8f484fb29080 ti: ffff8f484fb34000 task.ti: ffff8f484fb34000 [ 8052.975779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.975780] RSP: 0018:ffff8f484fb37b58 EFLAGS: 00000246 [ 8052.975781] RAX: 0000000000000000 RBX: ffff8f65fd330d80 RCX: 0000000000d10000 [ 8052.975782] RDX: ffff8f487f51b8c0 RSI: 0000000000210001 RDI: ffff8f686e2b6b40 [ 8052.975783] RBP: ffff8f484fb37b58 R08: ffff8f687ee1b8c0 R09: 0000000000000000 [ 8052.975784] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.975784] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000008e17bb8 [ 8052.975786] FS: 0000000000000000(0000) GS:ffff8f687ee00000(0000) knlGS:0000000000000000 [ 8052.975787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.975788] CR2: 00002aaaad64527d CR3: 0000003f700dc000 CR4: 00000000003607e0 [ 8052.975789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.975789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.975790] Call Trace: [ 8052.975793] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.975795] [] _raw_spin_lock+0x30/0x40 [ 8052.975804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.975814] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.975846] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.975850] [] ? del_timer_sync+0x52/0x60 [ 8052.975881] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.975911] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.975946] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.975949] [] ? wake_up_state+0x20/0x20 [ 8052.975983] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.975985] [] kthread+0xd1/0xe0 [ 8052.975987] [] ? insert_kthread_work+0x40/0x40 [ 8052.975989] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.975991] [] ? insert_kthread_work+0x40/0x40 [ 8052.976012] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.981718] NMI watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [ptlrpcd_01_13:16848] [ 8052.981747] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.981770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.981773] CPU: 27 PID: 16848 Comm: ptlrpcd_01_13 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.981773] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.981775] task: ffff8f484fb99080 ti: ffff8f484fba0000 task.ti: ffff8f484fba0000 [ 8052.981779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.981780] RSP: 0018:ffff8f484fba3b58 EFLAGS: 00000246 [ 8052.981781] RAX: 0000000000000000 RBX: ffff8f65f8b41f80 RCX: 0000000000d90000 [ 8052.981782] RDX: ffff8f687ef9b8c0 RSI: 0000000001010001 RDI: ffff8f686e2b6b40 [ 8052.981783] RBP: ffff8f484fba3b58 R08: ffff8f687ee5b8c0 R09: 0000000000000000 [ 8052.981783] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.981784] R13: 0000000000000003 R14: 0000000000000013 R15: 000000006aa0a6de [ 8052.981786] FS: 0000000000000000(0000) GS:ffff8f687ee40000(0000) knlGS:0000000000000000 [ 8052.981786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.981787] CR2: 00002aaaad64527d CR3: 0000003f3769a000 CR4: 00000000003607e0 [ 8052.981788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.981789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.981789] Call Trace: [ 8052.981793] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.981795] [] _raw_spin_lock+0x30/0x40 [ 8052.981804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.981813] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.981846] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.981878] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.981909] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.981943] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.981946] [] ? wake_up_state+0x20/0x20 [ 8052.981980] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.981982] [] kthread+0xd1/0xe0 [ 8052.981984] [] ? insert_kthread_work+0x40/0x40 [ 8052.981986] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.981988] [] ? insert_kthread_work+0x40/0x40 [ 8052.982009] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8052.993717] NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [ptlrpcd_01_28:16863] [ 8052.993747] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8052.993770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8052.993772] CPU: 29 PID: 16863 Comm: ptlrpcd_01_28 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8052.993773] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8052.993774] task: ffff8f484fbea100 ti: ffff8f484fbf4000 task.ti: ffff8f484fbf4000 [ 8052.993778] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8052.993780] RSP: 0018:ffff8f484fbf7b58 EFLAGS: 00000246 [ 8052.993780] RAX: 0000000000000000 RBX: ffff8f65febf1200 RCX: 0000000000e90000 [ 8052.993781] RDX: ffff8f687ee1b8c0 RSI: 0000000000d10001 RDI: ffff8f686e2b6b40 [ 8052.993782] RBP: ffff8f484fbf7b58 R08: ffff8f687eedb8c0 R09: 0000000000000000 [ 8052.993783] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8052.993784] R13: 0000000000000003 R14: 0000000000000013 R15: 000000007c3a9fcc [ 8052.993785] FS: 0000000000000000(0000) GS:ffff8f687eec0000(0000) knlGS:0000000000000000 [ 8052.993786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.993787] CR2: 00002aaaad64527d CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8052.993788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8052.993789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8052.993789] Call Trace: [ 8052.993793] [] queued_spin_lock_slowpath+0xb/0xf [ 8052.993795] [] _raw_spin_lock+0x30/0x40 [ 8052.993803] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8052.993813] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8052.993845] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8052.993878] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8052.993908] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8052.993943] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8052.993946] [] ? wake_up_state+0x20/0x20 [ 8052.993980] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8052.993982] [] kthread+0xd1/0xe0 [ 8052.993985] [] ? insert_kthread_work+0x40/0x40 [ 8052.993986] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8052.993989] [] ? insert_kthread_work+0x40/0x40 [ 8052.994009] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.018717] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [ptlrpcd_01_05:16840] [ 8053.018746] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.018769] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.018772] CPU: 33 PID: 16840 Comm: ptlrpcd_01_05 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.018773] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.018774] task: ffff8f484fb28000 ti: ffff8f484fb30000 task.ti: ffff8f484fb30000 [ 8053.018778] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.018779] RSP: 0018:ffff8f484fb33b58 EFLAGS: 00000246 [ 8053.018780] RAX: 0000000000000000 RBX: ffff8f6844673f00 RCX: 0000000001090000 [ 8053.018781] RDX: ffff8f487fa5b8c0 RSI: 0000000001590001 RDI: ffff8f686e2b6b40 [ 8053.018782] RBP: ffff8f484fb33b58 R08: ffff8f687efdb8c0 R09: 0000000000000000 [ 8053.018783] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.018783] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000fb9209f2 [ 8053.018785] FS: 0000000000000000(0000) GS:ffff8f687efc0000(0000) knlGS:0000000000000000 [ 8053.018786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.018786] CR2: 00002aaaad64527d CR3: 0000003ecaedc000 CR4: 00000000003607e0 [ 8053.018787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.018788] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.018789] Call Trace: [ 8053.018792] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.018794] [] _raw_spin_lock+0x30/0x40 [ 8053.018803] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.018812] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.018845] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.018878] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.018908] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.018943] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.018945] [] ? wake_up_state+0x20/0x20 [ 8053.018979] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.018981] [] kthread+0xd1/0xe0 [ 8053.018984] [] ? insert_kthread_work+0x40/0x40 [ 8053.018986] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.018988] [] ? insert_kthread_work+0x40/0x40 [ 8053.019008] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.025717] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [ptlrpcd_01_04:16839] [ 8053.025748] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.025770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.025773] CPU: 34 PID: 16839 Comm: ptlrpcd_01_04 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.025774] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.025775] task: ffff8f484f83e300 ti: ffff8f484fb24000 task.ti: ffff8f484fb24000 [ 8053.025779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.025780] RSP: 0018:ffff8f484fb27b58 EFLAGS: 00000246 [ 8053.025781] RAX: 0000000000000000 RBX: ffff8f67e4731680 RCX: 0000000001110000 [ 8053.025782] RDX: ffff8f687f15b8c0 RSI: 0000000001c90001 RDI: ffff8f686e2b6b40 [ 8053.025783] RBP: ffff8f484fb27b58 R08: ffff8f687f01b8c0 R09: 0000000000000000 [ 8053.025784] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.025785] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003d57b1a1 [ 8053.025786] FS: 0000000000000000(0000) GS:ffff8f687f000000(0000) knlGS:0000000000000000 [ 8053.025787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.025788] CR2: 00002aaaad64527d CR3: 0000003f2f078000 CR4: 00000000003607e0 [ 8053.025789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.025789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.025790] Call Trace: [ 8053.025793] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.025796] [] _raw_spin_lock+0x30/0x40 [ 8053.025804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.025814] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.025847] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.025880] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.025911] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.025946] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.025949] [] ? wake_up_state+0x20/0x20 [ 8053.025982] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.025985] [] kthread+0xd1/0xe0 [ 8053.025987] [] ? insert_kthread_work+0x40/0x40 [ 8053.025989] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.025991] [] ? insert_kthread_work+0x40/0x40 [ 8053.026012] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.031717] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [ptlrpcd_01_16:16851] [ 8053.031747] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.031770] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.031773] CPU: 35 PID: 16851 Comm: ptlrpcd_01_16 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.031774] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.031775] task: ffff8f484fb9c200 ti: ffff8f484fbb4000 task.ti: ffff8f484fbb4000 [ 8053.031779] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.031780] RSP: 0018:ffff8f484fbb7b58 EFLAGS: 00000246 [ 8053.031781] RAX: 0000000000000000 RBX: ffff8f685cff9b00 RCX: 0000000001190000 [ 8053.031782] RDX: ffff8f687ed5b8c0 RSI: 0000000000b90001 RDI: ffff8f686e2b6b40 [ 8053.031783] RBP: ffff8f484fbb7b58 R08: ffff8f687f05b8c0 R09: 0000000000000000 [ 8053.031784] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.031785] R13: 0000000000000003 R14: 0000000000000013 R15: 000000002d9d0671 [ 8053.031786] FS: 0000000000000000(0000) GS:ffff8f687f040000(0000) knlGS:0000000000000000 [ 8053.031787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.031788] CR2: 00002aaaad64527d CR3: 0000003eef26a000 CR4: 00000000003607e0 [ 8053.031789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.031789] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.031790] Call Trace: [ 8053.031793] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.031796] [] _raw_spin_lock+0x30/0x40 [ 8053.031804] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.031815] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.031850] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.031883] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.031914] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.031949] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.031952] [] ? wake_up_state+0x20/0x20 [ 8053.031986] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.031988] [] kthread+0xd1/0xe0 [ 8053.031991] [] ? insert_kthread_work+0x40/0x40 [ 8053.031993] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.031995] [] ? insert_kthread_work+0x40/0x40 [ 8053.032016] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.094715] NMI watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [ptlrpcd_01_18:16853] [ 8053.094745] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.094767] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.094770] CPU: 54 PID: 16853 Comm: ptlrpcd_01_18 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.094771] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.094772] task: ffff8f484fb9e300 ti: ffff8f484fbbc000 task.ti: ffff8f484fbbc000 [ 8053.094775] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8053.094776] RSP: 0018:ffff8f484fbbfb58 EFLAGS: 00000246 [ 8053.094777] RAX: 0000000000000000 RBX: ffff8f47379e4380 RCX: 0000000001b10000 [ 8053.094778] RDX: ffff8f487fbdb8c0 RSI: 0000000001890001 RDI: ffff8f686e2b6b40 [ 8053.094778] RBP: ffff8f484fbbfb58 R08: ffff8f687f09b8c0 R09: 0000000000000000 [ 8053.094779] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.094780] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003dee356c [ 8053.094781] FS: 0000000000000000(0000) GS:ffff8f687f080000(0000) knlGS:0000000000000000 [ 8053.094782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.094783] CR2: 0000000000640558 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.094784] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.094785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.094785] Call Trace: [ 8053.094788] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.094790] [] _raw_spin_lock+0x30/0x40 [ 8053.094799] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.094809] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.094840] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.094873] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.094903] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.094938] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.094941] [] ? wake_up_state+0x20/0x20 [ 8053.094975] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.094977] [] kthread+0xd1/0xe0 [ 8053.094979] [] ? insert_kthread_work+0x40/0x40 [ 8053.094981] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.094983] [] ? insert_kthread_work+0x40/0x40 [ 8053.095004] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8053.097715] NMI watchdog: BUG: soft lockup - CPU#55 stuck for 23s! [ptlrpcd_01_26:16861] [ 8053.097744] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.097767] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.097770] CPU: 55 PID: 16861 Comm: ptlrpcd_01_26 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.097771] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.097772] task: ffff8f484fbe8000 ti: ffff8f484fbe4000 task.ti: ffff8f484fbe4000 [ 8053.097775] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.097776] RSP: 0018:ffff8f484fbe7b58 EFLAGS: 00000246 [ 8053.097777] RAX: 0000000000000000 RBX: ffff8f68460e1680 RCX: 0000000001b90000 [ 8053.097778] RDX: ffff8f487f99b8c0 RSI: 0000000001410001 RDI: ffff8f686e2b6b40 [ 8053.097779] RBP: ffff8f484fbe7b58 R08: ffff8f687f0db8c0 R09: 0000000000000000 [ 8053.097779] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.097780] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000018ef9bd [ 8053.097781] FS: 0000000000000000(0000) GS:ffff8f687f0c0000(0000) knlGS:0000000000000000 [ 8053.097782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.097783] CR2: 00000000006e9360 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.097784] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.097785] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.097785] Call Trace: [ 8053.097788] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.097790] [] _raw_spin_lock+0x30/0x40 [ 8053.097799] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.097809] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.097840] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.097873] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.097903] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.097938] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.097940] [] ? wake_up_state+0x20/0x20 [ 8053.097973] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.097975] [] kthread+0xd1/0xe0 [ 8053.097978] [] ? insert_kthread_work+0x40/0x40 [ 8053.097980] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.097982] [] ? insert_kthread_work+0x40/0x40 [ 8053.098002] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.100715] NMI watchdog: BUG: soft lockup - CPU#56 stuck for 23s! [ptlrpcd_01_25:16860] [ 8053.100744] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.100767] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.100769] CPU: 56 PID: 16860 Comm: ptlrpcd_01_25 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.100770] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.100771] task: ffff8f484fbc6300 ti: ffff8f484fbe0000 task.ti: ffff8f484fbe0000 [ 8053.100774] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8053.100775] RSP: 0018:ffff8f484fbe3b58 EFLAGS: 00000246 [ 8053.100776] RAX: 0000000000000000 RBX: ffff8f6846193a80 RCX: 0000000001c10000 [ 8053.100777] RDX: ffff8f687ee5b8c0 RSI: 0000000000d90001 RDI: ffff8f686e2b6b40 [ 8053.100778] RBP: ffff8f484fbe3b58 R08: ffff8f687f11b8c0 R09: 0000000000000000 [ 8053.100778] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.100779] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000013a84c86 [ 8053.100781] FS: 0000000000000000(0000) GS:ffff8f687f100000(0000) knlGS:0000000000000000 [ 8053.100781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.100782] CR2: 00002aaaaafbbd70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.100783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.100784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.100784] Call Trace: [ 8053.100787] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.100790] [] _raw_spin_lock+0x30/0x40 [ 8053.100798] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.100807] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.100839] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.100871] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.100901] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.100935] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.100938] [] ? wake_up_state+0x20/0x20 [ 8053.100972] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.100975] [] kthread+0xd1/0xe0 [ 8053.100977] [] ? insert_kthread_work+0x40/0x40 [ 8053.100979] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.100981] [] ? insert_kthread_work+0x40/0x40 [ 8053.101001] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8053.103715] NMI watchdog: BUG: soft lockup - CPU#57 stuck for 23s! [ptlrpcd_01_10:16845] [ 8053.103744] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.103767] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.103769] CPU: 57 PID: 16845 Comm: ptlrpcd_01_10 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.103770] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.103771] task: ffff8f484fb2d280 ti: ffff8f484fb8c000 task.ti: ffff8f484fb8c000 [ 8053.103774] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8053.103775] RSP: 0018:ffff8f484fb8fb58 EFLAGS: 00000246 [ 8053.103776] RAX: 0000000000000000 RBX: ffff8f6846795580 RCX: 0000000001c90000 [ 8053.103777] RDX: ffff8f487f55b8c0 RSI: 0000000000290001 RDI: ffff8f686e2b6b40 [ 8053.103778] RBP: ffff8f484fb8fb58 R08: ffff8f687f15b8c0 R09: 0000000000000000 [ 8053.103779] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.103780] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000048d6b28f [ 8053.103781] FS: 0000000000000000(0000) GS:ffff8f687f140000(0000) knlGS:0000000000000000 [ 8053.103782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.103783] CR2: 00000000006d2a70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.103783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.103784] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.103785] Call Trace: [ 8053.103788] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.103790] [] _raw_spin_lock+0x30/0x40 [ 8053.103798] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.103808] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.103843] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.103876] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.103906] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.103941] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.103944] [] ? wake_up_state+0x20/0x20 [ 8053.103978] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.103980] [] kthread+0xd1/0xe0 [ 8053.103982] [] ? insert_kthread_work+0x40/0x40 [ 8053.103984] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.103986] [] ? insert_kthread_work+0x40/0x40 [ 8053.104007] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8053.106715] NMI watchdog: BUG: soft lockup - CPU#58 stuck for 23s! [ptlrpcd_01_14:16849] [ 8053.106744] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.106766] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.106768] CPU: 58 PID: 16849 Comm: ptlrpcd_01_14 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.106769] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.106770] task: ffff8f484fb9a100 ti: ffff8f484fba4000 task.ti: ffff8f484fba4000 [ 8053.106774] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.106775] RSP: 0018:ffff8f484fba7b58 EFLAGS: 00000246 [ 8053.106776] RAX: 0000000000000000 RBX: ffff8f66041b1680 RCX: 0000000001d10000 [ 8053.106776] RDX: ffff8f687f25b8c0 RSI: 0000000001e90001 RDI: ffff8f686e2b6b40 [ 8053.106777] RBP: ffff8f484fba7b58 R08: ffff8f687f19b8c0 R09: 0000000000000000 [ 8053.106778] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.106779] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003cd214ac [ 8053.106780] FS: 0000000000000000(0000) GS:ffff8f687f180000(0000) knlGS:0000000000000000 [ 8053.106781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.106782] CR2: 00002aaaaad94d70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.106783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.106783] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.106784] Call Trace: [ 8053.106787] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.106789] [] _raw_spin_lock+0x30/0x40 [ 8053.106797] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.106807] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.106838] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.106871] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.106900] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.106935] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.106938] [] ? wake_up_state+0x20/0x20 [ 8053.106970] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.106973] [] kthread+0xd1/0xe0 [ 8053.106975] [] ? insert_kthread_work+0x40/0x40 [ 8053.106977] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.106979] [] ? insert_kthread_work+0x40/0x40 [ 8053.107000] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.124714] NMI watchdog: BUG: soft lockup - CPU#64 stuck for 23s! [ptlrpcd_01_09:16844] [ 8053.124745] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.124767] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.124770] CPU: 64 PID: 16844 Comm: ptlrpcd_01_09 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.124771] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.124772] task: ffff8f484fb2c200 ti: ffff8f484fb88000 task.ti: ffff8f484fb88000 [ 8053.124776] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8053.124777] RSP: 0018:ffff8f484fb8bb58 EFLAGS: 00000246 [ 8053.124778] RAX: 0000000000000000 RBX: ffff8f6841eb8000 RCX: 0000000002010000 [ 8053.124779] RDX: ffff8f487fa9b8c0 RSI: 0000000001610001 RDI: ffff8f686e2b6b40 [ 8053.124780] RBP: ffff8f484fb8bb58 R08: ffff8f687f31b8c0 R09: 0000000000000000 [ 8053.124781] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.124781] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000ac8b218f [ 8053.124783] FS: 0000000000000000(0000) GS:ffff8f687f300000(0000) knlGS:0000000000000000 [ 8053.124784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.124785] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.124785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.124786] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.124787] Call Trace: [ 8053.124790] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.124792] [] _raw_spin_lock+0x30/0x40 [ 8053.124801] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.124811] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.124843] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.124875] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.124905] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.124940] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.124943] [] ? wake_up_state+0x20/0x20 [ 8053.124976] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.124979] [] kthread+0xd1/0xe0 [ 8053.124981] [] ? insert_kthread_work+0x40/0x40 [ 8053.124983] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.124985] [] ? insert_kthread_work+0x40/0x40 [ 8053.125006] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8053.133714] NMI watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [ptlrpcd_01_33:16868] [ 8053.133745] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8053.133768] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8053.133770] CPU: 67 PID: 16868 Comm: ptlrpcd_01_33 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8053.133771] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8053.133772] task: ffff8f484d290000 ti: ffff8f484d298000 task.ti: ffff8f484d298000 [ 8053.133777] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8053.133778] RSP: 0018:ffff8f484d29bb58 EFLAGS: 00000246 [ 8053.133779] RAX: 0000000000000000 RBX: ffff8f68400c5e80 RCX: 0000000002190000 [ 8053.133780] RDX: ffff8f687ec1b8c0 RSI: 0000000000910001 RDI: ffff8f686e2b6b40 [ 8053.133781] RBP: ffff8f484d29bb58 R08: ffff8f687f3db8c0 R09: 0000000000000000 [ 8053.133781] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8053.133782] R13: 0000000000000003 R14: 0000000000000013 R15: 000000004ff7fd94 [ 8053.133784] FS: 0000000000000000(0000) GS:ffff8f687f3c0000(0000) knlGS:0000000000000000 [ 8053.133785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8053.133786] CR2: 0000000000630fb8 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8053.133787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8053.133787] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8053.133788] Call Trace: [ 8053.133791] [] queued_spin_lock_slowpath+0xb/0xf [ 8053.133794] [] _raw_spin_lock+0x30/0x40 [ 8053.133802] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8053.133812] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8053.133845] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8053.133878] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8053.133909] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8053.133944] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8053.133947] [] ? wake_up_state+0x20/0x20 [ 8053.133981] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8053.133983] [] kthread+0xd1/0xe0 [ 8053.133986] [] ? insert_kthread_work+0x40/0x40 [ 8053.133988] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8053.133990] [] ? insert_kthread_work+0x40/0x40 [ 8053.134011] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8053.346889] Lustre: lsh-OST0005-osc-ffff8f687d1f4800: Connection restored to 172.19.3.22@o2ib600 (at 172.19.3.22@o2ib600) [ 8053.346891] Lustre: Skipped 6 previous similar messages [ 8056.800625] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [ptlrpcd_00_24:16822] [ 8056.800665] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8056.800695] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8056.800697] CPU: 9 PID: 16822 Comm: ptlrpcd_00_24 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8056.800698] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8056.800699] task: ffff8f484d3dc200 ti: ffff8f484d3f8000 task.ti: ffff8f484d3f8000 [ 8056.800706] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8056.800707] RSP: 0018:ffff8f484d3fbb58 EFLAGS: 00000246 [ 8056.800708] RAX: 0000000000000000 RBX: ffff8f67cdd19680 RCX: 0000000000490000 [ 8056.800709] RDX: ffff8f687efdb8c0 RSI: 0000000001090001 RDI: ffff8f686e2b6b40 [ 8056.800710] RBP: ffff8f484d3fbb58 R08: ffff8f487f65b8c0 R09: 0000000000000000 [ 8056.800711] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8056.800712] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000b3f23bae [ 8056.800713] FS: 0000000000000000(0000) GS:ffff8f487f640000(0000) knlGS:0000000000000000 [ 8056.800714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.800715] CR2: 00002aaaabaa0aa0 CR3: 0000001f090d8000 CR4: 00000000003607e0 [ 8056.800716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8056.800717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8056.800717] Call Trace: [ 8056.800724] [] queued_spin_lock_slowpath+0xb/0xf [ 8056.800727] [] _raw_spin_lock+0x30/0x40 [ 8056.800742] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8056.800755] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8056.800805] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8056.800840] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8056.800870] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8056.800908] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8056.800912] [] ? wake_up_state+0x20/0x20 [ 8056.800945] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8056.800948] [] kthread+0xd1/0xe0 [ 8056.800951] [] ? insert_kthread_work+0x40/0x40 [ 8056.800954] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8056.800956] [] ? insert_kthread_work+0x40/0x40 [ 8056.800977] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8056.825625] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [ptlrpcd_00_08:16806] [ 8056.825655] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8056.825678] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8056.825680] CPU: 13 PID: 16806 Comm: ptlrpcd_00_08 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8056.825681] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8056.825682] task: ffff8f484c67a100 ti: ffff8f484c600000 task.ti: ffff8f484c600000 [ 8056.825687] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8056.825688] RSP: 0018:ffff8f484c603b58 EFLAGS: 00000246 [ 8056.825689] RAX: 0000000000000000 RBX: ffff8f4726858d80 RCX: 0000000000690000 [ 8056.825689] RDX: ffff8f487fb5b8c0 RSI: 0000000001790001 RDI: ffff8f686e2b6b40 [ 8056.825690] RBP: ffff8f484c603b58 R08: ffff8f487f75b8c0 R09: 0000000000000000 [ 8056.825691] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8056.825692] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e69de9aa [ 8056.825693] FS: 0000000000000000(0000) GS:ffff8f487f740000(0000) knlGS:0000000000000000 [ 8056.825694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.825695] CR2: 00007ffff7ff8000 CR3: 0000001ff8618000 CR4: 00000000003607e0 [ 8056.825696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8056.825697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8056.825697] Call Trace: [ 8056.825701] [] queued_spin_lock_slowpath+0xb/0xf [ 8056.825703] [] _raw_spin_lock+0x30/0x40 [ 8056.825712] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8056.825721] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8056.825754] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8056.825789] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8056.825822] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8056.825857] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8056.825860] [] ? wake_up_state+0x20/0x20 [ 8056.825894] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8056.825896] [] kthread+0xd1/0xe0 [ 8056.825899] [] ? insert_kthread_work+0x40/0x40 [ 8056.825901] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8056.825903] [] ? insert_kthread_work+0x40/0x40 [ 8056.825923] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8056.850624] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ptlrpcd_00_10:16808] [ 8056.850656] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8056.850678] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8056.850681] CPU: 17 PID: 16808 Comm: ptlrpcd_00_10 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8056.850682] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8056.850683] task: ffff8f484c67c200 ti: ffff8f484c608000 task.ti: ffff8f484c608000 [ 8056.850687] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8056.850688] RSP: 0018:ffff8f484c60bb58 EFLAGS: 00000246 [ 8056.850689] RAX: 0000000000000000 RBX: ffff8f465f060000 RCX: 0000000000890000 [ 8056.850690] RDX: ffff8f687f35b8c0 RSI: 0000000002090001 RDI: ffff8f686e2b6b40 [ 8056.850691] RBP: ffff8f484c60bb58 R08: ffff8f487f85b8c0 R09: 0000000000000000 [ 8056.850692] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8056.850692] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000bb7c4242 [ 8056.850694] FS: 0000000000000000(0000) GS:ffff8f487f840000(0000) knlGS:0000000000000000 [ 8056.850695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.850695] CR2: 00002aaaad64527d CR3: 0000003ffe4dc000 CR4: 00000000003607e0 [ 8056.850696] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8056.850697] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8056.850698] Call Trace: [ 8056.850701] [] queued_spin_lock_slowpath+0xb/0xf [ 8056.850703] [] _raw_spin_lock+0x30/0x40 [ 8056.850712] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8056.850722] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8056.850755] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8056.850788] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8056.850819] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8056.850854] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8056.850857] [] ? wake_up_state+0x20/0x20 [ 8056.850891] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8056.850893] [] kthread+0xd1/0xe0 [ 8056.850896] [] ? insert_kthread_work+0x40/0x40 [ 8056.850898] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8056.850900] [] ? insert_kthread_work+0x40/0x40 [ 8056.850920] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8056.957621] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ptlrpcd_01_24:16859] [ 8056.957653] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8056.957675] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8056.957678] CPU: 23 PID: 16859 Comm: ptlrpcd_01_24 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8056.957679] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8056.957680] task: ffff8f484fbc5280 ti: ffff8f484fbdc000 task.ti: ffff8f484fbdc000 [ 8056.957685] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8056.957686] RSP: 0018:ffff8f484fbdfb58 EFLAGS: 00000246 [ 8056.957687] RAX: 0000000000000000 RBX: ffff8f683de85580 RCX: 0000000000b90000 [ 8056.957688] RDX: ffff8f487f49b8c0 RSI: 0000000000110001 RDI: ffff8f686e2b6b40 [ 8056.957688] RBP: ffff8f484fbdfb58 R08: ffff8f687ed5b8c0 R09: 0000000000000000 [ 8056.957689] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8056.957690] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a400a58d [ 8056.957691] FS: 0000000000000000(0000) GS:ffff8f687ed40000(0000) knlGS:0000000000000000 [ 8056.957692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.957693] CR2: 00002aaaad64527d CR3: 0000003ee53fa000 CR4: 00000000003607e0 [ 8056.957694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8056.957695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8056.957695] Call Trace: [ 8056.957699] [] queued_spin_lock_slowpath+0xb/0xf [ 8056.957701] [] _raw_spin_lock+0x30/0x40 [ 8056.957710] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8056.957720] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8056.957752] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8056.957785] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8056.957816] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8056.957851] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8056.957854] [] ? wake_up_state+0x20/0x20 [ 8056.957888] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8056.957890] [] kthread+0xd1/0xe0 [ 8056.957892] [] ? insert_kthread_work+0x40/0x40 [ 8056.957894] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8056.957896] [] ? insert_kthread_work+0x40/0x40 [ 8056.957917] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8056.963621] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [ptlrpcd_01_17:16852] [ 8056.963650] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8056.963673] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8056.963676] CPU: 24 PID: 16852 Comm: ptlrpcd_01_17 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8056.963677] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8056.963678] task: ffff8f484fb9d280 ti: ffff8f484fbb8000 task.ti: ffff8f484fbb8000 [ 8056.963682] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8056.963683] RSP: 0018:ffff8f484fbbbb58 EFLAGS: 00000246 [ 8056.963684] RAX: 0000000000000000 RBX: ffff8f6841168000 RCX: 0000000000c10000 [ 8056.963685] RDX: ffff8f487f81b8c0 RSI: 0000000000810001 RDI: ffff8f686e2b6b40 [ 8056.963686] RBP: ffff8f484fbbbb58 R08: ffff8f687ed9b8c0 R09: 0000000000000000 [ 8056.963686] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8056.963687] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000099bba806 [ 8056.963689] FS: 0000000000000000(0000) GS:ffff8f687ed80000(0000) knlGS:0000000000000000 [ 8056.963689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.963690] CR2: 00002aaaad64527d CR3: 0000003e741f4000 CR4: 00000000003607e0 [ 8056.963691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8056.963692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8056.963692] Call Trace: [ 8056.963696] [] queued_spin_lock_slowpath+0xb/0xf [ 8056.963698] [] _raw_spin_lock+0x30/0x40 [ 8056.963706] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8056.963716] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8056.963749] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8056.963782] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8056.963812] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8056.963847] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8056.963850] [] ? wake_up_state+0x20/0x20 [ 8056.963884] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8056.963886] [] kthread+0xd1/0xe0 [ 8056.963889] [] ? insert_kthread_work+0x40/0x40 [ 8056.963891] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8056.963893] [] ? insert_kthread_work+0x40/0x40 [ 8056.963914] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8057.012620] NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [ptlrpcd_01_35:16870] [ 8057.012650] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.012673] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.012675] CPU: 32 PID: 16870 Comm: ptlrpcd_01_35 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.012676] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.012677] task: ffff8f484d292100 ti: ffff8f484d2a0000 task.ti: ffff8f484d2a0000 [ 8057.012681] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8057.012683] RSP: 0018:ffff8f484d2a3b58 EFLAGS: 00000246 [ 8057.012684] RAX: 0000000000000000 RBX: ffff8f47632ff980 RCX: 0000000001010000 [ 8057.012685] RDX: ffff8f687ed9b8c0 RSI: 0000000000c10001 RDI: ffff8f686e2b6b40 [ 8057.012685] RBP: ffff8f484d2a3b58 R08: ffff8f687ef9b8c0 R09: 0000000000000000 [ 8057.012686] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.012687] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d0b041d1 [ 8057.012688] FS: 0000000000000000(0000) GS:ffff8f687ef80000(0000) knlGS:0000000000000000 [ 8057.012689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.012690] CR2: 00002aaaad64527d CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8057.012691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.012692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.012692] Call Trace: [ 8057.012696] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.012698] [] _raw_spin_lock+0x30/0x40 [ 8057.012706] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.012716] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.012749] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.012782] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.012812] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.012847] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.012850] [] ? wake_up_state+0x20/0x20 [ 8057.012883] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.012886] [] kthread+0xd1/0xe0 [ 8057.012888] [] ? insert_kthread_work+0x40/0x40 [ 8057.012890] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.012892] [] ? insert_kthread_work+0x40/0x40 [ 8057.012913] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8057.053619] NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [ptlrpcd_00_06:16804] [ 8057.053649] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.053672] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.053675] CPU: 41 PID: 16804 Comm: ptlrpcd_00_06 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.053675] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.053677] task: ffff8f484c678000 ti: ffff8f484c650000 task.ti: ffff8f484c650000 [ 8057.053681] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.053682] RSP: 0018:ffff8f484c653b58 EFLAGS: 00000246 [ 8057.053683] RAX: 0000000000000000 RBX: ffff8f474b4e5e80 RCX: 0000000001490000 [ 8057.053684] RDX: ffff8f687f4db8c0 RSI: 0000000002390001 RDI: ffff8f686e2b6b40 [ 8057.053684] RBP: ffff8f484c653b58 R08: ffff8f487f9db8c0 R09: 0000000000000000 [ 8057.053685] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.053686] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000337f499b [ 8057.053687] FS: 0000000000000000(0000) GS:ffff8f487f9c0000(0000) knlGS:0000000000000000 [ 8057.053688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.053689] CR2: 00002aaaaafbbd70 CR3: 0000003ffe2ea000 CR4: 00000000003607e0 [ 8057.053690] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.053691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.053691] Call Trace: [ 8057.053695] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.053697] [] _raw_spin_lock+0x30/0x40 [ 8057.053705] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.053716] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.053751] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.053784] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.053815] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.053850] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.053853] [] ? wake_up_state+0x20/0x20 [ 8057.053886] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.053889] [] kthread+0xd1/0xe0 [ 8057.053891] [] ? insert_kthread_work+0x40/0x40 [ 8057.053893] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.053895] [] ? insert_kthread_work+0x40/0x40 [ 8057.053916] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.056618] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [ptlrpcd_00_12:16810] [ 8057.056648] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.056671] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.056674] CPU: 42 PID: 16810 Comm: ptlrpcd_00_12 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.056675] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.056676] task: ffff8f484c67e300 ti: ffff8f484c61c000 task.ti: ffff8f484c61c000 [ 8057.056680] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8057.056681] RSP: 0018:ffff8f484c61fb58 EFLAGS: 00000246 [ 8057.056682] RAX: 0000000000000000 RBX: ffff8f465f0dbf00 RCX: 0000000001510000 [ 8057.056683] RDX: ffff8f687eddb8c0 RSI: 0000000000c90001 RDI: ffff8f686e2b6b40 [ 8057.056684] RBP: ffff8f484c61fb58 R08: ffff8f487fa1b8c0 R09: 0000000000000000 [ 8057.056685] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.056686] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000b73f4650 [ 8057.056687] FS: 0000000000000000(0000) GS:ffff8f487fa00000(0000) knlGS:0000000000000000 [ 8057.056688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.056689] CR2: 00002aaaaad94d70 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8057.056690] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.056691] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.056691] Call Trace: [ 8057.056695] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.056697] [] _raw_spin_lock+0x30/0x40 [ 8057.056706] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.056716] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.056750] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.056784] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.056816] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.056853] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.056856] [] ? wake_up_state+0x20/0x20 [ 8057.056890] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.056893] [] kthread+0xd1/0xe0 [ 8057.056895] [] ? insert_kthread_work+0x40/0x40 [ 8057.056897] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.056899] [] ? insert_kthread_work+0x40/0x40 [ 8057.056922] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8057.065618] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [ptlrpcd_00_27:16825] [ 8057.065647] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.065670] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.065672] CPU: 45 PID: 16825 Comm: ptlrpcd_00_27 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.065672] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.065674] task: ffff8f484f808000 ti: ffff8f484f804000 task.ti: ffff8f484f804000 [ 8057.065677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.065678] RSP: 0018:ffff8f484f807b58 EFLAGS: 00000246 [ 8057.065678] RAX: 0000000000000000 RBX: ffff8f477a6d6300 RCX: 0000000001690000 [ 8057.065679] RDX: ffff8f487f4db8c0 RSI: 0000000000190001 RDI: ffff8f686e2b6b40 [ 8057.065680] RBP: ffff8f484f807b58 R08: ffff8f487fadb8c0 R09: 0000000000000000 [ 8057.065681] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.065682] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000003f4d10b [ 8057.065683] FS: 0000000000000000(0000) GS:ffff8f487fac0000(0000) knlGS:0000000000000000 [ 8057.065684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.065685] CR2: 00002aaaab1114b1 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.065686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.065686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.065687] Call Trace: [ 8057.065690] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.065692] [] _raw_spin_lock+0x30/0x40 [ 8057.065700] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.065710] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.065741] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.065773] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.065803] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.065837] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.065840] [] ? wake_up_state+0x20/0x20 [ 8057.065873] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.065875] [] kthread+0xd1/0xe0 [ 8057.065877] [] ? insert_kthread_work+0x40/0x40 [ 8057.065879] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.065881] [] ? insert_kthread_work+0x40/0x40 [ 8057.065902] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.071618] NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ptlrpcd_00_32:16830] [ 8057.071648] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.071670] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.071673] CPU: 47 PID: 16830 Comm: ptlrpcd_00_32 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.071674] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.071675] task: ffff8f484f80d280 ti: ffff8f484f828000 task.ti: ffff8f484f828000 [ 8057.071679] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.071680] RSP: 0018:ffff8f484f82bb58 EFLAGS: 00000246 [ 8057.071681] RAX: 0000000000000000 RBX: ffff8f4779e15e80 RCX: 0000000001790000 [ 8057.071682] RDX: ffff8f687ef1b8c0 RSI: 0000000000f10001 RDI: ffff8f686e2b6b40 [ 8057.071683] RBP: ffff8f484f82bb58 R08: ffff8f487fb5b8c0 R09: 0000000000000000 [ 8057.071683] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.071684] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000ab54e064 [ 8057.071686] FS: 0000000000000000(0000) GS:ffff8f487fb40000(0000) knlGS:0000000000000000 [ 8057.071687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.071688] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.071688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.071689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.071690] Call Trace: [ 8057.071693] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.071695] [] _raw_spin_lock+0x30/0x40 [ 8057.071704] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.071714] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.071746] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.071751] [] ? del_timer_sync+0x52/0x60 [ 8057.071782] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.071812] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.071847] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.071849] [] ? wake_up_state+0x20/0x20 [ 8057.071882] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.071884] [] kthread+0xd1/0xe0 [ 8057.071887] [] ? insert_kthread_work+0x40/0x40 [ 8057.071889] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.071891] [] ? insert_kthread_work+0x40/0x40 [ 8057.071912] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.077618] NMI watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [ptlrpcd_00_01:16799] [ 8057.077647] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.077670] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.077672] CPU: 49 PID: 16799 Comm: ptlrpcd_00_01 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.077673] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.077674] task: ffff8f484fb7a100 ti: ffff8f484c664000 task.ti: ffff8f484c664000 [ 8057.077678] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.077679] RSP: 0018:ffff8f484c667b58 EFLAGS: 00000246 [ 8057.077679] RAX: 0000000000000000 RBX: ffff8f474d0c5100 RCX: 0000000001890000 [ 8057.077680] RDX: ffff8f687f2db8c0 RSI: 0000000001f90001 RDI: ffff8f686e2b6b40 [ 8057.077681] RBP: ffff8f484c667b58 R08: ffff8f487fbdb8c0 R09: 0000000000000000 [ 8057.077682] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.077683] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a3a15eaa [ 8057.077684] FS: 0000000000000000(0000) GS:ffff8f487fbc0000(0000) knlGS:0000000000000000 [ 8057.077685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.077686] CR2: 00002aaaaad94d70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.077687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.077687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.077688] Call Trace: [ 8057.077691] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.077693] [] _raw_spin_lock+0x30/0x40 [ 8057.077702] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.077712] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.077744] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.077777] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.077809] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.077844] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.077847] [] ? wake_up_state+0x20/0x20 [ 8057.077882] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.077884] [] kthread+0xd1/0xe0 [ 8057.077886] [] ? insert_kthread_work+0x40/0x40 [ 8057.077888] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.077890] [] ? insert_kthread_work+0x40/0x40 [ 8057.077911] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.109617] NMI watchdog: BUG: soft lockup - CPU#59 stuck for 22s! [ptlrpcd_01_32:16867] [ 8057.109648] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.109670] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.109673] CPU: 59 PID: 16867 Comm: ptlrpcd_01_32 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.109674] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.109675] task: ffff8f484fbee300 ti: ffff8f484d28c000 task.ti: ffff8f484d28c000 [ 8057.109678] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8057.109679] RSP: 0018:ffff8f484d28fb58 EFLAGS: 00000246 [ 8057.109680] RAX: 0000000000000000 RBX: ffff8f4757892880 RCX: 0000000001d90000 [ 8057.109681] RDX: ffff8f487fcdb8c0 RSI: 0000000001a90001 RDI: ffff8f686e2b6b40 [ 8057.109682] RBP: ffff8f484d28fb58 R08: ffff8f687f1db8c0 R09: 0000000000000000 [ 8057.109683] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.109683] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a91fcf32 [ 8057.109685] FS: 0000000000000000(0000) GS:ffff8f687f1c0000(0000) knlGS:0000000000000000 [ 8057.109686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.109686] CR2: 00002aaaaaad6f58 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8057.109687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.109688] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.109688] Call Trace: [ 8057.109692] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.109694] [] _raw_spin_lock+0x30/0x40 [ 8057.109702] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.109712] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.109744] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.109776] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.109806] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.109841] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.109844] [] ? wake_up_state+0x20/0x20 [ 8057.109877] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.109880] [] kthread+0xd1/0xe0 [ 8057.109882] [] ? insert_kthread_work+0x40/0x40 [ 8057.109884] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.109886] [] ? insert_kthread_work+0x40/0x40 [ 8057.109907] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8057.112617] NMI watchdog: BUG: soft lockup - CPU#60 stuck for 22s! [ptlrpcd_01_21:16856] [ 8057.112647] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.112670] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.112672] CPU: 60 PID: 16856 Comm: ptlrpcd_01_21 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.112673] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.112674] task: ffff8f484fbc2100 ti: ffff8f484fbd0000 task.ti: ffff8f484fbd0000 [ 8057.112677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8057.112679] RSP: 0018:ffff8f484fbd3b58 EFLAGS: 00000246 [ 8057.112679] RAX: 0000000000000000 RBX: ffff8f660c9ff980 RCX: 0000000001e10000 [ 8057.112680] RDX: ffff8f687f3db8c0 RSI: 0000000002190001 RDI: ffff8f686e2b6b40 [ 8057.112681] RBP: ffff8f484fbd3b58 R08: ffff8f687f21b8c0 R09: 0000000000000000 [ 8057.112682] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.112683] R13: 0000000000000003 R14: 0000000000000013 R15: 000000007600e3df [ 8057.112684] FS: 0000000000000000(0000) GS:ffff8f687f200000(0000) knlGS:0000000000000000 [ 8057.112685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.112686] CR2: 00002aaaaad94d70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.112687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.112687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.112688] Call Trace: [ 8057.112691] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.112693] [] _raw_spin_lock+0x30/0x40 [ 8057.112701] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.112711] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.112745] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.112777] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.112807] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.112841] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.112844] [] ? wake_up_state+0x20/0x20 [ 8057.112877] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.112880] [] kthread+0xd1/0xe0 [ 8057.112882] [] ? insert_kthread_work+0x40/0x40 [ 8057.112884] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.112886] [] ? insert_kthread_work+0x40/0x40 [ 8057.112907] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8057.118617] NMI watchdog: BUG: soft lockup - CPU#62 stuck for 22s! [ptlrpcd_01_23:16858] [ 8057.118646] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.118668] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.118671] CPU: 62 PID: 16858 Comm: ptlrpcd_01_23 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.118672] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.118673] task: ffff8f484fbc4200 ti: ffff8f484fbd8000 task.ti: ffff8f484fbd8000 [ 8057.118676] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.118677] RSP: 0018:ffff8f484fbdbb58 EFLAGS: 00000246 [ 8057.118678] RAX: 0000000000000000 RBX: ffff8f6840375100 RCX: 0000000001f10000 [ 8057.118679] RDX: ffff8f687f09b8c0 RSI: 0000000001b10001 RDI: ffff8f686e2b6b40 [ 8057.118680] RBP: ffff8f484fbdbb58 R08: ffff8f687f29b8c0 R09: 0000000000000000 [ 8057.118681] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.118682] R13: 0000000000000003 R14: 0000000000000013 R15: 000000008629e995 [ 8057.118683] FS: 0000000000000000(0000) GS:ffff8f687f280000(0000) knlGS:0000000000000000 [ 8057.118684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.118685] CR2: 00000000006e9360 CR3: 0000001f1aa7a000 CR4: 00000000003607e0 [ 8057.118685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.118686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.118687] Call Trace: [ 8057.118690] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.118692] [] _raw_spin_lock+0x30/0x40 [ 8057.118700] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.118710] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.118741] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.118774] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.118804] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.118838] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.118841] [] ? wake_up_state+0x20/0x20 [ 8057.118875] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.118877] [] kthread+0xd1/0xe0 [ 8057.118880] [] ? insert_kthread_work+0x40/0x40 [ 8057.118882] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.118884] [] ? insert_kthread_work+0x40/0x40 [ 8057.118904] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.121617] NMI watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [ptlrpcd_01_00:16834] [ 8057.121646] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.121669] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.121671] CPU: 63 PID: 16834 Comm: ptlrpcd_01_00 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.121672] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.121673] task: ffff8f484f83a100 ti: ffff8f484fb0c000 task.ti: ffff8f484fb0c000 [ 8057.121676] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.121677] RSP: 0018:ffff8f484fb0fb58 EFLAGS: 00000246 [ 8057.121678] RAX: 0000000000000000 RBX: ffff8f661bb6f500 RCX: 0000000001f90000 [ 8057.121679] RDX: ffff8f687f41b8c0 RSI: 0000000002210001 RDI: ffff8f686e2b6b40 [ 8057.121680] RBP: ffff8f484fb0fb58 R08: ffff8f687f2db8c0 R09: 0000000000000000 [ 8057.121681] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.121681] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003fb7af09 [ 8057.121683] FS: 0000000000000000(0000) GS:ffff8f687f2c0000(0000) knlGS:0000000000000000 [ 8057.121684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.121684] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.121685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.121686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.121686] Call Trace: [ 8057.121689] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.121692] [] _raw_spin_lock+0x30/0x40 [ 8057.121700] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.121710] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.121741] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.121773] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.121803] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.121838] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.121840] [] ? wake_up_state+0x20/0x20 [ 8057.121874] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.121876] [] kthread+0xd1/0xe0 [ 8057.121878] [] ? insert_kthread_work+0x40/0x40 [ 8057.121880] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.121882] [] ? insert_kthread_work+0x40/0x40 [ 8057.121903] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.130618] NMI watchdog: BUG: soft lockup - CPU#66 stuck for 22s! [ptlrpcd_01_08:16843] [ 8057.130648] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.130671] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.130674] CPU: 66 PID: 16843 Comm: ptlrpcd_01_08 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.130675] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.130676] task: ffff8f484fb2b180 ti: ffff8f484fb3c000 task.ti: ffff8f484fb3c000 [ 8057.130681] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 8057.130682] RSP: 0018:ffff8f484fb3fb58 EFLAGS: 00000246 [ 8057.130683] RAX: 0000000000000000 RBX: ffff8f674074f080 RCX: 0000000002110000 [ 8057.130684] RDX: ffff8f687ecdb8c0 RSI: 0000000000a90001 RDI: ffff8f686e2b6b40 [ 8057.130684] RBP: ffff8f484fb3fb58 R08: ffff8f687f39b8c0 R09: 0000000000000000 [ 8057.130685] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.130686] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000138cb389 [ 8057.130688] FS: 0000000000000000(0000) GS:ffff8f687f380000(0000) knlGS:0000000000000000 [ 8057.130689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.130690] CR2: 00002aaab8007088 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8057.130691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.130692] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.130692] Call Trace: [ 8057.130696] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.130698] [] _raw_spin_lock+0x30/0x40 [ 8057.130706] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.130717] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.130750] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.130753] [] ? del_timer_sync+0x52/0x60 [ 8057.130785] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.130816] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.130851] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.130854] [] ? wake_up_state+0x20/0x20 [ 8057.130888] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.130890] [] kthread+0xd1/0xe0 [ 8057.130893] [] ? insert_kthread_work+0x40/0x40 [ 8057.130895] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.130897] [] ? insert_kthread_work+0x40/0x40 [ 8057.130918] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [ 8057.136617] NMI watchdog: BUG: soft lockup - CPU#68 stuck for 22s! [ptlrpcd_01_22:16857] [ 8057.136646] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.136669] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.136671] CPU: 68 PID: 16857 Comm: ptlrpcd_01_22 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.136672] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.136673] task: ffff8f484fbc3180 ti: ffff8f484fbd4000 task.ti: ffff8f484fbd4000 [ 8057.136676] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.136678] RSP: 0018:ffff8f484fbd7b58 EFLAGS: 00000246 [ 8057.136678] RAX: 0000000000000000 RBX: ffff8f68473e2d00 RCX: 0000000002210000 [ 8057.136679] RDX: ffff8f687ed1b8c0 RSI: 0000000000b10001 RDI: ffff8f686e2b6b40 [ 8057.136680] RBP: ffff8f484fbd7b58 R08: ffff8f687f41b8c0 R09: 0000000000000000 [ 8057.136681] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.136682] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000162586b5 [ 8057.136683] FS: 0000000000000000(0000) GS:ffff8f687f400000(0000) knlGS:0000000000000000 [ 8057.136684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.136685] CR2: 00002aaaaafbbd70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.136685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.136686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.136687] Call Trace: [ 8057.136690] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.136692] [] _raw_spin_lock+0x30/0x40 [ 8057.136700] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.136710] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.136741] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.136745] [] ? del_timer_sync+0x52/0x60 [ 8057.136776] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.136806] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.136840] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.136842] [] ? wake_up_state+0x20/0x20 [ 8057.136876] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.136878] [] kthread+0xd1/0xe0 [ 8057.136880] [] ? insert_kthread_work+0x40/0x40 [ 8057.136882] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.136884] [] ? insert_kthread_work+0x40/0x40 [ 8057.136905] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8057.146617] NMI watchdog: BUG: soft lockup - CPU#71 stuck for 22s! [ptlrpcd_01_34:16869] [ 8057.146646] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8057.146669] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8057.146672] CPU: 71 PID: 16869 Comm: ptlrpcd_01_34 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8057.146673] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8057.146674] task: ffff8f484d291080 ti: ffff8f484d29c000 task.ti: ffff8f484d29c000 [ 8057.146677] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8057.146678] RSP: 0018:ffff8f484d29fb58 EFLAGS: 00000246 [ 8057.146679] RAX: 0000000000000000 RBX: ffff8f6846c21f80 RCX: 0000000002390000 [ 8057.146680] RDX: ffff8f687f11b8c0 RSI: 0000000001c10001 RDI: ffff8f686e2b6b40 [ 8057.146681] RBP: ffff8f484d29fb58 R08: ffff8f687f4db8c0 R09: 0000000000000000 [ 8057.146682] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8057.146682] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000078ecee51 [ 8057.146684] FS: 0000000000000000(0000) GS:ffff8f687f4c0000(0000) knlGS:0000000000000000 [ 8057.146685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8057.146685] CR2: 00002aaaabc2e288 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8057.146686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8057.146687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8057.146687] Call Trace: [ 8057.146691] [] queued_spin_lock_slowpath+0xb/0xf [ 8057.146693] [] _raw_spin_lock+0x30/0x40 [ 8057.146701] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8057.146711] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8057.146743] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8057.146776] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8057.146806] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8057.146841] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8057.146844] [] ? wake_up_state+0x20/0x20 [ 8057.146878] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8057.146880] [] kthread+0xd1/0xe0 [ 8057.146883] [] ? insert_kthread_work+0x40/0x40 [ 8057.146885] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8057.146887] [] ? insert_kthread_work+0x40/0x40 [ 8057.146908] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8060.758528] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_00_13:16811] [ 8060.758559] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8060.758581] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8060.758584] CPU: 2 PID: 16811 Comm: ptlrpcd_00_13 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8060.758585] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8060.758586] task: ffff8f484c620000 ti: ffff8f484c628000 task.ti: ffff8f484c628000 [ 8060.758590] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8060.758591] RSP: 0018:ffff8f484c62bb58 EFLAGS: 00000246 [ 8060.758592] RAX: 0000000000000000 RBX: ffff8f47546c0480 RCX: 0000000000110000 [ 8060.758592] RDX: ffff8f487fc5b8c0 RSI: 0000000001990001 RDI: ffff8f686e2b6b40 [ 8060.758593] RBP: ffff8f484c62bb58 R08: ffff8f487f49b8c0 R09: 0000000000000000 [ 8060.758594] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8060.758595] R13: 0000000000000003 R14: 0000000000000013 R15: 000000005957ed38 [ 8060.758596] FS: 0000000000000000(0000) GS:ffff8f487f480000(0000) knlGS:0000000000000000 [ 8060.758597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.758598] CR2: 00007ffff7f84330 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8060.758599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8060.758600] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8060.758600] Call Trace: [ 8060.758603] [] queued_spin_lock_slowpath+0xb/0xf [ 8060.758606] [] _raw_spin_lock+0x30/0x40 [ 8060.758614] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8060.758624] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8060.758657] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8060.758660] [] ? del_timer_sync+0x52/0x60 [ 8060.758692] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8060.758722] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8060.758756] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8060.758759] [] ? wake_up_state+0x20/0x20 [ 8060.758793] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8060.758795] [] kthread+0xd1/0xe0 [ 8060.758797] [] ? insert_kthread_work+0x40/0x40 [ 8060.758799] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8060.758801] [] ? insert_kthread_work+0x40/0x40 [ 8060.758822] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8060.782527] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [ptlrpcd_00_15:16813] [ 8060.782556] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8060.782579] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8060.782581] CPU: 6 PID: 16813 Comm: ptlrpcd_00_15 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8060.782582] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8060.782583] task: ffff8f484c622100 ti: ffff8f484c630000 task.ti: ffff8f484c630000 [ 8060.782586] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x158/0x200 [ 8060.782587] RSP: 0018:ffff8f484c633b58 EFLAGS: 00000202 [ 8060.782588] RAX: 0000000000000001 RBX: ffff8f4659b82400 RCX: 0000000000310000 [ 8060.782589] RDX: 0000000000510001 RSI: 0000000000090001 RDI: ffff8f686e2b6b40 [ 8060.782590] RBP: ffff8f484c633b58 R08: ffff8f487f59b8c0 R09: ffff8f487f6db8c0 [ 8060.782590] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8060.782591] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000471bd2fd [ 8060.782592] FS: 0000000000000000(0000) GS:ffff8f487f580000(0000) knlGS:0000000000000000 [ 8060.782593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.782594] CR2: 00002aaaabaa0aa0 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8060.782595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8060.782596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8060.782596] Call Trace: [ 8060.782599] [] queued_spin_lock_slowpath+0xb/0xf [ 8060.782601] [] _raw_spin_lock+0x30/0x40 [ 8060.782609] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8060.782619] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8060.782651] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8060.782683] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8060.782713] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8060.782748] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8060.782750] [] ? wake_up_state+0x20/0x20 [ 8060.782784] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8060.782786] [] kthread+0xd1/0xe0 [ 8060.782789] [] ? insert_kthread_work+0x40/0x40 [ 8060.782791] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8060.782793] [] ? insert_kthread_work+0x40/0x40 [ 8060.782813] Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 [ 8060.812528] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [ptlrpcd_00_23:16821] [ 8060.812557] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8060.812579] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8060.812581] CPU: 11 PID: 16821 Comm: ptlrpcd_00_23 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8060.812582] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8060.812583] task: ffff8f484d3db180 ti: ffff8f484d3ec000 task.ti: ffff8f484d3ec000 [ 8060.812586] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8060.812587] RSP: 0018:ffff8f484d3efb58 EFLAGS: 00000246 [ 8060.812588] RAX: 0000000000000000 RBX: ffff8f465ec87080 RCX: 0000000000590000 [ 8060.812589] RDX: ffff8f487f59b8c0 RSI: 0000000000310001 RDI: ffff8f686e2b6b40 [ 8060.812590] RBP: ffff8f484d3efb58 R08: ffff8f487f6db8c0 R09: 0000000000000000 [ 8060.812591] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8060.812591] R13: 0000000000000003 R14: 0000000000000013 R15: 000000006d460fff [ 8060.812593] FS: 0000000000000000(0000) GS:ffff8f487f6c0000(0000) knlGS:0000000000000000 [ 8060.812594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.812594] CR2: 00002aaaabaa0aa0 CR3: 0000001f25f2e000 CR4: 00000000003607e0 [ 8060.812595] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8060.812596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8060.812596] Call Trace: [ 8060.812600] [] queued_spin_lock_slowpath+0xb/0xf [ 8060.812602] [] _raw_spin_lock+0x30/0x40 [ 8060.812610] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8060.812619] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8060.812651] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8060.812654] [] ? del_timer_sync+0x52/0x60 [ 8060.812686] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8060.812716] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8060.812750] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8060.812752] [] ? wake_up_state+0x20/0x20 [ 8060.812785] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8060.812788] [] kthread+0xd1/0xe0 [ 8060.812790] [] ? insert_kthread_work+0x40/0x40 [ 8060.812792] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8060.812794] [] ? insert_kthread_work+0x40/0x40 [ 8060.812814] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8061.089520] NMI watchdog: BUG: soft lockup - CPU#53 stuck for 22s! [ptlrpcd_00_21:16819] [ 8061.089550] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8061.089572] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8061.089575] CPU: 53 PID: 16819 Comm: ptlrpcd_00_21 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8061.089576] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8061.089577] task: ffff8f484d3d9080 ti: ffff8f484d3e4000 task.ti: ffff8f484d3e4000 [ 8061.089580] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8061.089581] RSP: 0018:ffff8f484d3e7b58 EFLAGS: 00000246 [ 8061.089582] RAX: 0000000000000000 RBX: ffff8f4659d10480 RCX: 0000000001a90000 [ 8061.089583] RDX: ffff8f687f21b8c0 RSI: 0000000001e10001 RDI: ffff8f686e2b6b40 [ 8061.089584] RBP: ffff8f484d3e7b58 R08: ffff8f487fcdb8c0 R09: 0000000000000000 [ 8061.089584] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8061.089585] R13: 0000000000000003 R14: 0000000000000013 R15: 000000009abfa125 [ 8061.089586] FS: 0000000000000000(0000) GS:ffff8f487fcc0000(0000) knlGS:0000000000000000 [ 8061.089587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8061.089588] CR2: 00002aaaab0fc0a0 CR3: 0000003ffd858000 CR4: 00000000003607e0 [ 8061.089589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8061.089590] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8061.089590] Call Trace: [ 8061.089593] [] queued_spin_lock_slowpath+0xb/0xf [ 8061.089595] [] _raw_spin_lock+0x30/0x40 [ 8061.089604] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8061.089614] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8061.089646] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8061.089649] [] ? del_timer_sync+0x52/0x60 [ 8061.089681] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8061.089713] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8061.089748] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8061.089751] [] ? wake_up_state+0x20/0x20 [ 8061.089784] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8061.089787] [] kthread+0xd1/0xe0 [ 8061.089789] [] ? insert_kthread_work+0x40/0x40 [ 8061.089791] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8061.089793] [] ? insert_kthread_work+0x40/0x40 [ 8061.089814] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8065.059424] NMI watchdog: BUG: soft lockup - CPU#43 stuck for 23s! [ptlrpcd_00_14:16812] [ 8065.059453] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8065.059476] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8065.059478] CPU: 43 PID: 16812 Comm: ptlrpcd_00_14 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8065.059479] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8065.059480] task: ffff8f484c621080 ti: ffff8f484c62c000 task.ti: ffff8f484c62c000 [ 8065.059484] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8065.059485] RSP: 0018:ffff8f484c62fb58 EFLAGS: 00000246 [ 8065.059486] RAX: 0000000000000000 RBX: ffff8f4744a93600 RCX: 0000000001590000 [ 8065.059487] RDX: ffff8f487f85b8c0 RSI: 0000000000890001 RDI: ffff8f686e2b6b40 [ 8065.059488] RBP: ffff8f484c62fb58 R08: ffff8f487fa5b8c0 R09: 0000000000000000 [ 8065.059489] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8065.059490] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e057be9c [ 8065.059491] FS: 0000000000000000(0000) GS:ffff8f487fa40000(0000) knlGS:0000000000000000 [ 8065.059492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8065.059493] CR2: 00002aaab4006338 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8065.059494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8065.059495] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8065.059495] Call Trace: [ 8065.059498] [] queued_spin_lock_slowpath+0xb/0xf [ 8065.059501] [] _raw_spin_lock+0x30/0x40 [ 8065.059509] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8065.059519] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8065.059551] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8065.059555] [] ? del_timer_sync+0x52/0x60 [ 8065.059586] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8065.059616] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8065.059651] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8065.059654] [] ? wake_up_state+0x20/0x20 [ 8065.059688] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8065.059690] [] kthread+0xd1/0xe0 [ 8065.059692] [] ? insert_kthread_work+0x40/0x40 [ 8065.059694] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8065.059696] [] ? insert_kthread_work+0x40/0x40 [ 8065.059717] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8068.422333] RAX: 0000000000000000 RBX: ffffffffffffff10 RCX: ffffb005d9675390 [ 8068.429890] RDX: ffff8f6692bc49a0 RSI: 00000000045b09cd RDI: ffff8f686e872a40 [ 8068.437448] RBP: ffff8f484f837ba0 R08: ffff8f487f45b8c0 R09: ffff8f487f59b8c0 [ 8068.445004] R10: 0000000000000000 R11: 000000000000000f R12: 0000000000990001 [ 8068.452561] R13: ffff8f687ec5b8c0 R14: 0000000000090000 R15: 0000000000000000 [ 8068.460118] FS: 0000000000000000(0000) GS:ffff8f487f440000(0000) knlGS:0000000000000000 [ 8068.468628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.474800] CR2: 00007ffff7ff8000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8068.482357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8068.489913] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8068.497471] Call Trace: [ 8068.500359] [] LNetMDUnlink+0xac/0x180 [lnet] [ 8068.506821] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8068.514469] [] ? del_timer_sync+0x52/0x60 [ 8068.520581] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8068.528346] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8068.535509] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8068.541768] [] ? wake_up_state+0x20/0x20 [ 8068.547796] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8068.554749] [] kthread+0xd1/0xe0 [ 8068.560053] [] ? insert_kthread_work+0x40/0x40 [ 8068.566570] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8068.573435] [] ? insert_kthread_work+0x40/0x40 [ 8068.579950] Code: 00 48 89 f2 83 c1 02 48 d3 ea 48 89 d1 81 e1 ff 0f 00 00 48 c1 e1 04 48 03 4f 20 48 8b 11 48 39 ca 75 10 eb 17 66 0f 1f 44 00 00 <48> 8b 12 48 39 ca 74 10 48 39 72 10 75 f2 48 89 d0 5d c3 0f 1f [ 8068.600519] Lustre: 16832:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1650660209/real 1650660234] req@ffff8f465e678000 x1730835639225664/t0(0) o103->aspls2-MDT0005-mdc-ffff8f686c694000@172.19.3.186@o2ib600:17/18 lens 328/224 e 0 to 1 dl 1650660315 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 8068.630620] Lustre: 16832:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 300677 previous similar messages [ 8068.776334] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ptlrpcd_00_26:16824] [ 8068.788333] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ptlrpcd_00_17:16815] [ 8068.784762] Modules linked in: mgc(OE) [ 8068.788334] Modules linked in: [ 8068.788334] mgc(OE) [ 8068.788335] lustre(OE) [ 8068.788336] lmv(OE) [ 8068.788336] mdc(OE) [ 8068.788337] osc(OE) [ 8068.788338] lov(OE) [ 8068.788338] fid(OE) [ 8068.788339] fld(OE) [ 8068.788340] ptlrpc(OE) [ 8068.788340] obdclass(OE) [ 8068.788341] ko2iblnd(OE) [ 8068.788341] lnet(OE) [ 8068.788342] libcfs(OE) [ 8068.788343] gdrdrv(POE) [ 8068.788344] iTCO_wdt [ 8068.788344] iTCO_vendor_support [ 8068.788345] rpcrdma [ 8068.788346] nvidia_drm(POE) [ 8068.788347] ib_iser [ 8068.788347] joydev [ 8068.788348] sb_edac [ 8068.788349] intel_powerclamp [ 8068.788349] coretemp [ 8068.788350] intel_rapl [ 8068.788351] iosf_mbi [ 8068.788352] kvm_intel [ 8068.788352] kvm [ 8068.788353] irqbypass [ 8068.788354] nvidia_modeset(POE) [ 8068.788354] sg [ 8068.788355] pcspkr [ 8068.788356] i2c_i801 [ 8068.788356] lpc_ich [ 8068.788357] nf_log_ipv4 [ 8068.788358] nf_log_common [ 8068.788358] xt_LOG [ 8068.788359] nf_conntrack_ipv4 [ 8068.788359] nf_defrag_ipv4 [ 8068.788360] xt_multiport [ 8068.788361] xt_owner [ 8068.788362] xt_conntrack [ 8068.788362] nf_conntrack [ 8068.788363] libcrc32c [ 8068.788364] iptable_filter [ 8068.788364] ipmi_si [ 8068.788365] ipmi_devintf [ 8068.788366] ipmi_msghandler [ 8068.788366] acpi_power_meter [ 8068.788367] ib_ipoib [ 8068.788367] rdma_ucm [ 8068.788368] ib_umad [ 8068.788369] iw_cxgb4 [ 8068.788370] rdma_cm [ 8068.788370] iw_cm [ 8068.788371] ib_cm [ 8068.788372] iw_cxgb3 [ 8068.788372] sch_fq_codel [ 8068.788373] binfmt_misc [ 8068.788374] msr_safe(OE) [ 8068.788374] ip_tables [ 8068.788375] nfsv3 [ 8068.788376] nfs_acl [ 8068.788376] rpcsec_gss_krb5 [ 8068.788377] auth_rpcgss [ 8068.788378] nfsv4 [ 8068.788378] dns_resolver [ 8068.788379] nfs [ 8068.788379] lockd [ 8068.788380] grace [ 8068.788381] fscache [ 8068.788382] overlay(T) [ 8068.788383] ext4 [ 8068.788383] mbcache [ 8068.788384] jbd2 [ 8068.788385] sd_mod [ 8068.788385] crc_t10dif [ 8068.788386] crct10dif_generic [ 8068.788387] nvidia_uvm(OE) [ 8068.788387] mlx5_ib [ 8068.788388] ib_uverbs [ 8068.788389] be2iscsi [ 8068.788389] ib_core [ 8068.788390] bnx2i [ 8068.788390] cnic [ 8068.788391] uio [ 8068.788392] cxgb4i [ 8068.788392] cxgb4 [ 8068.788393] cxgb3i [ 8068.788394] cxgb3 [ 8068.788394] mdio [ 8068.788395] libcxgbi [ 8068.788396] libcxgb [ 8068.788397] qla4xxx [ 8068.788397] iscsi_boot_sysfs [ 8068.788398] 8021q [ 8068.788398] garp [ 8068.788399] mrp [ 8068.788400] stp [ 8068.788401] llc [ 8068.788402] nvidia(POE) [ 8068.788402] ast [ 8068.788403] drm_kms_helper [ 8068.788403] crct10dif_pclmul [ 8068.788404] crct10dif_common [ 8068.788405] crc32_pclmul [ 8068.788406] crc32c_intel [ 8068.788406] syscopyarea [ 8068.788407] sysfillrect [ 8068.788407] sysimgblt [ 8068.788408] ghash_clmulni_intel [ 8068.788409] mlx5_core [ 8068.788409] fb_sys_fops [ 8068.788410] igb [ 8068.788411] ttm [ 8068.788411] aesni_intel [ 8068.788412] mlxfw [ 8068.788413] lrw [ 8068.788413] devlink [ 8068.788414] gf128mul [ 8068.788414] dca [ 8068.788415] glue_helper [ 8068.788416] ablk_helper [ 8068.788416] drm [ 8068.788417] dm_multipath [ 8068.788418] ptp [ 8068.788418] cryptd [ 8068.788419] i2c_algo_bit [ 8068.788420] pps_core [ 8068.788421] drm_panel_orientation_quirks [ 8068.788421] wmi [ 8068.788422] sunrpc [ 8068.788422] dm_mirror [ 8068.788423] dm_region_hash [ 8068.788424] dm_log [ 8068.788424] dm_mod [ 8068.788425] iscsi_tcp [ 8068.788426] libiscsi_tcp [ 8068.788427] libiscsi [ 8068.788427] scsi_transport_iscsi [ 8068.788428] fuse [ 8068.788428] [ 8068.788431] CPU: 7 PID: 16815 Comm: ptlrpcd_00_17 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8068.788432] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8068.788433] task: ffff8f484c624200 ti: ffff8f484d3c4000 task.ti: ffff8f484d3c4000 [ 8068.788434] RIP: 0010:[] [ 8068.788441] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8068.788442] RSP: 0018:ffff8f484d3c7b58 EFLAGS: 00000246 [ 8068.788442] RAX: 0000000000000000 RBX: ffff8f481d751680 RCX: 0000000000390000 [ 8068.788443] RDX: ffff8f487fa1b8c0 RSI: 0000000001510001 RDI: ffff8f686e2b6b40 [ 8068.788444] RBP: ffff8f484d3c7b58 R08: ffff8f487f5db8c0 R09: 0000000000000000 [ 8068.788445] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8068.788446] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000099c3dd79 [ 8068.788447] FS: 0000000000000000(0000) GS:ffff8f487f5c0000(0000) knlGS:0000000000000000 [ 8068.788449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.788449] CR2: 00002aaaabaa0aa0 CR3: 0000001fbf0b8000 CR4: 00000000003607e0 [ 8068.788451] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8068.788452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8068.788452] Call Trace: [ 8068.788458] [] queued_spin_lock_slowpath+0xb/0xf [ 8068.788462] [] _raw_spin_lock+0x30/0x40 [ 8068.788481] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8068.788499] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8068.788550] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8068.788555] [] ? del_timer_sync+0x52/0x60 [ 8068.788589] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8068.788620] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8068.788658] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8068.788661] [] ? wake_up_state+0x20/0x20 [ 8068.788694] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8068.788698] [] kthread+0xd1/0xe0 [ 8068.788700] [] ? insert_kthread_work+0x40/0x40 [ 8068.788703] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8068.788705] [] ? insert_kthread_work+0x40/0x40 [ 8068.788706] Code: [ 8068.788707] 0d [ 8068.788707] 48 [ 8068.788707] 98 [ 8068.788708] 83 [ 8068.788708] e2 [ 8068.788709] 30 [ 8068.788709] 48 [ 8068.788709] 81 [ 8068.788710] c2 [ 8068.788710] c0 [ 8068.788710] b8 [ 8068.788711] 01 [ 8068.788711] 00 [ 8068.788712] 48 [ 8068.788712] 03 [ 8068.788712] 14 [ 8068.788713] c5 [ 8068.788713] e0 [ 8068.788714] 17 [ 8068.788714] 15 [ 8068.788714] 91 [ 8068.788715] 4c [ 8068.788715] 89 [ 8068.788716] 02 [ 8068.788716] 41 [ 8068.788716] 8b [ 8068.788717] 40 [ 8068.788717] 08 [ 8068.788718] 85 [ 8068.788718] c0 [ 8068.788718] 75 [ 8068.788719] 0f [ 8068.788719] 0f [ 8068.788719] 1f [ 8068.788720] 44 [ 8068.788721] 00 [ 8068.788721] 00 [ 8068.788721] f3 [ 8068.788722] 90 [ 8068.788722] 41 [ 8068.788722] 8b [ 8068.788723] 40 [ 8068.788723] 08 [ 8068.788724] <85> [ 8068.788724] c0 [ 8068.788724] 74 [ 8068.788725] f6 [ 8068.788725] 4d [ 8068.788726] 8b [ 8068.788726] 08 [ 8068.788726] 4d [ 8068.788727] 85 [ 8068.788727] c9 [ 8068.788728] 74 [ 8068.788728] 04 [ 8068.788728] 41 [ 8068.788729] 0f [ 8068.788729] 18 [ 8068.788729] 09 [ 8068.788730] 8b [ 8068.788730] 17 [ 8068.788731] 0f [ 8068.788731] b7 [ 8068.788731] c2 [ 8068.788732] [ 8068.843332] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ptlrpcd_00_04:16802] [ 8068.843332] Modules linked in: [ 8068.843333] mgc(OE) [ 8068.843334] lustre(OE) [ 8068.843334] lmv(OE) [ 8068.843335] mdc(OE) [ 8068.843335] osc(OE) [ 8068.843335] lov(OE) [ 8068.843336] fid(OE) [ 8068.843336] fld(OE) [ 8068.843337] ptlrpc(OE) [ 8068.843337] obdclass(OE) [ 8068.843338] ko2iblnd(OE) [ 8068.843338] lnet(OE) [ 8068.843339] libcfs(OE) [ 8068.843339] gdrdrv(POE) [ 8068.843340] iTCO_wdt [ 8068.843340] iTCO_vendor_support [ 8068.843341] rpcrdma [ 8068.843342] nvidia_drm(POE) [ 8068.843342] ib_iser [ 8068.843342] joydev [ 8068.843343] sb_edac [ 8068.843343] intel_powerclamp [ 8068.843344] coretemp [ 8068.843344] intel_rapl [ 8068.843345] iosf_mbi [ 8068.843345] kvm_intel [ 8068.843346] kvm [ 8068.843346] irqbypass [ 8068.843347] nvidia_modeset(POE) [ 8068.843347] sg [ 8068.843348] pcspkr [ 8068.843348] i2c_i801 [ 8068.843349] lpc_ich [ 8068.843349] nf_log_ipv4 [ 8068.843350] nf_log_common [ 8068.843350] xt_LOG [ 8068.843351] nf_conntrack_ipv4 [ 8068.843351] nf_defrag_ipv4 [ 8068.843352] xt_multiport [ 8068.843352] xt_owner [ 8068.843353] xt_conntrack [ 8068.843353] nf_conntrack [ 8068.843354] libcrc32c [ 8068.843354] iptable_filter [ 8068.843354] ipmi_si [ 8068.843355] ipmi_devintf [ 8068.843355] ipmi_msghandler [ 8068.843356] acpi_power_meter [ 8068.843356] ib_ipoib [ 8068.843357] rdma_ucm [ 8068.843357] ib_umad [ 8068.843358] iw_cxgb4 [ 8068.843358] rdma_cm [ 8068.843359] iw_cm [ 8068.843359] ib_cm [ 8068.843359] iw_cxgb3 [ 8068.843360] sch_fq_codel [ 8068.843360] binfmt_misc [ 8068.843361] msr_safe(OE) [ 8068.843361] ip_tables [ 8068.843362] nfsv3 [ 8068.843362] nfs_acl [ 8068.843363] rpcsec_gss_krb5 [ 8068.843363] auth_rpcgss [ 8068.843364] nfsv4 [ 8068.843364] dns_resolver [ 8068.843365] nfs [ 8068.843365] lockd [ 8068.843365] grace [ 8068.843366] fscache [ 8068.843367] overlay(T) [ 8068.843367] ext4 [ 8068.843367] mbcache [ 8068.843368] jbd2 [ 8068.843368] sd_mod [ 8068.843369] crc_t10dif [ 8068.843369] crct10dif_generic [ 8068.843370] nvidia_uvm(OE) [ 8068.843370] mlx5_ib [ 8068.843371] ib_uverbs [ 8068.843371] be2iscsi [ 8068.843371] ib_core [ 8068.843372] bnx2i [ 8068.843372] cnic [ 8068.843373] uio [ 8068.843373] cxgb4i [ 8068.843374] cxgb4 [ 8068.843374] cxgb3i [ 8068.843374] cxgb3 [ 8068.843375] mdio [ 8068.843375] libcxgbi [ 8068.843376] libcxgb [ 8068.843376] qla4xxx [ 8068.843377] iscsi_boot_sysfs [ 8068.843377] 8021q [ 8068.843377] garp [ 8068.843378] mrp [ 8068.843378] stp [ 8068.843379] llc [ 8068.843379] nvidia(POE) [ 8068.843380] ast [ 8068.843380] drm_kms_helper [ 8068.843381] crct10dif_pclmul [ 8068.843381] crct10dif_common [ 8068.843381] crc32_pclmul [ 8068.843382] crc32c_intel [ 8068.843382] syscopyarea [ 8068.843383] sysfillrect [ 8068.843383] sysimgblt [ 8068.843384] ghash_clmulni_intel [ 8068.843384] mlx5_core [ 8068.843385] fb_sys_fops [ 8068.843385] igb [ 8068.843385] ttm [ 8068.843386] aesni_intel [ 8068.843386] mlxfw [ 8068.843387] lrw [ 8068.843387] devlink [ 8068.843388] gf128mul [ 8068.843388] dca [ 8068.843389] glue_helper [ 8068.843389] ablk_helper [ 8068.843389] drm [ 8068.843390] dm_multipath [ 8068.843390] ptp [ 8068.843391] cryptd [ 8068.843391] i2c_algo_bit [ 8068.843392] pps_core [ 8068.843392] drm_panel_orientation_quirks [ 8068.843393] wmi [ 8068.843393] sunrpc [ 8068.843394] dm_mirror [ 8068.843394] dm_region_hash [ 8068.843394] dm_log [ 8068.843395] dm_mod [ 8068.843395] iscsi_tcp [ 8068.843396] libiscsi_tcp [ 8068.843396] libiscsi [ 8068.843397] scsi_transport_iscsi [ 8068.843397] fuse [ 8068.843398] [ 8068.843400] CPU: 16 PID: 16802 Comm: ptlrpcd_00_04 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8068.843401] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8068.843402] task: ffff8f484fb7d280 ti: ffff8f484c648000 task.ti: ffff8f484c648000 [ 8068.843403] RIP: 0010:[] [ 8068.843406] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8068.843407] RSP: 0018:ffff8f484c64bb58 EFLAGS: 00000246 [ 8068.843408] RAX: 0000000000000000 RBX: ffff8f4713375a00 RCX: 0000000000810000 [ 8068.843409] RDX: ffff8f687f39b8c0 RSI: 0000000002110001 RDI: ffff8f686e2b6b40 [ 8068.843410] RBP: ffff8f484c64bb58 R08: ffff8f487f81b8c0 R09: 0000000000000000 [ 8068.843410] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8068.843411] R13: 0000000000000003 R14: 0000000000000013 R15: 000000005b63345b [ 8068.843413] FS: 0000000000000000(0000) GS:ffff8f487f800000(0000) knlGS:0000000000000000 [ 8068.843414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8068.843415] CR2: 00002aaaabaa0aa0 CR3: 0000003ff8218000 CR4: 00000000003607e0 [ 8068.843416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8068.843416] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8068.843417] Call Trace: [ 8068.843420] [] queued_spin_lock_slowpath+0xb/0xf [ 8068.843423] [] _raw_spin_lock+0x30/0x40 [ 8068.843431] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8068.843440] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8068.843473] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8068.843476] [] ? del_timer_sync+0x52/0x60 [ 8068.843507] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8068.843538] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8068.843572] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8068.843575] [] ? wake_up_state+0x20/0x20 [ 8068.843608] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8068.843610] [] kthread+0xd1/0xe0 [ 8068.843613] [] ? insert_kthread_work+0x40/0x40 [ 8068.843615] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8068.843617] [] ? insert_kthread_work+0x40/0x40 [ 8068.843617] Code: [ 8068.843618] 13 [ 8068.843619] 48 [ 8068.843619] c1 [ 8068.843619] ea [ 8068.843620] 0d [ 8068.843620] 48 [ 8068.843621] 98 [ 8068.843621] 83 [ 8068.843621] e2 [ 8068.843622] 30 [ 8068.843622] 48 [ 8068.843623] 81 [ 8068.843623] c2 [ 8068.843623] c0 [ 8068.843624] b8 [ 8068.843624] 01 [ 8068.843625] 00 [ 8068.843625] 48 [ 8068.843625] 03 [ 8068.843626] 14 [ 8068.843626] c5 [ 8068.843626] e0 [ 8068.843627] 17 [ 8068.843627] 15 [ 8068.843628] 91 [ 8068.843628] 4c [ 8068.843628] 89 [ 8068.843629] 02 [ 8068.843629] 41 [ 8068.843630] 8b [ 8068.843630] 40 [ 8068.843630] 08 [ 8068.843631] 85 [ 8068.843631] c0 [ 8068.843632] 75 [ 8068.843632] 0f [ 8068.843632] 0f [ 8068.843633] 1f [ 8068.843633] 44 [ 8068.843634] 00 [ 8068.843634] 00 [ 8068.843634] f3 [ 8068.843635] 90 [ 8068.843635] <41> [ 8068.843636] 8b [ 8068.843636] 40 [ 8068.843636] 08 [ 8068.843637] 85 [ 8068.843637] c0 [ 8068.843637] 74 [ 8068.843638] f6 [ 8068.843638] 4d [ 8068.843639] 8b [ 8068.843639] 08 [ 8068.843639] 4d [ 8068.843640] 85 [ 8068.843640] c9 [ 8068.843640] 74 [ 8068.843641] 04 [ 8068.843641] 41 [ 8068.843642] 0f [ 8068.843642] 18 [ 8068.843642] 09 [ 8068.843643] 8b [ 8068.843643] [ 8069.035327] NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [ptlrpcd_00_03:16801] [ 8069.035327] Modules linked in: [ 8069.035328] mgc(OE) [ 8069.035329] lustre(OE) [ 8069.035330] lmv(OE) [ 8069.035330] mdc(OE) [ 8069.035331] osc(OE) [ 8069.035331] lov(OE) [ 8069.035332] fid(OE) [ 8069.035332] fld(OE) [ 8069.035333] ptlrpc(OE) [ 8069.035333] obdclass(OE) [ 8069.035334] ko2iblnd(OE) [ 8069.035334] lnet(OE) [ 8069.035334] libcfs(OE) [ 8069.035335] gdrdrv(POE) [ 8069.035336] iTCO_wdt [ 8069.035336] iTCO_vendor_support [ 8069.035337] rpcrdma [ 8069.035337] nvidia_drm(POE) [ 8069.035338] ib_iser [ 8069.035338] joydev [ 8069.035338] sb_edac [ 8069.035339] intel_powerclamp [ 8069.035339] coretemp [ 8069.035340] intel_rapl [ 8069.035340] iosf_mbi [ 8069.035341] kvm_intel [ 8069.035341] kvm [ 8069.035342] irqbypass [ 8069.035342] nvidia_modeset(POE) [ 8069.035343] sg [ 8069.035343] pcspkr [ 8069.035344] i2c_i801 [ 8069.035344] lpc_ich [ 8069.035345] nf_log_ipv4 [ 8069.035345] nf_log_common [ 8069.035345] xt_LOG [ 8069.035346] nf_conntrack_ipv4 [ 8069.035346] nf_defrag_ipv4 [ 8069.035347] xt_multiport [ 8069.035347] xt_owner [ 8069.035348] xt_conntrack [ 8069.035348] nf_conntrack [ 8069.035349] libcrc32c [ 8069.035349] iptable_filter [ 8069.035350] ipmi_si [ 8069.035350] ipmi_devintf [ 8069.035351] ipmi_msghandler [ 8069.035351] acpi_power_meter [ 8069.035352] ib_ipoib [ 8069.035352] rdma_ucm [ 8069.035352] ib_umad [ 8069.035353] iw_cxgb4 [ 8069.035353] rdma_cm [ 8069.035354] iw_cm [ 8069.035354] ib_cm [ 8069.035355] iw_cxgb3 [ 8069.035355] sch_fq_codel [ 8069.035356] binfmt_misc [ 8069.035356] msr_safe(OE) [ 8069.035357] ip_tables [ 8069.035357] nfsv3 [ 8069.035358] nfs_acl [ 8069.035358] rpcsec_gss_krb5 [ 8069.035358] auth_rpcgss [ 8069.035359] nfsv4 [ 8069.035359] dns_resolver [ 8069.035360] nfs [ 8069.035360] lockd [ 8069.035361] grace [ 8069.035361] fscache [ 8069.035362] overlay(T) [ 8069.035362] ext4 [ 8069.035363] mbcache [ 8069.035363] jbd2 [ 8069.035364] sd_mod [ 8069.035364] crc_t10dif [ 8069.035365] crct10dif_generic [ 8069.035365] nvidia_uvm(OE) [ 8069.035366] mlx5_ib [ 8069.035366] ib_uverbs [ 8069.035367] be2iscsi [ 8069.035367] ib_core [ 8069.035368] bnx2i [ 8069.035368] cnic [ 8069.035368] uio [ 8069.035369] cxgb4i [ 8069.035369] cxgb4 [ 8069.035370] cxgb3i [ 8069.035370] cxgb3 [ 8069.035371] mdio [ 8069.035371] libcxgbi [ 8069.035371] libcxgb [ 8069.035372] qla4xxx [ 8069.035372] iscsi_boot_sysfs [ 8069.035373] 8021q [ 8069.035373] garp [ 8069.035373] mrp [ 8069.035374] stp [ 8069.035374] llc [ 8069.035375] nvidia(POE) [ 8069.035375] ast [ 8069.035376] drm_kms_helper [ 8069.035376] crct10dif_pclmul [ 8069.035377] crct10dif_common [ 8069.035377] crc32_pclmul [ 8069.035378] crc32c_intel [ 8069.035378] syscopyarea [ 8069.035379] sysfillrect [ 8069.035379] sysimgblt [ 8069.035380] ghash_clmulni_intel [ 8069.035380] mlx5_core [ 8069.035381] fb_sys_fops [ 8069.035381] igb [ 8069.035381] ttm [ 8069.035382] aesni_intel [ 8069.035382] mlxfw [ 8069.035383] lrw [ 8069.035383] devlink [ 8069.035384] gf128mul [ 8069.035384] dca [ 8069.035385] glue_helper [ 8069.035385] ablk_helper [ 8069.035386] drm [ 8069.035386] dm_multipath [ 8069.035387] ptp [ 8069.035387] cryptd [ 8069.035387] i2c_algo_bit [ 8069.035388] pps_core [ 8069.035388] drm_panel_orientation_quirks [ 8069.035389] wmi [ 8069.035389] sunrpc [ 8069.035390] dm_mirror [ 8069.035390] dm_region_hash [ 8069.035391] dm_log [ 8069.035391] dm_mod [ 8069.035392] iscsi_tcp [ 8069.035392] libiscsi_tcp [ 8069.035393] libiscsi [ 8069.035393] scsi_transport_iscsi [ 8069.035394] fuse [ 8069.035394] [ 8069.035396] CPU: 36 PID: 16801 Comm: ptlrpcd_00_03 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.035397] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.035398] task: ffff8f484fb7c200 ti: ffff8f484c668000 task.ti: ffff8f484c668000 [ 8069.035399] RIP: 0010:[] [ 8069.035402] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8069.035403] RSP: 0018:ffff8f484c66bb58 EFLAGS: 00000246 [ 8069.035404] RAX: 0000000000000000 RBX: ffff8f482bd86780 RCX: 0000000001210000 [ 8069.035405] RDX: ffff8f487f75b8c0 RSI: 0000000000690001 RDI: ffff8f686e2b6b40 [ 8069.035406] RBP: ffff8f484c66bb58 R08: ffff8f487f89b8c0 R09: 0000000000000000 [ 8069.035407] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.035408] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a2b8b603 [ 8069.035409] FS: 0000000000000000(0000) GS:ffff8f487f880000(0000) knlGS:0000000000000000 [ 8069.035410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.035411] CR2: 00002aaaab1139e5 CR3: 0000003ff84f6000 CR4: 00000000003607e0 [ 8069.035412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.035413] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.035413] Call Trace: [ 8069.035417] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.035419] [] _raw_spin_lock+0x30/0x40 [ 8069.035427] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.035438] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.035473] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.035476] [] ? del_timer_sync+0x52/0x60 [ 8069.035508] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.035539] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.035574] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.035577] [] ? wake_up_state+0x20/0x20 [ 8069.035610] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.035613] [] kthread+0xd1/0xe0 [ 8069.035615] [] ? insert_kthread_work+0x40/0x40 [ 8069.035617] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.035619] [] ? insert_kthread_work+0x40/0x40 [ 8069.035620] Code: [ 8069.035621] 0d [ 8069.035621] 48 [ 8069.035621] 98 [ 8069.035622] 83 [ 8069.035622] e2 [ 8069.035622] 30 [ 8069.035623] 48 [ 8069.035623] 81 [ 8069.035624] c2 [ 8069.035624] c0 [ 8069.035624] b8 [ 8069.035625] 01 [ 8069.035625] 00 [ 8069.035626] 48 [ 8069.035626] 03 [ 8069.035626] 14 [ 8069.035627] c5 [ 8069.035627] e0 [ 8069.035628] 17 [ 8069.035628] 15 [ 8069.035629] 91 [ 8069.035629] 4c [ 8069.035629] 89 [ 8069.035630] 02 [ 8069.035630] 41 [ 8069.035630] 8b [ 8069.035631] 40 [ 8069.035631] 08 [ 8069.035632] 85 [ 8069.035632] c0 [ 8069.035632] 75 [ 8069.035633] 0f [ 8069.035633] 0f [ 8069.035634] 1f [ 8069.035634] 44 [ 8069.035634] 00 [ 8069.035635] 00 [ 8069.035635] f3 [ 8069.035636] 90 [ 8069.035636] 41 [ 8069.035636] 8b [ 8069.035637] 40 [ 8069.035637] 08 [ 8069.035638] <85> [ 8069.035638] c0 [ 8069.035639] 74 [ 8069.035639] f6 [ 8069.035639] 4d [ 8069.035640] 8b [ 8069.035640] 08 [ 8069.035640] 4d [ 8069.035641] 85 [ 8069.035641] c9 [ 8069.035642] 74 [ 8069.035642] 04 [ 8069.035642] 41 [ 8069.035643] 0f [ 8069.035643] 18 [ 8069.035643] 09 [ 8069.035644] 8b [ 8069.035644] 17 [ 8069.035645] 0f [ 8069.035645] b7 [ 8069.035645] c2 [ 8069.035646] [ 8069.050326] NMI watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [ptlrpcd_00_29:16827] [ 8069.050327] Modules linked in: [ 8069.050327] mgc(OE) [ 8069.050328] lustre(OE) [ 8069.050328] lmv(OE) [ 8069.050328] mdc(OE) [ 8069.050329] osc(OE) [ 8069.050329] lov(OE) [ 8069.050329] fid(OE) [ 8069.050330] fld(OE) [ 8069.050330] ptlrpc(OE) [ 8069.050330] obdclass(OE) [ 8069.050331] ko2iblnd(OE) [ 8069.050331] lnet(OE) [ 8069.050331] libcfs(OE) [ 8069.050332] gdrdrv(POE) [ 8069.050332] iTCO_wdt [ 8069.050332] iTCO_vendor_support [ 8069.050333] rpcrdma [ 8069.050333] nvidia_drm(POE) [ 8069.050333] ib_iser [ 8069.050334] joydev [ 8069.050334] sb_edac [ 8069.050334] intel_powerclamp [ 8069.050334] coretemp [ 8069.050335] intel_rapl [ 8069.050335] iosf_mbi [ 8069.050335] kvm_intel [ 8069.050336] kvm [ 8069.050336] irqbypass [ 8069.050336] nvidia_modeset(POE) [ 8069.050337] sg [ 8069.050337] pcspkr [ 8069.050337] i2c_i801 [ 8069.050337] lpc_ich [ 8069.050338] nf_log_ipv4 [ 8069.050338] nf_log_common [ 8069.050338] xt_LOG [ 8069.050339] nf_conntrack_ipv4 [ 8069.050339] nf_defrag_ipv4 [ 8069.050339] xt_multiport [ 8069.050339] xt_owner [ 8069.050340] xt_conntrack [ 8069.050340] nf_conntrack [ 8069.050340] libcrc32c [ 8069.050341] iptable_filter [ 8069.050341] ipmi_si [ 8069.050341] ipmi_devintf [ 8069.050342] ipmi_msghandler [ 8069.050342] acpi_power_meter [ 8069.050342] ib_ipoib [ 8069.050342] rdma_ucm [ 8069.050343] ib_umad [ 8069.050343] iw_cxgb4 [ 8069.050343] rdma_cm [ 8069.050344] iw_cm [ 8069.050344] ib_cm [ 8069.050344] iw_cxgb3 [ 8069.050344] sch_fq_codel [ 8069.050345] binfmt_misc [ 8069.050345] msr_safe(OE) [ 8069.050346] ip_tables [ 8069.050346] nfsv3 [ 8069.050346] nfs_acl [ 8069.050346] rpcsec_gss_krb5 [ 8069.050347] auth_rpcgss [ 8069.050347] nfsv4 [ 8069.050347] dns_resolver [ 8069.050348] nfs [ 8069.050348] lockd [ 8069.050348] grace [ 8069.050348] fscache [ 8069.050349] overlay(T) [ 8069.050349] ext4 [ 8069.050349] mbcache [ 8069.050350] jbd2 [ 8069.050350] sd_mod [ 8069.050350] crc_t10dif [ 8069.050351] crct10dif_generic [ 8069.050351] nvidia_uvm(OE) [ 8069.050351] mlx5_ib [ 8069.050352] ib_uverbs [ 8069.050352] be2iscsi [ 8069.050352] ib_core [ 8069.050352] bnx2i [ 8069.050353] cnic [ 8069.050353] uio [ 8069.050354] cxgb4i [ 8069.050354] cxgb4 [ 8069.050354] cxgb3i [ 8069.050354] cxgb3 [ 8069.050355] mdio [ 8069.050355] libcxgbi [ 8069.050355] libcxgb [ 8069.050356] qla4xxx [ 8069.050356] iscsi_boot_sysfs [ 8069.050356] 8021q [ 8069.050356] garp [ 8069.050357] mrp [ 8069.050357] stp [ 8069.050357] llc [ 8069.050358] nvidia(POE) [ 8069.050358] ast [ 8069.050358] drm_kms_helper [ 8069.050359] crct10dif_pclmul [ 8069.050359] crct10dif_common [ 8069.050359] crc32_pclmul [ 8069.050360] crc32c_intel [ 8069.050360] syscopyarea [ 8069.050360] sysfillrect [ 8069.050360] sysimgblt [ 8069.050361] ghash_clmulni_intel [ 8069.050361] mlx5_core [ 8069.050361] fb_sys_fops [ 8069.050362] igb [ 8069.050362] ttm [ 8069.050362] aesni_intel [ 8069.050362] mlxfw [ 8069.050363] lrw [ 8069.050363] devlink [ 8069.050363] gf128mul [ 8069.050363] dca [ 8069.050364] glue_helper [ 8069.050364] ablk_helper [ 8069.050364] drm [ 8069.050365] dm_multipath [ 8069.050365] ptp [ 8069.050365] cryptd [ 8069.050366] i2c_algo_bit [ 8069.050366] pps_core [ 8069.050366] drm_panel_orientation_quirks [ 8069.050367] wmi [ 8069.050367] sunrpc [ 8069.050367] dm_mirror [ 8069.050367] dm_region_hash [ 8069.050368] dm_log [ 8069.050368] dm_mod [ 8069.050368] iscsi_tcp [ 8069.050369] libiscsi_tcp [ 8069.050369] libiscsi [ 8069.050369] scsi_transport_iscsi [ 8069.050369] fuse [ 8069.050370] [ 8069.050371] CPU: 40 PID: 16827 Comm: ptlrpcd_00_29 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.050372] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.050373] task: ffff8f484f80a100 ti: ffff8f484f814000 task.ti: ffff8f484f814000 [ 8069.050373] RIP: 0010:[] [ 8069.050376] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8069.050377] RSP: 0018:ffff8f484f817b58 EFLAGS: 00000246 [ 8069.050377] RAX: 0000000000000000 RBX: ffff8f477bdf5e80 RCX: 0000000001410000 [ 8069.050378] RDX: ffff8f687f25b8c0 RSI: 0000000001e90001 RDI: ffff8f686e2b6b40 [ 8069.050378] RBP: ffff8f484f817b58 R08: ffff8f487f99b8c0 R09: 0000000000000000 [ 8069.050379] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.050379] R13: 0000000000000003 R14: 0000000000000013 R15: 000000006c28f74f [ 8069.050380] FS: 0000000000000000(0000) GS:ffff8f487f980000(0000) knlGS:0000000000000000 [ 8069.050381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.050381] CR2: 00002aaaab0fc0a0 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8069.050382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.050383] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.050383] Call Trace: [ 8069.050386] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.050388] [] _raw_spin_lock+0x30/0x40 [ 8069.050394] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.050401] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.050425] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.050450] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.050471] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.050498] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.050500] [] ? wake_up_state+0x20/0x20 [ 8069.050525] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.050527] [] kthread+0xd1/0xe0 [ 8069.050528] [] ? insert_kthread_work+0x40/0x40 [ 8069.050530] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.050531] [] ? insert_kthread_work+0x40/0x40 [ 8069.050532] Code: [ 8069.050533] 0d [ 8069.050533] 48 [ 8069.050533] 98 [ 8069.050533] 83 [ 8069.050534] e2 [ 8069.050534] 30 [ 8069.050534] 48 [ 8069.050534] 81 [ 8069.050534] c2 [ 8069.050535] c0 [ 8069.050535] b8 [ 8069.050535] 01 [ 8069.050535] 00 [ 8069.050536] 48 [ 8069.050536] 03 [ 8069.050536] 14 [ 8069.050536] c5 [ 8069.050537] e0 [ 8069.050537] 17 [ 8069.050537] 15 [ 8069.050537] 91 [ 8069.050538] 4c [ 8069.050538] 89 [ 8069.050538] 02 [ 8069.050538] 41 [ 8069.050539] 8b [ 8069.050539] 40 [ 8069.050539] 08 [ 8069.050539] 85 [ 8069.050540] c0 [ 8069.050540] 75 [ 8069.050540] 0f [ 8069.050540] 0f [ 8069.050541] 1f [ 8069.050541] 44 [ 8069.050541] 00 [ 8069.050541] 00 [ 8069.050541] f3 [ 8069.050542] 90 [ 8069.050542] 41 [ 8069.050542] 8b [ 8069.050542] 40 [ 8069.050543] 08 [ 8069.050543] <85> [ 8069.050543] c0 [ 8069.050544] 74 [ 8069.050544] f6 [ 8069.050544] 4d [ 8069.050544] 8b [ 8069.050545] 08 [ 8069.050545] 4d [ 8069.050545] 85 [ 8069.050545] c9 [ 8069.050545] 74 [ 8069.050546] 04 [ 8069.050546] 41 [ 8069.050546] 0f [ 8069.050546] 18 [ 8069.050547] 09 [ 8069.050547] 8b [ 8069.050547] 17 [ 8069.050547] 0f [ 8069.050548] b7 [ 8069.050548] c2 [ 8069.050548] [ 8069.062326] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [ptlrpcd_00_28:16826] [ 8069.062327] Modules linked in: [ 8069.062328] mgc(OE) [ 8069.062328] lustre(OE) [ 8069.062329] lmv(OE) [ 8069.062329] mdc(OE) [ 8069.062330] osc(OE) [ 8069.062330] lov(OE) [ 8069.062331] fid(OE) [ 8069.062331] fld(OE) [ 8069.062332] ptlrpc(OE) [ 8069.062332] obdclass(OE) [ 8069.062333] ko2iblnd(OE) [ 8069.062333] lnet(OE) [ 8069.062334] libcfs(OE) [ 8069.062334] gdrdrv(POE) [ 8069.062335] iTCO_wdt [ 8069.062335] iTCO_vendor_support [ 8069.062336] rpcrdma [ 8069.062336] nvidia_drm(POE) [ 8069.062337] ib_iser [ 8069.062337] joydev [ 8069.062338] sb_edac [ 8069.062338] intel_powerclamp [ 8069.062338] coretemp [ 8069.062339] intel_rapl [ 8069.062339] iosf_mbi [ 8069.062340] kvm_intel [ 8069.062340] kvm [ 8069.062341] irqbypass [ 8069.062341] nvidia_modeset(POE) [ 8069.062342] sg [ 8069.062342] pcspkr [ 8069.062342] i2c_i801 [ 8069.062343] lpc_ich [ 8069.062343] nf_log_ipv4 [ 8069.062344] nf_log_common [ 8069.062344] xt_LOG [ 8069.062345] nf_conntrack_ipv4 [ 8069.062345] nf_defrag_ipv4 [ 8069.062346] xt_multiport [ 8069.062346] xt_owner [ 8069.062346] xt_conntrack [ 8069.062347] nf_conntrack [ 8069.062347] libcrc32c [ 8069.062348] iptable_filter [ 8069.062348] ipmi_si [ 8069.062349] ipmi_devintf [ 8069.062350] ipmi_msghandler [ 8069.062350] acpi_power_meter [ 8069.062350] ib_ipoib [ 8069.062351] rdma_ucm [ 8069.062351] ib_umad [ 8069.062352] iw_cxgb4 [ 8069.062352] rdma_cm [ 8069.062353] iw_cm [ 8069.062353] ib_cm [ 8069.062354] iw_cxgb3 [ 8069.062354] sch_fq_codel [ 8069.062354] binfmt_misc [ 8069.062355] msr_safe(OE) [ 8069.062355] ip_tables [ 8069.062356] nfsv3 [ 8069.062356] nfs_acl [ 8069.062357] rpcsec_gss_krb5 [ 8069.062357] auth_rpcgss [ 8069.062358] nfsv4 [ 8069.062358] dns_resolver [ 8069.062359] nfs [ 8069.062359] lockd [ 8069.062359] grace [ 8069.062360] fscache [ 8069.062361] overlay(T) [ 8069.062361] ext4 [ 8069.062361] mbcache [ 8069.062362] jbd2 [ 8069.062362] sd_mod [ 8069.062363] crc_t10dif [ 8069.062363] crct10dif_generic [ 8069.062364] nvidia_uvm(OE) [ 8069.062364] mlx5_ib [ 8069.062365] ib_uverbs [ 8069.062365] be2iscsi [ 8069.062366] ib_core [ 8069.062366] bnx2i [ 8069.062366] cnic [ 8069.062367] uio [ 8069.062367] cxgb4i [ 8069.062368] cxgb4 [ 8069.062368] cxgb3i [ 8069.062369] cxgb3 [ 8069.062369] mdio [ 8069.062370] libcxgbi [ 8069.062370] libcxgb [ 8069.062370] qla4xxx [ 8069.062371] iscsi_boot_sysfs [ 8069.062371] 8021q [ 8069.062372] garp [ 8069.062372] mrp [ 8069.062373] stp [ 8069.062373] llc [ 8069.062374] nvidia(POE) [ 8069.062374] ast [ 8069.062374] drm_kms_helper [ 8069.062375] crct10dif_pclmul [ 8069.062375] crct10dif_common [ 8069.062376] crc32_pclmul [ 8069.062376] crc32c_intel [ 8069.062377] syscopyarea [ 8069.062377] sysfillrect [ 8069.062378] sysimgblt [ 8069.062378] ghash_clmulni_intel [ 8069.062379] mlx5_core [ 8069.062379] fb_sys_fops [ 8069.062380] igb [ 8069.062380] ttm [ 8069.062380] aesni_intel [ 8069.062381] mlxfw [ 8069.062381] lrw [ 8069.062382] devlink [ 8069.062382] gf128mul [ 8069.062383] dca [ 8069.062383] glue_helper [ 8069.062383] ablk_helper [ 8069.062384] drm [ 8069.062384] dm_multipath [ 8069.062385] ptp [ 8069.062385] cryptd [ 8069.062386] i2c_algo_bit [ 8069.062386] pps_core [ 8069.062387] drm_panel_orientation_quirks [ 8069.062387] wmi [ 8069.062388] sunrpc [ 8069.062388] dm_mirror [ 8069.062388] dm_region_hash [ 8069.062389] dm_log [ 8069.062389] dm_mod [ 8069.062390] iscsi_tcp [ 8069.062390] libiscsi_tcp [ 8069.062391] libiscsi [ 8069.062391] scsi_transport_iscsi [ 8069.062392] fuse [ 8069.062392] [ 8069.062394] CPU: 44 PID: 16826 Comm: ptlrpcd_00_28 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.062395] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.062396] task: ffff8f484f809080 ti: ffff8f484f810000 task.ti: ffff8f484f810000 [ 8069.062397] RIP: 0010:[] [ 8069.062400] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8069.062401] RSP: 0018:ffff8f484f813b58 EFLAGS: 00000246 [ 8069.062402] RAX: 0000000000000000 RBX: ffff8f4713fbf080 RCX: 0000000001610000 [ 8069.062403] RDX: ffff8f687f19b8c0 RSI: 0000000001d10001 RDI: ffff8f686e2b6b40 [ 8069.062404] RBP: ffff8f484f813b58 R08: ffff8f487fa9b8c0 R09: 0000000000000000 [ 8069.062405] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.062406] R13: 0000000000000003 R14: 0000000000000013 R15: 000000008d8972f1 [ 8069.062407] FS: 0000000000000000(0000) GS:ffff8f487fa80000(0000) knlGS:0000000000000000 [ 8069.062408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.062409] CR2: 00007ffff7ff8000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8069.062410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.062411] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.062411] Call Trace: [ 8069.062415] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.062417] [] _raw_spin_lock+0x30/0x40 [ 8069.062425] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.062435] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.062467] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.062500] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.062530] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.062565] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.062568] [] ? wake_up_state+0x20/0x20 [ 8069.062601] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.062603] [] kthread+0xd1/0xe0 [ 8069.062606] [] ? insert_kthread_work+0x40/0x40 [ 8069.062608] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.062610] [] ? insert_kthread_work+0x40/0x40 [ 8069.062611] Code: [ 8069.062611] 0d [ 8069.062612] 48 [ 8069.062612] 98 [ 8069.062612] 83 [ 8069.062613] e2 [ 8069.062613] 30 [ 8069.062614] 48 [ 8069.062614] 81 [ 8069.062614] c2 [ 8069.062615] c0 [ 8069.062615] b8 [ 8069.062615] 01 [ 8069.062616] 00 [ 8069.062616] 48 [ 8069.062617] 03 [ 8069.062617] 14 [ 8069.062617] c5 [ 8069.062618] e0 [ 8069.062618] 17 [ 8069.062619] 15 [ 8069.062619] 91 [ 8069.062619] 4c [ 8069.062620] 89 [ 8069.062620] 02 [ 8069.062621] 41 [ 8069.062621] 8b [ 8069.062621] 40 [ 8069.062622] 08 [ 8069.062622] 85 [ 8069.062623] c0 [ 8069.062623] 75 [ 8069.062623] 0f [ 8069.062624] 0f [ 8069.062624] 1f [ 8069.062625] 44 [ 8069.062625] 00 [ 8069.062625] 00 [ 8069.062626] f3 [ 8069.062626] 90 [ 8069.062627] 41 [ 8069.062627] 8b [ 8069.062627] 40 [ 8069.062628] 08 [ 8069.062628] <85> [ 8069.062629] c0 [ 8069.062629] 74 [ 8069.062630] f6 [ 8069.062630] 4d [ 8069.062630] 8b [ 8069.062631] 08 [ 8069.062631] 4d [ 8069.062631] 85 [ 8069.062632] c9 [ 8069.062632] 74 [ 8069.062633] 04 [ 8069.062633] 41 [ 8069.062633] 0f [ 8069.062634] 18 [ 8069.062634] 09 [ 8069.062635] 8b [ 8069.062635] 17 [ 8069.062635] 0f [ 8069.062636] b7 [ 8069.062636] c2 [ 8069.062637] [ 8069.068326] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [ptlrpcd_00_07:16805] [ 8069.068327] Modules linked in: [ 8069.068328] mgc(OE) [ 8069.068328] lustre(OE) [ 8069.068329] lmv(OE) [ 8069.068329] mdc(OE) [ 8069.068330] osc(OE) [ 8069.068330] lov(OE) [ 8069.068331] fid(OE) [ 8069.068331] fld(OE) [ 8069.068332] ptlrpc(OE) [ 8069.068332] obdclass(OE) [ 8069.068333] ko2iblnd(OE) [ 8069.068333] lnet(OE) [ 8069.068333] libcfs(OE) [ 8069.068334] gdrdrv(POE) [ 8069.068335] iTCO_wdt [ 8069.068335] iTCO_vendor_support [ 8069.068335] rpcrdma [ 8069.068336] nvidia_drm(POE) [ 8069.068337] ib_iser [ 8069.068337] joydev [ 8069.068337] sb_edac [ 8069.068338] intel_powerclamp [ 8069.068338] coretemp [ 8069.068339] intel_rapl [ 8069.068339] iosf_mbi [ 8069.068340] kvm_intel [ 8069.068340] kvm [ 8069.068340] irqbypass [ 8069.068341] nvidia_modeset(POE) [ 8069.068342] sg [ 8069.068342] pcspkr [ 8069.068342] i2c_i801 [ 8069.068343] lpc_ich [ 8069.068343] nf_log_ipv4 [ 8069.068344] nf_log_common [ 8069.068344] xt_LOG [ 8069.068345] nf_conntrack_ipv4 [ 8069.068345] nf_defrag_ipv4 [ 8069.068346] xt_multiport [ 8069.068346] xt_owner [ 8069.068346] xt_conntrack [ 8069.068347] nf_conntrack [ 8069.068347] libcrc32c [ 8069.068348] iptable_filter [ 8069.068348] ipmi_si [ 8069.068349] ipmi_devintf [ 8069.068349] ipmi_msghandler [ 8069.068350] acpi_power_meter [ 8069.068350] ib_ipoib [ 8069.068351] rdma_ucm [ 8069.068351] ib_umad [ 8069.068351] iw_cxgb4 [ 8069.068352] rdma_cm [ 8069.068352] iw_cm [ 8069.068353] ib_cm [ 8069.068353] iw_cxgb3 [ 8069.068354] sch_fq_codel [ 8069.068354] binfmt_misc [ 8069.068355] msr_safe(OE) [ 8069.068355] ip_tables [ 8069.068356] nfsv3 [ 8069.068356] nfs_acl [ 8069.068357] rpcsec_gss_krb5 [ 8069.068357] auth_rpcgss [ 8069.068357] nfsv4 [ 8069.068358] dns_resolver [ 8069.068358] nfs [ 8069.068359] lockd [ 8069.068359] grace [ 8069.068360] fscache [ 8069.068360] overlay(T) [ 8069.068361] ext4 [ 8069.068361] mbcache [ 8069.068361] jbd2 [ 8069.068362] sd_mod [ 8069.068362] crc_t10dif [ 8069.068363] crct10dif_generic [ 8069.068363] nvidia_uvm(OE) [ 8069.068364] mlx5_ib [ 8069.068364] ib_uverbs [ 8069.068365] be2iscsi [ 8069.068365] ib_core [ 8069.068366] bnx2i [ 8069.068366] cnic [ 8069.068367] uio [ 8069.068367] cxgb4i [ 8069.068367] cxgb4 [ 8069.068368] cxgb3i [ 8069.068368] cxgb3 [ 8069.068369] mdio [ 8069.068369] libcxgbi [ 8069.068370] libcxgb [ 8069.068370] qla4xxx [ 8069.068370] iscsi_boot_sysfs [ 8069.068371] 8021q [ 8069.068371] garp [ 8069.068372] mrp [ 8069.068372] stp [ 8069.068373] llc [ 8069.068373] nvidia(POE) [ 8069.068374] ast [ 8069.068374] drm_kms_helper [ 8069.068375] crct10dif_pclmul [ 8069.068375] crct10dif_common [ 8069.068376] crc32_pclmul [ 8069.068376] crc32c_intel [ 8069.068376] syscopyarea [ 8069.068377] sysfillrect [ 8069.068377] sysimgblt [ 8069.068378] ghash_clmulni_intel [ 8069.068378] mlx5_core [ 8069.068379] fb_sys_fops [ 8069.068379] igb [ 8069.068380] ttm [ 8069.068380] aesni_intel [ 8069.068380] mlxfw [ 8069.068381] lrw [ 8069.068381] devlink [ 8069.068382] gf128mul [ 8069.068382] dca [ 8069.068383] glue_helper [ 8069.068383] ablk_helper [ 8069.068383] drm [ 8069.068384] dm_multipath [ 8069.068384] ptp [ 8069.068385] cryptd [ 8069.068385] i2c_algo_bit [ 8069.068386] pps_core [ 8069.068386] drm_panel_orientation_quirks [ 8069.068387] wmi [ 8069.068387] sunrpc [ 8069.068388] dm_mirror [ 8069.068388] dm_region_hash [ 8069.068388] dm_log [ 8069.068389] dm_mod [ 8069.068389] iscsi_tcp [ 8069.068390] libiscsi_tcp [ 8069.068390] libiscsi [ 8069.068391] scsi_transport_iscsi [ 8069.068391] fuse [ 8069.068391] [ 8069.068394] CPU: 46 PID: 16805 Comm: ptlrpcd_00_07 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.068395] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.068396] task: ffff8f484c679080 ti: ffff8f484c654000 task.ti: ffff8f484c654000 [ 8069.068397] RIP: 0010:[] [ 8069.068400] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8069.068401] RSP: 0018:ffff8f484c657b58 EFLAGS: 00000246 [ 8069.068402] RAX: 0000000000000000 RBX: ffff8f47246ae780 RCX: 0000000001710000 [ 8069.068403] RDX: ffff8f487fb9b8c0 RSI: 0000000001810000 RDI: ffff8f686e2b6b40 [ 8069.068404] RBP: ffff8f484c657b58 R08: ffff8f487fb1b8c0 R09: 0000000000000000 [ 8069.068404] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.068405] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000af1fa668 [ 8069.068407] FS: 0000000000000000(0000) GS:ffff8f487fb00000(0000) knlGS:0000000000000000 [ 8069.068408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.068408] CR2: 00002aaaab1139e5 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8069.068409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.068410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.068411] Call Trace: [ 8069.068414] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.068416] [] _raw_spin_lock+0x30/0x40 [ 8069.068424] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.068434] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.068467] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.068470] [] ? del_timer_sync+0x52/0x60 [ 8069.068501] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.068532] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.068566] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.068569] [] ? wake_up_state+0x20/0x20 [ 8069.068602] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.068604] [] kthread+0xd1/0xe0 [ 8069.068607] [] ? insert_kthread_work+0x40/0x40 [ 8069.068608] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.068610] [] ? insert_kthread_work+0x40/0x40 [ 8069.068611] Code: [ 8069.068612] 13 [ 8069.068613] 48 [ 8069.068613] c1 [ 8069.068613] ea [ 8069.068614] 0d [ 8069.068614] 48 [ 8069.068614] 98 [ 8069.068615] 83 [ 8069.068615] e2 [ 8069.068616] 30 [ 8069.068616] 48 [ 8069.068616] 81 [ 8069.068617] c2 [ 8069.068617] c0 [ 8069.068618] b8 [ 8069.068618] 01 [ 8069.068618] 00 [ 8069.068619] 48 [ 8069.068619] 03 [ 8069.068620] 14 [ 8069.068620] c5 [ 8069.068620] e0 [ 8069.068621] 17 [ 8069.068621] 15 [ 8069.068622] 91 [ 8069.068622] 4c [ 8069.068622] 89 [ 8069.068623] 02 [ 8069.068623] 41 [ 8069.068624] 8b [ 8069.068624] 40 [ 8069.068624] 08 [ 8069.068625] 85 [ 8069.068625] c0 [ 8069.068626] 75 [ 8069.068626] 0f [ 8069.068626] 0f [ 8069.068627] 1f [ 8069.068627] 44 [ 8069.068628] 00 [ 8069.068628] 00 [ 8069.068628] f3 [ 8069.068629] 90 [ 8069.068629] <41> [ 8069.068630] 8b [ 8069.068630] 40 [ 8069.068630] 08 [ 8069.068631] 85 [ 8069.068631] c0 [ 8069.068632] 74 [ 8069.068632] f6 [ 8069.068632] 4d [ 8069.068633] 8b [ 8069.068633] 08 [ 8069.068634] 4d [ 8069.068634] 85 [ 8069.068634] c9 [ 8069.068635] 74 [ 8069.068635] 04 [ 8069.068636] 41 [ 8069.068636] 0f [ 8069.068636] 18 [ 8069.068637] 09 [ 8069.068637] 8b [ 8069.068637] [ 8069.080325] NMI watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [ptlrpcd_00_20:16818] [ 8069.080326] Modules linked in: [ 8069.080327] mgc(OE) [ 8069.080327] lustre(OE) [ 8069.080328] lmv(OE) [ 8069.080328] mdc(OE) [ 8069.080329] osc(OE) [ 8069.080329] lov(OE) [ 8069.080330] fid(OE) [ 8069.080330] fld(OE) [ 8069.080331] ptlrpc(OE) [ 8069.080331] obdclass(OE) [ 8069.080332] ko2iblnd(OE) [ 8069.080332] lnet(OE) [ 8069.080333] libcfs(OE) [ 8069.080333] gdrdrv(POE) [ 8069.080334] iTCO_wdt [ 8069.080334] iTCO_vendor_support [ 8069.080335] rpcrdma [ 8069.080335] nvidia_drm(POE) [ 8069.080336] ib_iser [ 8069.080336] joydev [ 8069.080337] sb_edac [ 8069.080337] intel_powerclamp [ 8069.080338] coretemp [ 8069.080338] intel_rapl [ 8069.080339] iosf_mbi [ 8069.080339] kvm_intel [ 8069.080339] kvm [ 8069.080340] irqbypass [ 8069.080340] nvidia_modeset(POE) [ 8069.080341] sg [ 8069.080341] pcspkr [ 8069.080342] i2c_i801 [ 8069.080342] lpc_ich [ 8069.080342] nf_log_ipv4 [ 8069.080343] nf_log_common [ 8069.080343] xt_LOG [ 8069.080344] nf_conntrack_ipv4 [ 8069.080344] nf_defrag_ipv4 [ 8069.080345] xt_multiport [ 8069.080345] xt_owner [ 8069.080346] xt_conntrack [ 8069.080346] nf_conntrack [ 8069.080346] libcrc32c [ 8069.080347] iptable_filter [ 8069.080347] ipmi_si [ 8069.080348] ipmi_devintf [ 8069.080348] ipmi_msghandler [ 8069.080349] acpi_power_meter [ 8069.080349] ib_ipoib [ 8069.080350] rdma_ucm [ 8069.080350] ib_umad [ 8069.080351] iw_cxgb4 [ 8069.080351] rdma_cm [ 8069.080352] iw_cm [ 8069.080352] ib_cm [ 8069.080352] iw_cxgb3 [ 8069.080353] sch_fq_codel [ 8069.080353] binfmt_misc [ 8069.080354] msr_safe(OE) [ 8069.080354] ip_tables [ 8069.080355] nfsv3 [ 8069.080355] nfs_acl [ 8069.080356] rpcsec_gss_krb5 [ 8069.080356] auth_rpcgss [ 8069.080357] nfsv4 [ 8069.080357] dns_resolver [ 8069.080358] nfs [ 8069.080358] lockd [ 8069.080358] grace [ 8069.080359] fscache [ 8069.080359] overlay(T) [ 8069.080360] ext4 [ 8069.080360] mbcache [ 8069.080361] jbd2 [ 8069.080361] sd_mod [ 8069.080362] crc_t10dif [ 8069.080362] crct10dif_generic [ 8069.080363] nvidia_uvm(OE) [ 8069.080363] mlx5_ib [ 8069.080363] ib_uverbs [ 8069.080364] be2iscsi [ 8069.080364] ib_core [ 8069.080365] bnx2i [ 8069.080365] cnic [ 8069.080366] uio [ 8069.080366] cxgb4i [ 8069.080367] cxgb4 [ 8069.080367] cxgb3i [ 8069.080368] cxgb3 [ 8069.080368] mdio [ 8069.080369] libcxgbi [ 8069.080369] libcxgb [ 8069.080369] qla4xxx [ 8069.080370] iscsi_boot_sysfs [ 8069.080370] 8021q [ 8069.080371] garp [ 8069.080371] mrp [ 8069.080372] stp [ 8069.080372] llc [ 8069.080373] nvidia(POE) [ 8069.080373] ast [ 8069.080374] drm_kms_helper [ 8069.080374] crct10dif_pclmul [ 8069.080374] crct10dif_common [ 8069.080375] crc32_pclmul [ 8069.080375] crc32c_intel [ 8069.080376] syscopyarea [ 8069.080376] sysfillrect [ 8069.080377] sysimgblt [ 8069.080377] ghash_clmulni_intel [ 8069.080378] mlx5_core [ 8069.080378] fb_sys_fops [ 8069.080379] igb [ 8069.080379] ttm [ 8069.080379] aesni_intel [ 8069.080380] mlxfw [ 8069.080380] lrw [ 8069.080381] devlink [ 8069.080381] gf128mul [ 8069.080382] dca [ 8069.080382] glue_helper [ 8069.080382] ablk_helper [ 8069.080383] drm [ 8069.080383] dm_multipath [ 8069.080384] ptp [ 8069.080384] cryptd [ 8069.080385] i2c_algo_bit [ 8069.080385] pps_core [ 8069.080386] drm_panel_orientation_quirks [ 8069.080386] wmi [ 8069.080386] sunrpc [ 8069.080387] dm_mirror [ 8069.080387] dm_region_hash [ 8069.080388] dm_log [ 8069.080388] dm_mod [ 8069.080389] iscsi_tcp [ 8069.080389] libiscsi_tcp [ 8069.080390] libiscsi [ 8069.080390] scsi_transport_iscsi [ 8069.080391] fuse [ 8069.080391] [ 8069.080393] CPU: 50 PID: 16818 Comm: ptlrpcd_00_20 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.080394] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.080395] task: ffff8f484d3d8000 ti: ffff8f484d3e0000 task.ti: ffff8f484d3e0000 [ 8069.080396] RIP: 0010:[] [ 8069.080399] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8069.080400] RSP: 0018:ffff8f484d3e3b58 EFLAGS: 00000246 [ 8069.080401] RAX: 0000000000000000 RBX: ffff8f671301f080 RCX: 0000000001910000 [ 8069.080402] RDX: ffff8f487f89b8c0 RSI: 0000000001210001 RDI: ffff8f686e2b6b40 [ 8069.080403] RBP: ffff8f484d3e3b58 R08: ffff8f487fc1b8c0 R09: 0000000000000000 [ 8069.080404] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.080405] R13: 0000000000000003 R14: 0000000000000013 R15: 000000000dcee365 [ 8069.080406] FS: 0000000000000000(0000) GS:ffff8f487fc00000(0000) knlGS:0000000000000000 [ 8069.080407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.080408] CR2: 00007fe199cff000 CR3: 0000003ffe4dc000 CR4: 00000000003607e0 [ 8069.080409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.080410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.080410] Call Trace: [ 8069.080414] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.080416] [] _raw_spin_lock+0x30/0x40 [ 8069.080424] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.080434] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.080466] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.080499] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.080529] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.080564] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.080567] [] ? wake_up_state+0x20/0x20 [ 8069.080600] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.080602] [] kthread+0xd1/0xe0 [ 8069.080605] [] ? insert_kthread_work+0x40/0x40 [ 8069.080607] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.080609] [] ? insert_kthread_work+0x40/0x40 [ 8069.080609] Code: [ 8069.080610] 0d [ 8069.080611] 48 [ 8069.080611] 98 [ 8069.080611] 83 [ 8069.080612] e2 [ 8069.080612] 30 [ 8069.080613] 48 [ 8069.080613] 81 [ 8069.080613] c2 [ 8069.080614] c0 [ 8069.080614] b8 [ 8069.080615] 01 [ 8069.080615] 00 [ 8069.080615] 48 [ 8069.080616] 03 [ 8069.080616] 14 [ 8069.080617] c5 [ 8069.080617] e0 [ 8069.080617] 17 [ 8069.080618] 15 [ 8069.080618] 91 [ 8069.080619] 4c [ 8069.080619] 89 [ 8069.080619] 02 [ 8069.080620] 41 [ 8069.080620] 8b [ 8069.080621] 40 [ 8069.080621] 08 [ 8069.080621] 85 [ 8069.080622] c0 [ 8069.080622] 75 [ 8069.080623] 0f [ 8069.080623] 0f [ 8069.080623] 1f [ 8069.080624] 44 [ 8069.080624] 00 [ 8069.080624] 00 [ 8069.080625] f3 [ 8069.080625] 90 [ 8069.080626] 41 [ 8069.080626] 8b [ 8069.080627] 40 [ 8069.080627] 08 [ 8069.080627] <85> [ 8069.080628] c0 [ 8069.080628] 74 [ 8069.080629] f6 [ 8069.080629] 4d [ 8069.080629] 8b [ 8069.080630] 08 [ 8069.080630] 4d [ 8069.080630] 85 [ 8069.080631] c9 [ 8069.080631] 74 [ 8069.080632] 04 [ 8069.080632] 41 [ 8069.080632] 0f [ 8069.080633] 18 [ 8069.080633] 09 [ 8069.080633] 8b [ 8069.080634] 17 [ 8069.080634] 0f [ 8069.080635] b7 [ 8069.080635] c2 [ 8069.080635] [ 8069.083327] NMI watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [ptlrpcd_00_16:16814] [ 8069.083327] Modules linked in: [ 8069.083328] mgc(OE) [ 8069.083329] lustre(OE) [ 8069.083329] lmv(OE) [ 8069.083330] mdc(OE) [ 8069.083330] osc(OE) [ 8069.083331] lov(OE) [ 8069.083331] fid(OE) [ 8069.083332] fld(OE) [ 8069.083332] ptlrpc(OE) [ 8069.083333] obdclass(OE) [ 8069.083333] ko2iblnd(OE) [ 8069.083334] lnet(OE) [ 8069.083334] libcfs(OE) [ 8069.083335] gdrdrv(POE) [ 8069.083335] iTCO_wdt [ 8069.083336] iTCO_vendor_support [ 8069.083336] rpcrdma [ 8069.083337] nvidia_drm(POE) [ 8069.083337] ib_iser [ 8069.083337] joydev [ 8069.083338] sb_edac [ 8069.083338] intel_powerclamp [ 8069.083339] coretemp [ 8069.083339] intel_rapl [ 8069.083340] iosf_mbi [ 8069.083340] kvm_intel [ 8069.083340] kvm [ 8069.083341] irqbypass [ 8069.083341] nvidia_modeset(POE) [ 8069.083342] sg [ 8069.083342] pcspkr [ 8069.083343] i2c_i801 [ 8069.083343] lpc_ich [ 8069.083344] nf_log_ipv4 [ 8069.083344] nf_log_common [ 8069.083344] xt_LOG [ 8069.083345] nf_conntrack_ipv4 [ 8069.083345] nf_defrag_ipv4 [ 8069.083346] xt_multiport [ 8069.083346] xt_owner [ 8069.083347] xt_conntrack [ 8069.083347] nf_conntrack [ 8069.083347] libcrc32c [ 8069.083348] iptable_filter [ 8069.083348] ipmi_si [ 8069.083349] ipmi_devintf [ 8069.083349] ipmi_msghandler [ 8069.083350] acpi_power_meter [ 8069.083350] ib_ipoib [ 8069.083351] rdma_ucm [ 8069.083351] ib_umad [ 8069.083351] iw_cxgb4 [ 8069.083352] rdma_cm [ 8069.083352] iw_cm [ 8069.083353] ib_cm [ 8069.083353] iw_cxgb3 [ 8069.083353] sch_fq_codel [ 8069.083354] binfmt_misc [ 8069.083354] msr_safe(OE) [ 8069.083355] ip_tables [ 8069.083355] nfsv3 [ 8069.083356] nfs_acl [ 8069.083356] rpcsec_gss_krb5 [ 8069.083357] auth_rpcgss [ 8069.083357] nfsv4 [ 8069.083358] dns_resolver [ 8069.083358] nfs [ 8069.083358] lockd [ 8069.083359] grace [ 8069.083359] fscache [ 8069.083360] overlay(T) [ 8069.083360] ext4 [ 8069.083361] mbcache [ 8069.083361] jbd2 [ 8069.083362] sd_mod [ 8069.083362] crc_t10dif [ 8069.083363] crct10dif_generic [ 8069.083363] nvidia_uvm(OE) [ 8069.083364] mlx5_ib [ 8069.083364] ib_uverbs [ 8069.083365] be2iscsi [ 8069.083365] ib_core [ 8069.083366] bnx2i [ 8069.083366] cnic [ 8069.083366] uio [ 8069.083367] cxgb4i [ 8069.083367] cxgb4 [ 8069.083368] cxgb3i [ 8069.083368] cxgb3 [ 8069.083369] mdio [ 8069.083369] libcxgbi [ 8069.083369] libcxgb [ 8069.083370] qla4xxx [ 8069.083370] iscsi_boot_sysfs [ 8069.083371] 8021q [ 8069.083371] garp [ 8069.083372] mrp [ 8069.083372] stp [ 8069.083372] llc [ 8069.083373] nvidia(POE) [ 8069.083373] ast [ 8069.083374] drm_kms_helper [ 8069.083374] crct10dif_pclmul [ 8069.083375] crct10dif_common [ 8069.083375] crc32_pclmul [ 8069.083376] crc32c_intel [ 8069.083376] syscopyarea [ 8069.083377] sysfillrect [ 8069.083377] sysimgblt [ 8069.083378] ghash_clmulni_intel [ 8069.083378] mlx5_core [ 8069.083379] fb_sys_fops [ 8069.083379] igb [ 8069.083379] ttm [ 8069.083380] aesni_intel [ 8069.083380] mlxfw [ 8069.083381] lrw [ 8069.083381] devlink [ 8069.083382] gf128mul [ 8069.083382] dca [ 8069.083382] glue_helper [ 8069.083383] ablk_helper [ 8069.083383] drm [ 8069.083384] dm_multipath [ 8069.083384] ptp [ 8069.083385] cryptd [ 8069.083385] i2c_algo_bit [ 8069.083386] pps_core [ 8069.083386] drm_panel_orientation_quirks [ 8069.083387] wmi [ 8069.083387] sunrpc [ 8069.083387] dm_mirror [ 8069.083388] dm_region_hash [ 8069.083388] dm_log [ 8069.083389] dm_mod [ 8069.083389] iscsi_tcp [ 8069.083390] libiscsi_tcp [ 8069.083390] libiscsi [ 8069.083391] scsi_transport_iscsi [ 8069.083391] fuse [ 8069.083391] [ 8069.083394] CPU: 51 PID: 16814 Comm: ptlrpcd_00_16 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8069.083395] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8069.083396] task: ffff8f484c623180 ti: ffff8f484c634000 task.ti: ffff8f484c634000 [ 8069.083397] RIP: 0010:[] [ 8069.083400] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8069.083401] RSP: 0018:ffff8f484c637b58 EFLAGS: 00000246 [ 8069.083402] RAX: 0000000000000000 RBX: ffff8f47213fad00 RCX: 0000000001990000 [ 8069.083403] RDX: ffff8f687f31b8c0 RSI: 0000000002010001 RDI: ffff8f686e2b6b40 [ 8069.083403] RBP: ffff8f484c637b58 R08: ffff8f487fc5b8c0 R09: 0000000000000000 [ 8069.083404] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8069.083405] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000001fe4b42 [ 8069.083406] FS: 0000000000000000(0000) GS:ffff8f487fc40000(0000) knlGS:0000000000000000 [ 8069.083407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8069.083408] CR2: 00002aaaaad94d70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8069.083409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8069.083410] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8069.083410] Call Trace: [ 8069.083414] [] queued_spin_lock_slowpath+0xb/0xf [ 8069.083416] [] _raw_spin_lock+0x30/0x40 [ 8069.083424] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8069.083435] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8069.083467] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8069.083500] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8069.083530] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8069.083565] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8069.083567] [] ? wake_up_state+0x20/0x20 [ 8069.083600] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8069.083603] [] kthread+0xd1/0xe0 [ 8069.083605] [] ? insert_kthread_work+0x40/0x40 [ 8069.083607] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8069.083609] [] ? insert_kthread_work+0x40/0x40 [ 8069.083610] Code: [ 8069.083611] 13 [ 8069.083611] 48 [ 8069.083612] c1 [ 8069.083612] ea [ 8069.083612] 0d [ 8069.083613] 48 [ 8069.083613] 98 [ 8069.083614] 83 [ 8069.083614] e2 [ 8069.083614] 30 [ 8069.083615] 48 [ 8069.083615] 81 [ 8069.083616] c2 [ 8069.083616] c0 [ 8069.083616] b8 [ 8069.083617] 01 [ 8069.083617] 00 [ 8069.083618] 48 [ 8069.083618] 03 [ 8069.083618] 14 [ 8069.083619] c5 [ 8069.083619] e0 [ 8069.083620] 17 [ 8069.083620] 15 [ 8069.083621] 91 [ 8069.083621] 4c [ 8069.083621] 89 [ 8069.083622] 02 [ 8069.083622] 41 [ 8069.083623] 8b [ 8069.083623] 40 [ 8069.083623] 08 [ 8069.083624] 85 [ 8069.083624] c0 [ 8069.083624] 75 [ 8069.083625] 0f [ 8069.083625] 0f [ 8069.083626] 1f [ 8069.083626] 44 [ 8069.083626] 00 [ 8069.083627] 00 [ 8069.083627] f3 [ 8069.083628] 90 [ 8069.083628] <41> [ 8069.083629] 8b [ 8069.083629] 40 [ 8069.083629] 08 [ 8069.083630] 85 [ 8069.083630] c0 [ 8069.083631] 74 [ 8069.083631] f6 [ 8069.083631] 4d [ 8069.083632] 8b [ 8069.083632] 08 [ 8069.083632] 4d [ 8069.083633] 85 [ 8069.083633] c9 [ 8069.083634] 74 [ 8069.083634] 04 [ 8069.083634] 41 [ 8069.083635] 0f [ 8069.083635] 18 [ 8069.083635] 09 [ 8069.083636] 8b [ 8069.083636] [ 8071.778940] lustre(OE) [ 8071.781835] lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic nvidia_uvm(OE) mlx5_ib ib_uverbs [ 8071.852344] be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8071.901851] CPU: 5 PID: 16824 Comm: ptlrpcd_00_26 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8071.914951] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8071.923809] task: ffff8f484d3de300 ti: ffff8f484f800000 task.ti: ffff8f484f800000 [ 8071.931714] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8071.942060] RSP: 0018:ffff8f484f803b58 EFLAGS: 00000246 [ 8071.947798] RAX: 0000000000000000 RBX: ffff8f481d540d80 RCX: 0000000000290000 [ 8071.955354] RDX: ffff8f487fb1b8c0 RSI: 0000000001710001 RDI: ffff8f686e2b6b40 [ 8071.962911] RBP: ffff8f484f803b58 R08: ffff8f487f55b8c0 R09: 0000000000000000 [ 8071.970469] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8071.978027] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000fc20c75e [ 8071.985582] FS: 0000000000000000(0000) GS:ffff8f487f540000(0000) knlGS:0000000000000000 [ 8071.994094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8072.000264] CR2: 00002aaaabaa0aa0 CR3: 0000001dff1b8000 CR4: 00000000003607e0 [ 8072.007820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8072.015377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8072.022934] Call Trace: [ 8072.025816] [] queued_spin_lock_slowpath+0xb/0xf [ 8072.032504] [] _raw_spin_lock+0x30/0x40 [ 8072.038421] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8072.045294] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8072.051756] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8072.059406] [] ? del_timer_sync+0x52/0x60 [ 8072.065520] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8072.073285] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8072.080446] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8072.086705] [] ? wake_up_state+0x20/0x20 [ 8072.092733] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8072.099688] [] kthread+0xd1/0xe0 [ 8072.104989] [] ? insert_kthread_work+0x40/0x40 [ 8072.111507] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8072.118372] [] ? insert_kthread_work+0x40/0x40 [ 8072.124886] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8072.987232] NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ptlrpcd_01_15:16850] [ 8072.995750] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8073.068340] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8073.120758] CPU: 28 PID: 16850 Comm: ptlrpcd_01_15 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8073.133953] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8073.142811] task: ffff8f484fb9b180 ti: ffff8f484fbb0000 task.ti: ffff8f484fbb0000 [ 8073.150714] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8073.161070] RSP: 0018:ffff8f484fbb3b58 EFLAGS: 00000246 [ 8073.166806] RAX: 0000000000000000 RBX: ffff8f660394e780 RCX: 0000000000e10000 [ 8073.174366] RDX: ffff8f487f65b8c0 RSI: 0000000000490001 RDI: ffff8f686e2b6b40 [ 8073.181921] RBP: ffff8f484fbb3b58 R08: ffff8f687ee9b8c0 R09: 0000000000000000 [ 8073.189479] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8073.197037] R13: 0000000000000003 R14: 0000000000000013 R15: 000000002add0325 [ 8073.204595] FS: 0000000000000000(0000) GS:ffff8f687ee80000(0000) knlGS:0000000000000000 [ 8073.213103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8073.219275] CR2: 00002aaaabaa0aa0 CR3: 0000003f1892c000 CR4: 00000000003607e0 [ 8073.226830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8073.234389] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8073.241944] Call Trace: [ 8073.244830] [] queued_spin_lock_slowpath+0xb/0xf [ 8073.251525] [] _raw_spin_lock+0x30/0x40 [ 8073.257449] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8073.264328] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8073.270819] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8073.278471] [] ? del_timer_sync+0x52/0x60 [ 8073.284591] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8073.292352] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8073.299510] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8073.305771] [] ? wake_up_state+0x20/0x20 [ 8073.311802] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8073.318760] [] kthread+0xd1/0xe0 [ 8073.324070] [] ? insert_kthread_work+0x40/0x40 [ 8073.330588] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8073.337450] [] ? insert_kthread_work+0x40/0x40 [ 8073.343967] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8076.794139] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [ptlrpcd_00_11:16809] [ 8076.802570] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8076.875159] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8076.927576] CPU: 8 PID: 16809 Comm: ptlrpcd_00_11 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8076.940677] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8076.949534] task: ffff8f484c67d280 ti: ffff8f484c618000 task.ti: ffff8f484c618000 [ 8076.957439] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8076.967793] RSP: 0018:ffff8f484c61bb58 EFLAGS: 00000246 [ 8076.973532] RAX: 0000000000000000 RBX: ffff8f46aa4ee780 RCX: 0000000000410000 [ 8076.981089] RDX: ffff8f687ee9b8c0 RSI: 0000000000e10001 RDI: ffff8f686e2b6b40 [ 8076.988645] RBP: ffff8f484c61bb58 R08: ffff8f487f61b8c0 R09: 0000000000000000 [ 8076.996202] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8077.003759] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000f73b8ee9 [ 8077.011317] FS: 0000000000000000(0000) GS:ffff8f487f600000(0000) knlGS:0000000000000000 [ 8077.019829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8077.025997] CR2: 00002aaaabaa0aa0 CR3: 0000001ff5752000 CR4: 00000000003607e0 [ 8077.033556] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8077.041112] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8077.048670] Call Trace: [ 8077.051551] [] queued_spin_lock_slowpath+0xb/0xf [ 8077.058250] [] _raw_spin_lock+0x30/0x40 [ 8077.064184] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8077.071061] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8077.077545] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8077.085194] [] ? del_timer_sync+0x52/0x60 [ 8077.091315] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8077.099078] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8077.106243] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8077.112503] [] ? wake_up_state+0x20/0x20 [ 8077.118535] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8077.125491] [] kthread+0xd1/0xe0 [ 8077.130794] [] ? insert_kthread_work+0x40/0x40 [ 8077.137312] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8077.144174] [] ? insert_kthread_work+0x40/0x40 [ 8077.150690] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8080.743045] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_00_05:16803] [ 8080.806040] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [ptlrpcd_00_18:16816] [ 8080.751476] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 [ 8080.806041] Modules linked in: [ 8080.806041] mgc(OE) [ 8080.806042] lustre(OE) [ 8080.806042] lmv(OE) [ 8080.806043] mdc(OE) [ 8080.806043] osc(OE) [ 8080.806044] lov(OE) [ 8080.806044] fid(OE) [ 8080.806045] fld(OE) [ 8080.806045] ptlrpc(OE) [ 8080.806046] obdclass(OE) [ 8080.806046] ko2iblnd(OE) [ 8080.806047] lnet(OE) [ 8080.806047] libcfs(OE) [ 8080.806048] gdrdrv(POE) [ 8080.806048] iTCO_wdt [ 8080.806049] iTCO_vendor_support [ 8080.806049] rpcrdma [ 8080.806050] nvidia_drm(POE) [ 8080.806050] ib_iser [ 8080.806051] joydev [ 8080.806051] sb_edac [ 8080.806052] intel_powerclamp [ 8080.806052] coretemp [ 8080.806052] intel_rapl [ 8080.806053] iosf_mbi [ 8080.806053] kvm_intel [ 8080.806054] kvm [ 8080.806054] irqbypass [ 8080.806055] nvidia_modeset(POE) [ 8080.806055] sg [ 8080.806056] pcspkr [ 8080.806056] i2c_i801 [ 8080.806056] lpc_ich [ 8080.806057] nf_log_ipv4 [ 8080.806057] nf_log_common [ 8080.806058] xt_LOG [ 8080.806058] nf_conntrack_ipv4 [ 8080.806059] nf_defrag_ipv4 [ 8080.806059] xt_multiport [ 8080.806060] xt_owner [ 8080.806060] xt_conntrack [ 8080.806060] nf_conntrack [ 8080.806061] libcrc32c [ 8080.806061] iptable_filter [ 8080.806062] ipmi_si [ 8080.806062] ipmi_devintf [ 8080.806063] ipmi_msghandler [ 8080.806063] acpi_power_meter [ 8080.806064] ib_ipoib [ 8080.806064] rdma_ucm [ 8080.806065] ib_umad [ 8080.806065] iw_cxgb4 [ 8080.806066] rdma_cm [ 8080.806066] iw_cm [ 8080.806066] ib_cm [ 8080.806067] iw_cxgb3 [ 8080.806068] sch_fq_codel [ 8080.806068] binfmt_misc [ 8080.806069] msr_safe(OE) [ 8080.806070] ip_tables [ 8080.806070] nfsv3 [ 8080.806071] nfs_acl [ 8080.806071] rpcsec_gss_krb5 [ 8080.806072] auth_rpcgss [ 8080.806073] nfsv4 [ 8080.806073] dns_resolver [ 8080.806074] nfs [ 8080.806074] lockd [ 8080.806075] grace [ 8080.806076] fscache [ 8080.806077] overlay(T) [ 8080.806078] ext4 [ 8080.806078] mbcache [ 8080.806079] jbd2 [ 8080.806079] sd_mod [ 8080.806080] crc_t10dif [ 8080.806081] crct10dif_generic [ 8080.806082] nvidia_uvm(OE) [ 8080.806082] mlx5_ib [ 8080.806083] ib_uverbs [ 8080.806084] be2iscsi [ 8080.806084] ib_core [ 8080.806085] bnx2i [ 8080.806085] cnic [ 8080.806086] uio [ 8080.806087] cxgb4i [ 8080.806087] cxgb4 [ 8080.806088] cxgb3i [ 8080.806088] cxgb3 [ 8080.806089] mdio [ 8080.806090] libcxgbi [ 8080.806090] libcxgb [ 8080.806091] qla4xxx [ 8080.806092] iscsi_boot_sysfs [ 8080.806092] 8021q [ 8080.806093] garp [ 8080.806093] mrp [ 8080.806094] stp [ 8080.806094] llc [ 8080.806095] nvidia(POE) [ 8080.806096] ast [ 8080.806096] drm_kms_helper [ 8080.806097] crct10dif_pclmul [ 8080.806098] crct10dif_common [ 8080.806098] crc32_pclmul [ 8080.806099] crc32c_intel [ 8080.806099] syscopyarea [ 8080.806100] sysfillrect [ 8080.806101] sysimgblt [ 8080.806101] ghash_clmulni_intel [ 8080.806102] mlx5_core [ 8080.806103] fb_sys_fops [ 8080.806104] igb [ 8080.806104] ttm [ 8080.806105] aesni_intel [ 8080.806105] mlxfw [ 8080.806106] lrw [ 8080.806107] devlink [ 8080.806107] gf128mul [ 8080.806108] dca [ 8080.806108] glue_helper [ 8080.806109] ablk_helper [ 8080.806110] drm [ 8080.806111] dm_multipath [ 8080.806111] ptp [ 8080.806112] cryptd [ 8080.806112] i2c_algo_bit [ 8080.806113] pps_core [ 8080.806114] drm_panel_orientation_quirks [ 8080.806114] wmi [ 8080.806115] sunrpc [ 8080.806116] dm_mirror [ 8080.806116] dm_region_hash [ 8080.806117] dm_log [ 8080.806118] dm_mod [ 8080.806118] iscsi_tcp [ 8080.806119] libiscsi_tcp [ 8080.806120] libiscsi [ 8080.806121] scsi_transport_iscsi [ 8080.806121] fuse [ 8080.806122] [ 8080.806125] CPU: 10 PID: 16816 Comm: ptlrpcd_00_18 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8080.806126] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8080.806127] task: ffff8f484c625280 ti: ffff8f484d3c8000 task.ti: ffff8f484d3c8000 [ 8080.806128] RIP: 0010:[] [ 8080.806134] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8080.806135] RSP: 0018:ffff8f484d3cbb58 EFLAGS: 00000246 [ 8080.806136] RAX: 0000000000000000 RBX: ffff8f47b82a7980 RCX: 0000000000510000 [ 8080.806137] RDX: ffff8f487f41b8c0 RSI: 0000000000010001 RDI: ffff8f686e2b6b40 [ 8080.806137] RBP: ffff8f484d3cbb58 R08: ffff8f487f69b8c0 R09: 0000000000000000 [ 8080.806138] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8080.806139] R13: 0000000000000003 R14: 0000000000000013 R15: 000000002cb6a5eb [ 8080.806140] FS: 0000000000000000(0000) GS:ffff8f487f680000(0000) knlGS:0000000000000000 [ 8080.806141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.806142] CR2: 00002aaaabaa0aa0 CR3: 0000001fd0788000 CR4: 00000000003607e0 [ 8080.806143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8080.806144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8080.806144] Call Trace: [ 8080.806150] [] queued_spin_lock_slowpath+0xb/0xf [ 8080.806154] [] _raw_spin_lock+0x30/0x40 [ 8080.806172] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8080.806192] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8080.806241] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8080.806246] [] ? del_timer_sync+0x52/0x60 [ 8080.806280] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8080.806310] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8080.806346] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8080.806349] [] ? wake_up_state+0x20/0x20 [ 8080.806382] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8080.806385] [] kthread+0xd1/0xe0 [ 8080.806388] [] ? insert_kthread_work+0x40/0x40 [ 8080.806390] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8080.806392] [] ? insert_kthread_work+0x40/0x40 [ 8080.806393] Code: [ 8080.806394] 0d [ 8080.806394] 48 [ 8080.806394] 98 [ 8080.806395] 83 [ 8080.806395] e2 [ 8080.806395] 30 [ 8080.806396] 48 [ 8080.806396] 81 [ 8080.806397] c2 [ 8080.806397] c0 [ 8080.806397] b8 [ 8080.806398] 01 [ 8080.806398] 00 [ 8080.806399] 48 [ 8080.806399] 03 [ 8080.806399] 14 [ 8080.806400] c5 [ 8080.806400] e0 [ 8080.806401] 17 [ 8080.806401] 15 [ 8080.806401] 91 [ 8080.806402] 4c [ 8080.806402] 89 [ 8080.806403] 02 [ 8080.806403] 41 [ 8080.806403] 8b [ 8080.806404] 40 [ 8080.806404] 08 [ 8080.806405] 85 [ 8080.806405] c0 [ 8080.806405] 75 [ 8080.806406] 0f [ 8080.806406] 0f [ 8080.806406] 1f [ 8080.806407] 44 [ 8080.806407] 00 [ 8080.806408] 00 [ 8080.806408] f3 [ 8080.806408] 90 [ 8080.806409] 41 [ 8080.806409] 8b [ 8080.806410] 40 [ 8080.806410] 08 [ 8080.806411] <85> [ 8080.806411] c0 [ 8080.806411] 74 [ 8080.806412] f6 [ 8080.806412] 4d [ 8080.806412] 8b [ 8080.806413] 08 [ 8080.806413] 4d [ 8080.806414] 85 [ 8080.806414] c9 [ 8080.806414] 74 [ 8080.806415] 04 [ 8080.806415] 41 [ 8080.806415] 0f [ 8080.806416] 18 [ 8080.806416] 09 [ 8080.806417] 8b [ 8080.806417] 17 [ 8080.806417] 0f [ 8080.806418] b7 [ 8080.806418] c2 [ 8080.806418] [ 8080.831039] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [ptlrpcd_00_25:16823] [ 8080.831040] Modules linked in: [ 8080.831041] mgc(OE) [ 8080.831041] lustre(OE) [ 8080.831042] lmv(OE) [ 8080.831042] mdc(OE) [ 8080.831043] osc(OE) [ 8080.831043] lov(OE) [ 8080.831044] fid(OE) [ 8080.831044] fld(OE) [ 8080.831044] ptlrpc(OE) [ 8080.831045] obdclass(OE) [ 8080.831045] ko2iblnd(OE) [ 8080.831046] lnet(OE) [ 8080.831046] libcfs(OE) [ 8080.831047] gdrdrv(POE) [ 8080.831047] iTCO_wdt [ 8080.831048] iTCO_vendor_support [ 8080.831048] rpcrdma [ 8080.831049] nvidia_drm(POE) [ 8080.831049] ib_iser [ 8080.831050] joydev [ 8080.831050] sb_edac [ 8080.831051] intel_powerclamp [ 8080.831051] coretemp [ 8080.831052] intel_rapl [ 8080.831052] iosf_mbi [ 8080.831053] kvm_intel [ 8080.831053] kvm [ 8080.831053] irqbypass [ 8080.831054] nvidia_modeset(POE) [ 8080.831055] sg [ 8080.831055] pcspkr [ 8080.831056] i2c_i801 [ 8080.831056] lpc_ich [ 8080.831057] nf_log_ipv4 [ 8080.831057] nf_log_common [ 8080.831057] xt_LOG [ 8080.831058] nf_conntrack_ipv4 [ 8080.831058] nf_defrag_ipv4 [ 8080.831059] xt_multiport [ 8080.831059] xt_owner [ 8080.831060] xt_conntrack [ 8080.831060] nf_conntrack [ 8080.831061] libcrc32c [ 8080.831061] iptable_filter [ 8080.831062] ipmi_si [ 8080.831062] ipmi_devintf [ 8080.831062] ipmi_msghandler [ 8080.831063] acpi_power_meter [ 8080.831063] ib_ipoib [ 8080.831064] rdma_ucm [ 8080.831064] ib_umad [ 8080.831065] iw_cxgb4 [ 8080.831065] rdma_cm [ 8080.831066] iw_cm [ 8080.831066] ib_cm [ 8080.831067] iw_cxgb3 [ 8080.831067] sch_fq_codel [ 8080.831067] binfmt_misc [ 8080.831068] msr_safe(OE) [ 8080.831068] ip_tables [ 8080.831069] nfsv3 [ 8080.831069] nfs_acl [ 8080.831070] rpcsec_gss_krb5 [ 8080.831070] auth_rpcgss [ 8080.831071] nfsv4 [ 8080.831071] dns_resolver [ 8080.831071] nfs [ 8080.831072] lockd [ 8080.831072] grace [ 8080.831073] fscache [ 8080.831073] overlay(T) [ 8080.831074] ext4 [ 8080.831074] mbcache [ 8080.831075] jbd2 [ 8080.831075] sd_mod [ 8080.831075] crc_t10dif [ 8080.831076] crct10dif_generic [ 8080.831076] nvidia_uvm(OE) [ 8080.831077] mlx5_ib [ 8080.831077] ib_uverbs [ 8080.831078] be2iscsi [ 8080.831078] ib_core [ 8080.831079] bnx2i [ 8080.831079] cnic [ 8080.831080] uio [ 8080.831080] cxgb4i [ 8080.831080] cxgb4 [ 8080.831081] cxgb3i [ 8080.831081] cxgb3 [ 8080.831082] mdio [ 8080.831082] libcxgbi [ 8080.831083] libcxgb [ 8080.831083] qla4xxx [ 8080.831083] iscsi_boot_sysfs [ 8080.831084] 8021q [ 8080.831084] garp [ 8080.831085] mrp [ 8080.831085] stp [ 8080.831086] llc [ 8080.831086] nvidia(POE) [ 8080.831087] ast [ 8080.831087] drm_kms_helper [ 8080.831087] crct10dif_pclmul [ 8080.831088] crct10dif_common [ 8080.831088] crc32_pclmul [ 8080.831089] crc32c_intel [ 8080.831089] syscopyarea [ 8080.831090] sysfillrect [ 8080.831090] sysimgblt [ 8080.831091] ghash_clmulni_intel [ 8080.831091] mlx5_core [ 8080.831091] fb_sys_fops [ 8080.831092] igb [ 8080.831092] ttm [ 8080.831093] aesni_intel [ 8080.831093] mlxfw [ 8080.831094] lrw [ 8080.831094] devlink [ 8080.831095] gf128mul [ 8080.831095] dca [ 8080.831095] glue_helper [ 8080.831096] ablk_helper [ 8080.831096] drm [ 8080.831097] dm_multipath [ 8080.831097] ptp [ 8080.831098] cryptd [ 8080.831098] i2c_algo_bit [ 8080.831098] pps_core [ 8080.831099] drm_panel_orientation_quirks [ 8080.831099] wmi [ 8080.831100] sunrpc [ 8080.831100] dm_mirror [ 8080.831101] dm_region_hash [ 8080.831102] dm_log [ 8080.831102] dm_mod [ 8080.831102] iscsi_tcp [ 8080.831103] libiscsi_tcp [ 8080.831103] libiscsi [ 8080.831104] scsi_transport_iscsi [ 8080.831104] fuse [ 8080.831105] [ 8080.831107] CPU: 14 PID: 16823 Comm: ptlrpcd_00_25 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8080.831108] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8080.831109] task: ffff8f484d3dd280 ti: ffff8f484d3fc000 task.ti: ffff8f484d3fc000 [ 8080.831110] RIP: 0010:[] [ 8080.831112] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8080.831113] RSP: 0018:ffff8f484d3ffb58 EFLAGS: 00000246 [ 8080.831114] RAX: 0000000000000000 RBX: ffff8f46cff48000 RCX: 0000000000710000 [ 8080.831115] RDX: ffff8f487fc9b8c0 RSI: 0000000001a10001 RDI: ffff8f686e2b6b40 [ 8080.831116] RBP: ffff8f484d3ffb58 R08: ffff8f487f79b8c0 R09: 0000000000000000 [ 8080.831117] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8080.831118] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d6c53e91 [ 8080.831119] FS: 0000000000000000(0000) GS:ffff8f487f780000(0000) knlGS:0000000000000000 [ 8080.831120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.831121] CR2: 00002aaaabaa0aa0 CR3: 0000001ef7036000 CR4: 00000000003607e0 [ 8080.831122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8080.831122] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8080.831123] Call Trace: [ 8080.831126] [] queued_spin_lock_slowpath+0xb/0xf [ 8080.831128] [] _raw_spin_lock+0x30/0x40 [ 8080.831136] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8080.831146] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8080.831178] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8080.831181] [] ? del_timer_sync+0x52/0x60 [ 8080.831212] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8080.831242] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8080.831277] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8080.831279] [] ? wake_up_state+0x20/0x20 [ 8080.831312] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8080.831315] [] kthread+0xd1/0xe0 [ 8080.831317] [] ? insert_kthread_work+0x40/0x40 [ 8080.831319] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8080.831321] [] ? insert_kthread_work+0x40/0x40 [ 8080.831322] Code: [ 8080.831322] 13 [ 8080.831323] 48 [ 8080.831323] c1 [ 8080.831323] ea [ 8080.831324] 0d [ 8080.831324] 48 [ 8080.831325] 98 [ 8080.831325] 83 [ 8080.831325] e2 [ 8080.831326] 30 [ 8080.831326] 48 [ 8080.831326] 81 [ 8080.831327] c2 [ 8080.831327] c0 [ 8080.831328] b8 [ 8080.831328] 01 [ 8080.831328] 00 [ 8080.831329] 48 [ 8080.831329] 03 [ 8080.831329] 14 [ 8080.831330] c5 [ 8080.831330] e0 [ 8080.831331] 17 [ 8080.831331] 15 [ 8080.831331] 91 [ 8080.831332] 4c [ 8080.831332] 89 [ 8080.831332] 02 [ 8080.831333] 41 [ 8080.831333] 8b [ 8080.831334] 40 [ 8080.831334] 08 [ 8080.831334] 85 [ 8080.831335] c0 [ 8080.831335] 75 [ 8080.831335] 0f [ 8080.831336] 0f [ 8080.831336] 1f [ 8080.831337] 44 [ 8080.831337] 00 [ 8080.831337] 00 [ 8080.831338] f3 [ 8080.831338] 90 [ 8080.831339] <41> [ 8080.831339] 8b [ 8080.831339] 40 [ 8080.831340] 08 [ 8080.831340] 85 [ 8080.831340] c0 [ 8080.831341] 74 [ 8080.831341] f6 [ 8080.831342] 4d [ 8080.831342] 8b [ 8080.831342] 08 [ 8080.831343] 4d [ 8080.831343] 85 [ 8080.831343] c9 [ 8080.831344] 74 [ 8080.831344] 04 [ 8080.831345] 41 [ 8080.831345] 0f [ 8080.831345] 18 [ 8080.831346] 09 [ 8080.831346] 8b [ 8080.831346] [ 8080.837039] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [ptlrpcd_00_35:16833] [ 8080.837040] Modules linked in: [ 8080.837040] mgc(OE) [ 8080.837041] lustre(OE) [ 8080.837042] lmv(OE) [ 8080.837042] mdc(OE) [ 8080.837042] osc(OE) [ 8080.837043] lov(OE) [ 8080.837043] fid(OE) [ 8080.837044] fld(OE) [ 8080.837044] ptlrpc(OE) [ 8080.837045] obdclass(OE) [ 8080.837045] ko2iblnd(OE) [ 8080.837046] lnet(OE) [ 8080.837046] libcfs(OE) [ 8080.837047] gdrdrv(POE) [ 8080.837047] iTCO_wdt [ 8080.837048] iTCO_vendor_support [ 8080.837048] rpcrdma [ 8080.837049] nvidia_drm(POE) [ 8080.837049] ib_iser [ 8080.837050] joydev [ 8080.837050] sb_edac [ 8080.837051] intel_powerclamp [ 8080.837051] coretemp [ 8080.837052] intel_rapl [ 8080.837052] iosf_mbi [ 8080.837052] kvm_intel [ 8080.837053] kvm [ 8080.837053] irqbypass [ 8080.837054] nvidia_modeset(POE) [ 8080.837054] sg [ 8080.837055] pcspkr [ 8080.837055] i2c_i801 [ 8080.837056] lpc_ich [ 8080.837056] nf_log_ipv4 [ 8080.837057] nf_log_common [ 8080.837057] xt_LOG [ 8080.837058] nf_conntrack_ipv4 [ 8080.837058] nf_defrag_ipv4 [ 8080.837059] xt_multiport [ 8080.837059] xt_owner [ 8080.837059] xt_conntrack [ 8080.837060] nf_conntrack [ 8080.837060] libcrc32c [ 8080.837061] iptable_filter [ 8080.837061] ipmi_si [ 8080.837062] ipmi_devintf [ 8080.837062] ipmi_msghandler [ 8080.837063] acpi_power_meter [ 8080.837063] ib_ipoib [ 8080.837064] rdma_ucm [ 8080.837064] ib_umad [ 8080.837064] iw_cxgb4 [ 8080.837065] rdma_cm [ 8080.837065] iw_cm [ 8080.837066] ib_cm [ 8080.837066] iw_cxgb3 [ 8080.837067] sch_fq_codel [ 8080.837067] binfmt_misc [ 8080.837068] msr_safe(OE) [ 8080.837068] ip_tables [ 8080.837069] nfsv3 [ 8080.837069] nfs_acl [ 8080.837070] rpcsec_gss_krb5 [ 8080.837070] auth_rpcgss [ 8080.837071] nfsv4 [ 8080.837071] dns_resolver [ 8080.837071] nfs [ 8080.837072] lockd [ 8080.837072] grace [ 8080.837073] fscache [ 8080.837073] overlay(T) [ 8080.837074] ext4 [ 8080.837074] mbcache [ 8080.837075] jbd2 [ 8080.837075] sd_mod [ 8080.837076] crc_t10dif [ 8080.837076] crct10dif_generic [ 8080.837077] nvidia_uvm(OE) [ 8080.837077] mlx5_ib [ 8080.837078] ib_uverbs [ 8080.837078] be2iscsi [ 8080.837079] ib_core [ 8080.837079] bnx2i [ 8080.837080] cnic [ 8080.837080] uio [ 8080.837080] cxgb4i [ 8080.837081] cxgb4 [ 8080.837081] cxgb3i [ 8080.837082] cxgb3 [ 8080.837082] mdio [ 8080.837083] libcxgbi [ 8080.837083] libcxgb [ 8080.837083] qla4xxx [ 8080.837084] iscsi_boot_sysfs [ 8080.837084] 8021q [ 8080.837085] garp [ 8080.837085] mrp [ 8080.837086] stp [ 8080.837086] llc [ 8080.837087] nvidia(POE) [ 8080.837087] ast [ 8080.837088] drm_kms_helper [ 8080.837088] crct10dif_pclmul [ 8080.837088] crct10dif_common [ 8080.837089] crc32_pclmul [ 8080.837089] crc32c_intel [ 8080.837090] syscopyarea [ 8080.837090] sysfillrect [ 8080.837091] sysimgblt [ 8080.837091] ghash_clmulni_intel [ 8080.837092] mlx5_core [ 8080.837092] fb_sys_fops [ 8080.837092] igb [ 8080.837093] ttm [ 8080.837093] aesni_intel [ 8080.837094] mlxfw [ 8080.837094] lrw [ 8080.837095] devlink [ 8080.837095] gf128mul [ 8080.837095] dca [ 8080.837096] glue_helper [ 8080.837096] ablk_helper [ 8080.837097] drm [ 8080.837097] dm_multipath [ 8080.837098] ptp [ 8080.837098] cryptd [ 8080.837098] i2c_algo_bit [ 8080.837099] pps_core [ 8080.837099] drm_panel_orientation_quirks [ 8080.837100] wmi [ 8080.837100] sunrpc [ 8080.837101] dm_mirror [ 8080.837101] dm_region_hash [ 8080.837102] dm_log [ 8080.837102] dm_mod [ 8080.837102] iscsi_tcp [ 8080.837103] libiscsi_tcp [ 8080.837103] libiscsi [ 8080.837104] scsi_transport_iscsi [ 8080.837104] fuse [ 8080.837105] [ 8080.837107] CPU: 15 PID: 16833 Comm: ptlrpcd_00_35 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8080.837108] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8080.837109] task: ffff8f484f839080 ti: ffff8f484fb08000 task.ti: ffff8f484fb08000 [ 8080.837110] RIP: 0010:[] [ 8080.837112] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8080.837113] RSP: 0018:ffff8f484fb0bb58 EFLAGS: 00000246 [ 8080.837114] RAX: 0000000000000000 RBX: ffff8f46d46a3180 RCX: 0000000000790000 [ 8080.837115] RDX: ffff8f487f91b8c0 RSI: 0000000001310000 RDI: ffff8f686e2b6b40 [ 8080.837116] RBP: ffff8f484fb0bb58 R08: ffff8f487f7db8c0 R09: 0000000000000000 [ 8080.837117] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8080.837118] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000086998502 [ 8080.837119] FS: 0000000000000000(0000) GS:ffff8f487f7c0000(0000) knlGS:0000000000000000 [ 8080.837120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.837121] CR2: 00002aaaabaa0aa0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8080.837122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8080.837123] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8080.837123] Call Trace: [ 8080.837126] [] queued_spin_lock_slowpath+0xb/0xf [ 8080.837129] [] _raw_spin_lock+0x30/0x40 [ 8080.837137] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8080.837147] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8080.837179] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8080.837211] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8080.837241] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8080.837275] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8080.837278] [] ? wake_up_state+0x20/0x20 [ 8080.837311] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8080.837313] [] kthread+0xd1/0xe0 [ 8080.837316] [] ? insert_kthread_work+0x40/0x40 [ 8080.837317] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8080.837319] [] ? insert_kthread_work+0x40/0x40 [ 8080.837320] Code: [ 8080.837321] 13 [ 8080.837321] 48 [ 8080.837322] c1 [ 8080.837322] ea [ 8080.837322] 0d [ 8080.837323] 48 [ 8080.837323] 98 [ 8080.837324] 83 [ 8080.837324] e2 [ 8080.837324] 30 [ 8080.837325] 48 [ 8080.837325] 81 [ 8080.837326] c2 [ 8080.837326] c0 [ 8080.837326] b8 [ 8080.837327] 01 [ 8080.837327] 00 [ 8080.837328] 48 [ 8080.837328] 03 [ 8080.837328] 14 [ 8080.837329] c5 [ 8080.837329] e0 [ 8080.837330] 17 [ 8080.837330] 15 [ 8080.837330] 91 [ 8080.837331] 4c [ 8080.837331] 89 [ 8080.837332] 02 [ 8080.837332] 41 [ 8080.837332] 8b [ 8080.837333] 40 [ 8080.837333] 08 [ 8080.837333] 85 [ 8080.837334] c0 [ 8080.837334] 75 [ 8080.837335] 0f [ 8080.837335] 0f [ 8080.837335] 1f [ 8080.837336] 44 [ 8080.837336] 00 [ 8080.837337] 00 [ 8080.837337] f3 [ 8080.837337] 90 [ 8080.837338] <41> [ 8080.837338] 8b [ 8080.837339] 40 [ 8080.837339] 08 [ 8080.837339] 85 [ 8080.837340] c0 [ 8080.837340] 74 [ 8080.837341] f6 [ 8080.837341] 4d [ 8080.837341] 8b [ 8080.837342] 08 [ 8080.837342] 4d [ 8080.837342] 85 [ 8080.837343] c9 [ 8080.837343] 74 [ 8080.837344] 04 [ 8080.837344] 41 [ 8080.837344] 0f [ 8080.837345] 18 [ 8080.837345] 09 [ 8080.837345] 8b [ 8080.837346] [ 8080.932038] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [ptlrpcd_01_31:16866] [ 8080.932039] Modules linked in: [ 8080.932040] mgc(OE) [ 8080.932041] lustre(OE) [ 8080.932042] lmv(OE) [ 8080.932042] mdc(OE) [ 8080.932043] osc(OE) [ 8080.932044] lov(OE) [ 8080.932044] fid(OE) [ 8080.932045] fld(OE) [ 8080.932046] ptlrpc(OE) [ 8080.932046] obdclass(OE) [ 8080.932047] ko2iblnd(OE) [ 8080.932048] lnet(OE) [ 8080.932048] libcfs(OE) [ 8080.932049] gdrdrv(POE) [ 8080.932049] iTCO_wdt [ 8080.932050] iTCO_vendor_support [ 8080.932051] rpcrdma [ 8080.932051] nvidia_drm(POE) [ 8080.932052] ib_iser [ 8080.932053] joydev [ 8080.932054] sb_edac [ 8080.932055] intel_powerclamp [ 8080.932055] coretemp [ 8080.932056] intel_rapl [ 8080.932057] iosf_mbi [ 8080.932057] kvm_intel [ 8080.932058] kvm [ 8080.932058] irqbypass [ 8080.932059] nvidia_modeset(POE) [ 8080.932060] sg [ 8080.932060] pcspkr [ 8080.932061] i2c_i801 [ 8080.932062] lpc_ich [ 8080.932062] nf_log_ipv4 [ 8080.932063] nf_log_common [ 8080.932064] xt_LOG [ 8080.932064] nf_conntrack_ipv4 [ 8080.932065] nf_defrag_ipv4 [ 8080.932066] xt_multiport [ 8080.932066] xt_owner [ 8080.932067] xt_conntrack [ 8080.932068] nf_conntrack [ 8080.932068] libcrc32c [ 8080.932069] iptable_filter [ 8080.932070] ipmi_si [ 8080.932070] ipmi_devintf [ 8080.932071] ipmi_msghandler [ 8080.932072] acpi_power_meter [ 8080.932072] ib_ipoib [ 8080.932073] rdma_ucm [ 8080.932074] ib_umad [ 8080.932074] iw_cxgb4 [ 8080.932075] rdma_cm [ 8080.932076] iw_cm [ 8080.932076] ib_cm [ 8080.932077] iw_cxgb3 [ 8080.932077] sch_fq_codel [ 8080.932079] binfmt_misc [ 8080.932080] msr_safe(OE) [ 8080.932080] ip_tables [ 8080.932081] nfsv3 [ 8080.932082] nfs_acl [ 8080.932082] rpcsec_gss_krb5 [ 8080.932083] auth_rpcgss [ 8080.932084] nfsv4 [ 8080.932084] dns_resolver [ 8080.932085] nfs [ 8080.932085] lockd [ 8080.932086] grace [ 8080.932087] fscache [ 8080.932087] overlay(T) [ 8080.932088] ext4 [ 8080.932089] mbcache [ 8080.932089] jbd2 [ 8080.932090] sd_mod [ 8080.932091] crc_t10dif [ 8080.932091] crct10dif_generic [ 8080.932092] nvidia_uvm(OE) [ 8080.932093] mlx5_ib [ 8080.932093] ib_uverbs [ 8080.932094] be2iscsi [ 8080.932095] ib_core [ 8080.932095] bnx2i [ 8080.932096] cnic [ 8080.932097] uio [ 8080.932097] cxgb4i [ 8080.932098] cxgb4 [ 8080.932098] cxgb3i [ 8080.932099] cxgb3 [ 8080.932100] mdio [ 8080.932101] libcxgbi [ 8080.932101] libcxgb [ 8080.932102] qla4xxx [ 8080.932102] iscsi_boot_sysfs [ 8080.932103] 8021q [ 8080.932104] garp [ 8080.932104] mrp [ 8080.932105] stp [ 8080.932105] llc [ 8080.932106] nvidia(POE) [ 8080.932107] ast [ 8080.932108] drm_kms_helper [ 8080.932108] crct10dif_pclmul [ 8080.932109] crct10dif_common [ 8080.932110] crc32_pclmul [ 8080.932111] crc32c_intel [ 8080.932111] syscopyarea [ 8080.932112] sysfillrect [ 8080.932113] sysimgblt [ 8080.932113] ghash_clmulni_intel [ 8080.932114] mlx5_core [ 8080.932114] fb_sys_fops [ 8080.932115] igb [ 8080.932116] ttm [ 8080.932117] aesni_intel [ 8080.932117] mlxfw [ 8080.932118] lrw [ 8080.932118] devlink [ 8080.932119] gf128mul [ 8080.932120] dca [ 8080.932121] glue_helper [ 8080.932121] ablk_helper [ 8080.932122] drm [ 8080.932123] dm_multipath [ 8080.932123] ptp [ 8080.932124] cryptd [ 8080.932124] i2c_algo_bit [ 8080.932125] pps_core [ 8080.932126] drm_panel_orientation_quirks [ 8080.932127] wmi [ 8080.932128] sunrpc [ 8080.932128] dm_mirror [ 8080.932129] dm_region_hash [ 8080.932129] dm_log [ 8080.932130] dm_mod [ 8080.932131] iscsi_tcp [ 8080.932132] libiscsi_tcp [ 8080.932132] libiscsi [ 8080.932133] scsi_transport_iscsi [ 8080.932134] fuse [ 8080.932134] [ 8080.932137] CPU: 19 PID: 16866 Comm: ptlrpcd_01_31 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8080.932138] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8080.932139] task: ffff8f484fbed280 ti: ffff8f484d288000 task.ti: ffff8f484d288000 [ 8080.932141] RIP: 0010:[] [ 8080.932148] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8080.932149] RSP: 0018:ffff8f484d28bb58 EFLAGS: 00000246 [ 8080.932150] RAX: 0000000000000000 RBX: ffff8f65fd0f8000 RCX: 0000000000990000 [ 8080.932151] RDX: ffff8f487fb5b8c0 RSI: 0000000001790001 RDI: ffff8f686e2b6b40 [ 8080.932152] RBP: ffff8f484d28bb58 R08: ffff8f687ec5b8c0 R09: 0000000000000000 [ 8080.932153] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8080.932153] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000c64e37e3 [ 8080.932155] FS: 0000000000000000(0000) GS:ffff8f687ec40000(0000) knlGS:0000000000000000 [ 8080.932156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.932157] CR2: 00002aaaad64527d CR3: 0000003e066a4000 CR4: 00000000003607e0 [ 8080.932158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8080.932159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8080.932160] Call Trace: [ 8080.932167] [] queued_spin_lock_slowpath+0xb/0xf [ 8080.932172] [] _raw_spin_lock+0x30/0x40 [ 8080.932188] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8080.932205] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8080.932267] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8080.932273] [] ? del_timer_sync+0x52/0x60 [ 8080.932322] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8080.932370] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8080.932424] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8080.932428] [] ? wake_up_state+0x20/0x20 [ 8080.932479] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8080.932482] [] kthread+0xd1/0xe0 [ 8080.932484] [] ? insert_kthread_work+0x40/0x40 [ 8080.932487] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8080.932489] [] ? insert_kthread_work+0x40/0x40 [ 8080.932490] Code: [ 8080.932491] 0d [ 8080.932491] 48 [ 8080.932492] 98 [ 8080.932492] 83 [ 8080.932492] e2 [ 8080.932493] 30 [ 8080.932493] 48 [ 8080.932494] 81 [ 8080.932494] c2 [ 8080.932494] c0 [ 8080.932495] b8 [ 8080.932495] 01 [ 8080.932496] 00 [ 8080.932496] 48 [ 8080.932496] 03 [ 8080.932497] 14 [ 8080.932497] c5 [ 8080.932498] e0 [ 8080.932498] 17 [ 8080.932498] 15 [ 8080.932499] 91 [ 8080.932499] 4c [ 8080.932500] 89 [ 8080.932500] 02 [ 8080.932500] 41 [ 8080.932501] 8b [ 8080.932501] 40 [ 8080.932502] 08 [ 8080.932502] 85 [ 8080.932502] c0 [ 8080.932503] 75 [ 8080.932503] 0f [ 8080.932504] 0f [ 8080.932504] 1f [ 8080.932505] 44 [ 8080.932505] 00 [ 8080.932505] 00 [ 8080.932506] f3 [ 8080.932506] 90 [ 8080.932507] 41 [ 8080.932507] 8b [ 8080.932507] 40 [ 8080.932508] 08 [ 8080.932508] <85> [ 8080.932509] c0 [ 8080.932509] 74 [ 8080.932510] f6 [ 8080.932510] 4d [ 8080.932510] 8b [ 8080.932511] 08 [ 8080.932511] 4d [ 8080.932512] 85 [ 8080.932512] c9 [ 8080.932512] 74 [ 8080.932513] 04 [ 8080.932513] 41 [ 8080.932513] 0f [ 8080.932514] 18 [ 8080.932514] 09 [ 8080.932515] 8b [ 8080.932515] 17 [ 8080.932515] 0f [ 8080.932516] b7 [ 8080.932516] c2 [ 8080.932516] [ 8080.938037] NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [ptlrpcd_01_19:16854] [ 8080.938038] Modules linked in: [ 8080.938038] mgc(OE) [ 8080.938039] lustre(OE) [ 8080.938039] lmv(OE) [ 8080.938040] mdc(OE) [ 8080.938040] osc(OE) [ 8080.938040] lov(OE) [ 8080.938041] fid(OE) [ 8080.938041] fld(OE) [ 8080.938042] ptlrpc(OE) [ 8080.938042] obdclass(OE) [ 8080.938043] ko2iblnd(OE) [ 8080.938043] lnet(OE) [ 8080.938044] libcfs(OE) [ 8080.938044] gdrdrv(POE) [ 8080.938045] iTCO_wdt [ 8080.938045] iTCO_vendor_support [ 8080.938046] rpcrdma [ 8080.938047] nvidia_drm(POE) [ 8080.938047] ib_iser [ 8080.938047] joydev [ 8080.938048] sb_edac [ 8080.938048] intel_powerclamp [ 8080.938049] coretemp [ 8080.938049] intel_rapl [ 8080.938050] iosf_mbi [ 8080.938050] kvm_intel [ 8080.938050] kvm [ 8080.938051] irqbypass [ 8080.938051] nvidia_modeset(POE) [ 8080.938052] sg [ 8080.938052] pcspkr [ 8080.938053] i2c_i801 [ 8080.938053] lpc_ich [ 8080.938054] nf_log_ipv4 [ 8080.938054] nf_log_common [ 8080.938055] xt_LOG [ 8080.938055] nf_conntrack_ipv4 [ 8080.938056] nf_defrag_ipv4 [ 8080.938056] xt_multiport [ 8080.938057] xt_owner [ 8080.938057] xt_conntrack [ 8080.938057] nf_conntrack [ 8080.938058] libcrc32c [ 8080.938058] iptable_filter [ 8080.938059] ipmi_si [ 8080.938059] ipmi_devintf [ 8080.938060] ipmi_msghandler [ 8080.938060] acpi_power_meter [ 8080.938061] ib_ipoib [ 8080.938061] rdma_ucm [ 8080.938062] ib_umad [ 8080.938062] iw_cxgb4 [ 8080.938063] rdma_cm [ 8080.938063] iw_cm [ 8080.938064] ib_cm [ 8080.938064] iw_cxgb3 [ 8080.938065] sch_fq_codel [ 8080.938065] binfmt_misc [ 8080.938066] msr_safe(OE) [ 8080.938066] ip_tables [ 8080.938067] nfsv3 [ 8080.938067] nfs_acl [ 8080.938068] rpcsec_gss_krb5 [ 8080.938068] auth_rpcgss [ 8080.938069] nfsv4 [ 8080.938069] dns_resolver [ 8080.938069] nfs [ 8080.938070] lockd [ 8080.938070] grace [ 8080.938071] fscache [ 8080.938071] overlay(T) [ 8080.938072] ext4 [ 8080.938072] mbcache [ 8080.938073] jbd2 [ 8080.938073] sd_mod [ 8080.938074] crc_t10dif [ 8080.938074] crct10dif_generic [ 8080.938075] nvidia_uvm(OE) [ 8080.938075] mlx5_ib [ 8080.938076] ib_uverbs [ 8080.938076] be2iscsi [ 8080.938076] ib_core [ 8080.938077] bnx2i [ 8080.938077] cnic [ 8080.938078] uio [ 8080.938078] cxgb4i [ 8080.938079] cxgb4 [ 8080.938079] cxgb3i [ 8080.938080] cxgb3 [ 8080.938080] mdio [ 8080.938080] libcxgbi [ 8080.938081] libcxgb [ 8080.938081] qla4xxx [ 8080.938082] iscsi_boot_sysfs [ 8080.938082] 8021q [ 8080.938082] garp [ 8080.938083] mrp [ 8080.938083] stp [ 8080.938084] llc [ 8080.938084] nvidia(POE) [ 8080.938085] ast [ 8080.938085] drm_kms_helper [ 8080.938086] crct10dif_pclmul [ 8080.938086] crct10dif_common [ 8080.938087] crc32_pclmul [ 8080.938087] crc32c_intel [ 8080.938088] syscopyarea [ 8080.938088] sysfillrect [ 8080.938088] sysimgblt [ 8080.938089] ghash_clmulni_intel [ 8080.938090] mlx5_core [ 8080.938090] fb_sys_fops [ 8080.938090] igb [ 8080.938091] ttm [ 8080.938091] aesni_intel [ 8080.938092] mlxfw [ 8080.938092] lrw [ 8080.938093] devlink [ 8080.938093] gf128mul [ 8080.938093] dca [ 8080.938094] glue_helper [ 8080.938094] ablk_helper [ 8080.938095] drm [ 8080.938095] dm_multipath [ 8080.938096] ptp [ 8080.938096] cryptd [ 8080.938097] i2c_algo_bit [ 8080.938097] pps_core [ 8080.938098] drm_panel_orientation_quirks [ 8080.938098] wmi [ 8080.938099] sunrpc [ 8080.938099] dm_mirror [ 8080.938099] dm_region_hash [ 8080.938100] dm_log [ 8080.938100] dm_mod [ 8080.938101] iscsi_tcp [ 8080.938101] libiscsi_tcp [ 8080.938102] libiscsi [ 8080.938102] scsi_transport_iscsi [ 8080.938103] fuse [ 8080.938103] [ 8080.938105] CPU: 20 PID: 16854 Comm: ptlrpcd_01_19 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8080.938106] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8080.938107] task: ffff8f484fbc0000 ti: ffff8f484fbc8000 task.ti: ffff8f484fbc8000 [ 8080.938108] RIP: 0010:[] [ 8080.938110] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8080.938111] RSP: 0018:ffff8f484fbcbb58 EFLAGS: 00000246 [ 8080.938112] RAX: 0000000000000000 RBX: ffff8f65fa29d100 RCX: 0000000000a10000 [ 8080.938113] RDX: ffff8f687f09b8c0 RSI: 0000000001b10001 RDI: ffff8f686e2b6b40 [ 8080.938114] RBP: ffff8f484fbcbb58 R08: ffff8f687ec9b8c0 R09: 0000000000000000 [ 8080.938115] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8080.938116] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d52ecab8 [ 8080.938117] FS: 0000000000000000(0000) GS:ffff8f687ec80000(0000) knlGS:0000000000000000 [ 8080.938118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.938119] CR2: 00002aaaad64527d CR3: 0000003f50bc4000 CR4: 00000000003607e0 [ 8080.938119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8080.938120] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8080.938121] Call Trace: [ 8080.938123] [] queued_spin_lock_slowpath+0xb/0xf [ 8080.938125] [] _raw_spin_lock+0x30/0x40 [ 8080.938134] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8080.938144] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8080.938176] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8080.938208] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8080.938238] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8080.938275] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8080.938277] [] ? wake_up_state+0x20/0x20 [ 8080.938311] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8080.938313] [] kthread+0xd1/0xe0 [ 8080.938315] [] ? insert_kthread_work+0x40/0x40 [ 8080.938317] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8080.938319] [] ? insert_kthread_work+0x40/0x40 [ 8080.938320] Code: [ 8080.938321] 13 [ 8080.938321] 48 [ 8080.938321] c1 [ 8080.938322] ea [ 8080.938322] 0d [ 8080.938323] 48 [ 8080.938323] 98 [ 8080.938323] 83 [ 8080.938324] e2 [ 8080.938324] 30 [ 8080.938325] 48 [ 8080.938325] 81 [ 8080.938325] c2 [ 8080.938326] c0 [ 8080.938326] b8 [ 8080.938327] 01 [ 8080.938327] 00 [ 8080.938327] 48 [ 8080.938328] 03 [ 8080.938328] 14 [ 8080.938329] c5 [ 8080.938329] e0 [ 8080.938329] 17 [ 8080.938330] 15 [ 8080.938330] 91 [ 8080.938331] 4c [ 8080.938331] 89 [ 8080.938332] 02 [ 8080.938332] 41 [ 8080.938332] 8b [ 8080.938333] 40 [ 8080.938333] 08 [ 8080.938333] 85 [ 8080.938334] c0 [ 8080.938334] 75 [ 8080.938335] 0f [ 8080.938335] 0f [ 8080.938336] 1f [ 8080.938336] 44 [ 8080.938336] 00 [ 8080.938337] 00 [ 8080.938337] f3 [ 8080.938337] 90 [ 8080.938338] <41> [ 8080.938338] 8b [ 8080.938339] 40 [ 8080.938339] 08 [ 8080.938339] 85 [ 8080.938340] c0 [ 8080.938340] 74 [ 8080.938341] f6 [ 8080.938341] 4d [ 8080.938341] 8b [ 8080.938342] 08 [ 8080.938342] 4d [ 8080.938342] 85 [ 8080.938343] c9 [ 8080.938343] 74 [ 8080.938344] 04 [ 8080.938344] 41 [ 8080.938344] 0f [ 8080.938345] 18 [ 8080.938345] 09 [ 8080.938345] 8b [ 8080.938346] [ 8081.018034] NMI watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [ptlrpcd_01_05:16840] [ 8081.018035] Modules linked in: [ 8081.018035] mgc(OE) [ 8081.018036] lustre(OE) [ 8081.018036] lmv(OE) [ 8081.018036] mdc(OE) [ 8081.018037] osc(OE) [ 8081.018037] lov(OE) [ 8081.018037] fid(OE) [ 8081.018038] fld(OE) [ 8081.018038] ptlrpc(OE) [ 8081.018038] obdclass(OE) [ 8081.018039] ko2iblnd(OE) [ 8081.018039] lnet(OE) [ 8081.018039] libcfs(OE) [ 8081.018039] gdrdrv(POE) [ 8081.018040] iTCO_wdt [ 8081.018040] iTCO_vendor_support [ 8081.018040] rpcrdma [ 8081.018041] nvidia_drm(POE) [ 8081.018041] ib_iser [ 8081.018041] joydev [ 8081.018042] sb_edac [ 8081.018042] intel_powerclamp [ 8081.018043] coretemp [ 8081.018043] intel_rapl [ 8081.018043] iosf_mbi [ 8081.018044] kvm_intel [ 8081.018044] kvm [ 8081.018044] irqbypass [ 8081.018045] nvidia_modeset(POE) [ 8081.018045] sg [ 8081.018045] pcspkr [ 8081.018046] i2c_i801 [ 8081.018046] lpc_ich [ 8081.018046] nf_log_ipv4 [ 8081.018047] nf_log_common [ 8081.018047] xt_LOG [ 8081.018047] nf_conntrack_ipv4 [ 8081.018048] nf_defrag_ipv4 [ 8081.018048] xt_multiport [ 8081.018048] xt_owner [ 8081.018049] xt_conntrack [ 8081.018049] nf_conntrack [ 8081.018049] libcrc32c [ 8081.018050] iptable_filter [ 8081.018050] ipmi_si [ 8081.018050] ipmi_devintf [ 8081.018051] ipmi_msghandler [ 8081.018051] acpi_power_meter [ 8081.018051] ib_ipoib [ 8081.018051] rdma_ucm [ 8081.018052] ib_umad [ 8081.018052] iw_cxgb4 [ 8081.018052] rdma_cm [ 8081.018053] iw_cm [ 8081.018053] ib_cm [ 8081.018053] iw_cxgb3 [ 8081.018054] sch_fq_codel [ 8081.018054] binfmt_misc [ 8081.018054] msr_safe(OE) [ 8081.018055] ip_tables [ 8081.018055] nfsv3 [ 8081.018055] nfs_acl [ 8081.018056] rpcsec_gss_krb5 [ 8081.018056] auth_rpcgss [ 8081.018056] nfsv4 [ 8081.018056] dns_resolver [ 8081.018057] nfs [ 8081.018057] lockd [ 8081.018057] grace [ 8081.018058] fscache [ 8081.018058] overlay(T) [ 8081.018058] ext4 [ 8081.018059] mbcache [ 8081.018059] jbd2 [ 8081.018059] sd_mod [ 8081.018060] crc_t10dif [ 8081.018060] crct10dif_generic [ 8081.018060] nvidia_uvm(OE) [ 8081.018061] mlx5_ib [ 8081.018061] ib_uverbs [ 8081.018061] be2iscsi [ 8081.018062] ib_core [ 8081.018062] bnx2i [ 8081.018062] cnic [ 8081.018062] uio [ 8081.018063] cxgb4i [ 8081.018063] cxgb4 [ 8081.018063] cxgb3i [ 8081.018064] cxgb3 [ 8081.018064] mdio [ 8081.018064] libcxgbi [ 8081.018064] libcxgb [ 8081.018065] qla4xxx [ 8081.018065] iscsi_boot_sysfs [ 8081.018065] 8021q [ 8081.018066] garp [ 8081.018066] mrp [ 8081.018066] stp [ 8081.018067] llc [ 8081.018067] nvidia(POE) [ 8081.018067] ast [ 8081.018068] drm_kms_helper [ 8081.018068] crct10dif_pclmul [ 8081.018068] crct10dif_common [ 8081.018069] crc32_pclmul [ 8081.018069] crc32c_intel [ 8081.018069] syscopyarea [ 8081.018070] sysfillrect [ 8081.018070] sysimgblt [ 8081.018070] ghash_clmulni_intel [ 8081.018070] mlx5_core [ 8081.018071] fb_sys_fops [ 8081.018071] igb [ 8081.018071] ttm [ 8081.018072] aesni_intel [ 8081.018072] mlxfw [ 8081.018072] lrw [ 8081.018072] devlink [ 8081.018073] gf128mul [ 8081.018073] dca [ 8081.018073] glue_helper [ 8081.018074] ablk_helper [ 8081.018074] drm [ 8081.018074] dm_multipath [ 8081.018075] ptp [ 8081.018075] cryptd [ 8081.018075] i2c_algo_bit [ 8081.018075] pps_core [ 8081.018076] drm_panel_orientation_quirks [ 8081.018076] wmi [ 8081.018076] sunrpc [ 8081.018077] dm_mirror [ 8081.018077] dm_region_hash [ 8081.018077] dm_log [ 8081.018078] dm_mod [ 8081.018078] iscsi_tcp [ 8081.018078] libiscsi_tcp [ 8081.018079] libiscsi [ 8081.018079] scsi_transport_iscsi [ 8081.018079] fuse [ 8081.018080] [ 8081.018081] CPU: 33 PID: 16840 Comm: ptlrpcd_01_05 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.018082] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.018082] task: ffff8f484fb28000 ti: ffff8f484fb30000 task.ti: ffff8f484fb30000 [ 8081.018083] RIP: 0010:[] [ 8081.018086] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.018086] RSP: 0018:ffff8f484fb33b58 EFLAGS: 00000246 [ 8081.018087] RAX: 0000000000000000 RBX: ffff8f65fca04380 RCX: 0000000001090000 [ 8081.018088] RDX: ffff8f487fcdb8c0 RSI: 0000000001a90001 RDI: ffff8f686e2b6b40 [ 8081.018088] RBP: ffff8f484fb33b58 R08: ffff8f687efdb8c0 R09: 0000000000000000 [ 8081.018089] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.018089] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000fb9209f2 [ 8081.018090] FS: 0000000000000000(0000) GS:ffff8f687efc0000(0000) knlGS:0000000000000000 [ 8081.018091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.018091] CR2: 00002aaaad64527d CR3: 0000003ecaedc000 CR4: 00000000003607e0 [ 8081.018092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.018092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.018093] Call Trace: [ 8081.018095] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.018097] [] _raw_spin_lock+0x30/0x40 [ 8081.018103] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.018111] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.018135] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.018159] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.018181] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.018207] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.018210] [] ? wake_up_state+0x20/0x20 [ 8081.018235] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.018236] [] kthread+0xd1/0xe0 [ 8081.018238] [] ? insert_kthread_work+0x40/0x40 [ 8081.018240] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.018241] [] ? insert_kthread_work+0x40/0x40 [ 8081.018242] Code: [ 8081.018242] 0d [ 8081.018243] 48 [ 8081.018243] 98 [ 8081.018243] 83 [ 8081.018243] e2 [ 8081.018244] 30 [ 8081.018244] 48 [ 8081.018244] 81 [ 8081.018245] c2 [ 8081.018245] c0 [ 8081.018245] b8 [ 8081.018245] 01 [ 8081.018246] 00 [ 8081.018246] 48 [ 8081.018246] 03 [ 8081.018247] 14 [ 8081.018247] c5 [ 8081.018247] e0 [ 8081.018247] 17 [ 8081.018248] 15 [ 8081.018248] 91 [ 8081.018248] 4c [ 8081.018248] 89 [ 8081.018249] 02 [ 8081.018249] 41 [ 8081.018249] 8b [ 8081.018249] 40 [ 8081.018250] 08 [ 8081.018250] 85 [ 8081.018250] c0 [ 8081.018250] 75 [ 8081.018251] 0f [ 8081.018251] 0f [ 8081.018251] 1f [ 8081.018251] 44 [ 8081.018252] 00 [ 8081.018252] 00 [ 8081.018252] f3 [ 8081.018252] 90 [ 8081.018253] 41 [ 8081.018253] 8b [ 8081.018253] 40 [ 8081.018253] 08 [ 8081.018254] <85> [ 8081.018254] c0 [ 8081.018254] 74 [ 8081.018254] f6 [ 8081.018254] 4d [ 8081.018255] 8b [ 8081.018255] 08 [ 8081.018255] 4d [ 8081.018255] 85 [ 8081.018256] c9 [ 8081.018256] 74 [ 8081.018256] 04 [ 8081.018256] 41 [ 8081.018257] 0f [ 8081.018257] 18 [ 8081.018257] 09 [ 8081.018257] 8b [ 8081.018257] 17 [ 8081.018258] 0f [ 8081.018258] b7 [ 8081.018258] c2 [ 8081.018258] [ 8081.025035] NMI watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [ptlrpcd_01_04:16839] [ 8081.025035] Modules linked in: [ 8081.025036] mgc(OE) [ 8081.025036] lustre(OE) [ 8081.025037] lmv(OE) [ 8081.025037] mdc(OE) [ 8081.025037] osc(OE) [ 8081.025037] lov(OE) [ 8081.025038] fid(OE) [ 8081.025038] fld(OE) [ 8081.025038] ptlrpc(OE) [ 8081.025039] obdclass(OE) [ 8081.025039] ko2iblnd(OE) [ 8081.025039] lnet(OE) [ 8081.025040] libcfs(OE) [ 8081.025040] gdrdrv(POE) [ 8081.025040] iTCO_wdt [ 8081.025041] iTCO_vendor_support [ 8081.025041] rpcrdma [ 8081.025041] nvidia_drm(POE) [ 8081.025042] ib_iser [ 8081.025042] joydev [ 8081.025042] sb_edac [ 8081.025043] intel_powerclamp [ 8081.025043] coretemp [ 8081.025043] intel_rapl [ 8081.025043] iosf_mbi [ 8081.025044] kvm_intel [ 8081.025044] kvm [ 8081.025044] irqbypass [ 8081.025045] nvidia_modeset(POE) [ 8081.025045] sg [ 8081.025045] pcspkr [ 8081.025046] i2c_i801 [ 8081.025046] lpc_ich [ 8081.025046] nf_log_ipv4 [ 8081.025046] nf_log_common [ 8081.025047] xt_LOG [ 8081.025047] nf_conntrack_ipv4 [ 8081.025047] nf_defrag_ipv4 [ 8081.025048] xt_multiport [ 8081.025048] xt_owner [ 8081.025048] xt_conntrack [ 8081.025048] nf_conntrack [ 8081.025049] libcrc32c [ 8081.025049] iptable_filter [ 8081.025049] ipmi_si [ 8081.025050] ipmi_devintf [ 8081.025050] ipmi_msghandler [ 8081.025050] acpi_power_meter [ 8081.025051] ib_ipoib [ 8081.025051] rdma_ucm [ 8081.025051] ib_umad [ 8081.025052] iw_cxgb4 [ 8081.025052] rdma_cm [ 8081.025052] iw_cm [ 8081.025052] ib_cm [ 8081.025053] iw_cxgb3 [ 8081.025053] sch_fq_codel [ 8081.025053] binfmt_misc [ 8081.025054] msr_safe(OE) [ 8081.025054] ip_tables [ 8081.025054] nfsv3 [ 8081.025055] nfs_acl [ 8081.025055] rpcsec_gss_krb5 [ 8081.025055] auth_rpcgss [ 8081.025056] nfsv4 [ 8081.025056] dns_resolver [ 8081.025056] nfs [ 8081.025057] lockd [ 8081.025057] grace [ 8081.025057] fscache [ 8081.025058] overlay(T) [ 8081.025058] ext4 [ 8081.025058] mbcache [ 8081.025059] jbd2 [ 8081.025059] sd_mod [ 8081.025059] crc_t10dif [ 8081.025060] crct10dif_generic [ 8081.025060] nvidia_uvm(OE) [ 8081.025060] mlx5_ib [ 8081.025061] ib_uverbs [ 8081.025061] be2iscsi [ 8081.025061] ib_core [ 8081.025062] bnx2i [ 8081.025062] cnic [ 8081.025062] uio [ 8081.025063] cxgb4i [ 8081.025063] cxgb4 [ 8081.025063] cxgb3i [ 8081.025063] cxgb3 [ 8081.025064] mdio [ 8081.025064] libcxgbi [ 8081.025064] libcxgb [ 8081.025065] qla4xxx [ 8081.025065] iscsi_boot_sysfs [ 8081.025065] 8021q [ 8081.025066] garp [ 8081.025066] mrp [ 8081.025066] stp [ 8081.025066] llc [ 8081.025067] nvidia(POE) [ 8081.025067] ast [ 8081.025067] drm_kms_helper [ 8081.025068] crct10dif_pclmul [ 8081.025068] crct10dif_common [ 8081.025068] crc32_pclmul [ 8081.025069] crc32c_intel [ 8081.025069] syscopyarea [ 8081.025069] sysfillrect [ 8081.025070] sysimgblt [ 8081.025070] ghash_clmulni_intel [ 8081.025070] mlx5_core [ 8081.025071] fb_sys_fops [ 8081.025071] igb [ 8081.025071] ttm [ 8081.025072] aesni_intel [ 8081.025072] mlxfw [ 8081.025072] lrw [ 8081.025073] devlink [ 8081.025073] gf128mul [ 8081.025073] dca [ 8081.025073] glue_helper [ 8081.025074] ablk_helper [ 8081.025074] drm [ 8081.025074] dm_multipath [ 8081.025075] ptp [ 8081.025075] cryptd [ 8081.025075] i2c_algo_bit [ 8081.025076] pps_core [ 8081.025076] drm_panel_orientation_quirks [ 8081.025076] wmi [ 8081.025077] sunrpc [ 8081.025077] dm_mirror [ 8081.025077] dm_region_hash [ 8081.025078] dm_log [ 8081.025078] dm_mod [ 8081.025078] iscsi_tcp [ 8081.025079] libiscsi_tcp [ 8081.025079] libiscsi [ 8081.025079] scsi_transport_iscsi [ 8081.025079] fuse [ 8081.025080] [ 8081.025081] CPU: 34 PID: 16839 Comm: ptlrpcd_01_04 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.025082] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.025083] task: ffff8f484f83e300 ti: ffff8f484fb24000 task.ti: ffff8f484fb24000 [ 8081.025084] RIP: 0010:[] [ 8081.025086] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.025086] RSP: 0018:ffff8f484fb27b58 EFLAGS: 00000246 [ 8081.025087] RAX: 0000000000000000 RBX: ffff8f672af14800 RCX: 0000000001110000 [ 8081.025088] RDX: ffff8f487f85b8c0 RSI: 0000000000890001 RDI: ffff8f686e2b6b40 [ 8081.025088] RBP: ffff8f484fb27b58 R08: ffff8f687f01b8c0 R09: 0000000000000000 [ 8081.025089] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.025089] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003d57b1a1 [ 8081.025090] FS: 0000000000000000(0000) GS:ffff8f687f000000(0000) knlGS:0000000000000000 [ 8081.025091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.025091] CR2: 00002aaaad64527d CR3: 0000003f2f078000 CR4: 00000000003607e0 [ 8081.025092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.025093] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.025093] Call Trace: [ 8081.025096] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.025097] [] _raw_spin_lock+0x30/0x40 [ 8081.025104] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.025111] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.025136] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.025160] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.025183] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.025216] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.025219] [] ? wake_up_state+0x20/0x20 [ 8081.025253] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.025255] [] kthread+0xd1/0xe0 [ 8081.025257] [] ? insert_kthread_work+0x40/0x40 [ 8081.025259] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.025260] [] ? insert_kthread_work+0x40/0x40 [ 8081.025261] Code: [ 8081.025261] 0d [ 8081.025262] 48 [ 8081.025262] 98 [ 8081.025262] 83 [ 8081.025262] e2 [ 8081.025263] 30 [ 8081.025263] 48 [ 8081.025263] 81 [ 8081.025263] c2 [ 8081.025264] c0 [ 8081.025264] b8 [ 8081.025264] 01 [ 8081.025265] 00 [ 8081.025265] 48 [ 8081.025265] 03 [ 8081.025265] 14 [ 8081.025266] c5 [ 8081.025266] e0 [ 8081.025266] 17 [ 8081.025267] 15 [ 8081.025267] 91 [ 8081.025267] 4c [ 8081.025267] 89 [ 8081.025268] 02 [ 8081.025268] 41 [ 8081.025268] 8b [ 8081.025268] 40 [ 8081.025269] 08 [ 8081.025269] 85 [ 8081.025269] c0 [ 8081.025269] 75 [ 8081.025270] 0f [ 8081.025270] 0f [ 8081.025270] 1f [ 8081.025270] 44 [ 8081.025271] 00 [ 8081.025271] 00 [ 8081.025271] f3 [ 8081.025272] 90 [ 8081.025272] 41 [ 8081.025272] 8b [ 8081.025272] 40 [ 8081.025273] 08 [ 8081.025273] <85> [ 8081.025273] c0 [ 8081.025273] 74 [ 8081.025274] f6 [ 8081.025274] 4d [ 8081.025274] 8b [ 8081.025274] 08 [ 8081.025275] 4d [ 8081.025275] 85 [ 8081.025275] c9 [ 8081.025275] 74 [ 8081.025276] 04 [ 8081.025276] 41 [ 8081.025276] 0f [ 8081.025277] 18 [ 8081.025277] 09 [ 8081.025277] 8b [ 8081.025277] 17 [ 8081.025278] 0f [ 8081.025278] b7 [ 8081.025278] c2 [ 8081.025278] [ 8081.031035] NMI watchdog: BUG: soft lockup - CPU#35 stuck for 23s! [ptlrpcd_01_16:16851] [ 8081.031035] Modules linked in: [ 8081.031036] mgc(OE) [ 8081.031036] lustre(OE) [ 8081.031037] lmv(OE) [ 8081.031037] mdc(OE) [ 8081.031037] osc(OE) [ 8081.031037] lov(OE) [ 8081.031038] fid(OE) [ 8081.031038] fld(OE) [ 8081.031038] ptlrpc(OE) [ 8081.031039] obdclass(OE) [ 8081.031039] ko2iblnd(OE) [ 8081.031039] lnet(OE) [ 8081.031040] libcfs(OE) [ 8081.031040] gdrdrv(POE) [ 8081.031040] iTCO_wdt [ 8081.031041] iTCO_vendor_support [ 8081.031041] rpcrdma [ 8081.031041] nvidia_drm(POE) [ 8081.031042] ib_iser [ 8081.031042] joydev [ 8081.031042] sb_edac [ 8081.031042] intel_powerclamp [ 8081.031043] coretemp [ 8081.031043] intel_rapl [ 8081.031043] iosf_mbi [ 8081.031044] kvm_intel [ 8081.031044] kvm [ 8081.031044] irqbypass [ 8081.031045] nvidia_modeset(POE) [ 8081.031045] sg [ 8081.031045] pcspkr [ 8081.031045] i2c_i801 [ 8081.031046] lpc_ich [ 8081.031046] nf_log_ipv4 [ 8081.031046] nf_log_common [ 8081.031047] xt_LOG [ 8081.031047] nf_conntrack_ipv4 [ 8081.031047] nf_defrag_ipv4 [ 8081.031048] xt_multiport [ 8081.031048] xt_owner [ 8081.031048] xt_conntrack [ 8081.031048] nf_conntrack [ 8081.031049] libcrc32c [ 8081.031049] iptable_filter [ 8081.031049] ipmi_si [ 8081.031050] ipmi_devintf [ 8081.031050] ipmi_msghandler [ 8081.031050] acpi_power_meter [ 8081.031051] ib_ipoib [ 8081.031051] rdma_ucm [ 8081.031051] ib_umad [ 8081.031052] iw_cxgb4 [ 8081.031052] rdma_cm [ 8081.031052] iw_cm [ 8081.031053] ib_cm [ 8081.031053] iw_cxgb3 [ 8081.031053] sch_fq_codel [ 8081.031054] binfmt_misc [ 8081.031054] msr_safe(OE) [ 8081.031054] ip_tables [ 8081.031055] nfsv3 [ 8081.031055] nfs_acl [ 8081.031055] rpcsec_gss_krb5 [ 8081.031056] auth_rpcgss [ 8081.031056] nfsv4 [ 8081.031056] dns_resolver [ 8081.031056] nfs [ 8081.031057] lockd [ 8081.031057] grace [ 8081.031057] fscache [ 8081.031058] overlay(T) [ 8081.031058] ext4 [ 8081.031058] mbcache [ 8081.031058] jbd2 [ 8081.031059] sd_mod [ 8081.031059] crc_t10dif [ 8081.031060] crct10dif_generic [ 8081.031060] nvidia_uvm(OE) [ 8081.031060] mlx5_ib [ 8081.031061] ib_uverbs [ 8081.031061] be2iscsi [ 8081.031061] ib_core [ 8081.031062] bnx2i [ 8081.031062] cnic [ 8081.031062] uio [ 8081.031062] cxgb4i [ 8081.031063] cxgb4 [ 8081.031063] cxgb3i [ 8081.031063] cxgb3 [ 8081.031064] mdio [ 8081.031064] libcxgbi [ 8081.031064] libcxgb [ 8081.031065] qla4xxx [ 8081.031065] iscsi_boot_sysfs [ 8081.031065] 8021q [ 8081.031065] garp [ 8081.031066] mrp [ 8081.031066] stp [ 8081.031066] llc [ 8081.031067] nvidia(POE) [ 8081.031067] ast [ 8081.031067] drm_kms_helper [ 8081.031068] crct10dif_pclmul [ 8081.031068] crct10dif_common [ 8081.031068] crc32_pclmul [ 8081.031069] crc32c_intel [ 8081.031069] syscopyarea [ 8081.031069] sysfillrect [ 8081.031070] sysimgblt [ 8081.031070] ghash_clmulni_intel [ 8081.031070] mlx5_core [ 8081.031070] fb_sys_fops [ 8081.031071] igb [ 8081.031071] ttm [ 8081.031071] aesni_intel [ 8081.031072] mlxfw [ 8081.031072] lrw [ 8081.031072] devlink [ 8081.031073] gf128mul [ 8081.031073] dca [ 8081.031073] glue_helper [ 8081.031073] ablk_helper [ 8081.031074] drm [ 8081.031074] dm_multipath [ 8081.031074] ptp [ 8081.031075] cryptd [ 8081.031075] i2c_algo_bit [ 8081.031075] pps_core [ 8081.031076] drm_panel_orientation_quirks [ 8081.031076] wmi [ 8081.031076] sunrpc [ 8081.031077] dm_mirror [ 8081.031077] dm_region_hash [ 8081.031077] dm_log [ 8081.031078] dm_mod [ 8081.031078] iscsi_tcp [ 8081.031078] libiscsi_tcp [ 8081.031079] libiscsi [ 8081.031079] scsi_transport_iscsi [ 8081.031079] fuse [ 8081.031080] [ 8081.031081] CPU: 35 PID: 16851 Comm: ptlrpcd_01_16 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.031082] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.031082] task: ffff8f484fb9c200 ti: ffff8f484fbb4000 task.ti: ffff8f484fbb4000 [ 8081.031083] RIP: 0010:[] [ 8081.031086] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8081.031087] RSP: 0018:ffff8f484fbb7b58 EFLAGS: 00000246 [ 8081.031087] RAX: 0000000000000000 RBX: ffff8f65f9570d80 RCX: 0000000001190000 [ 8081.031088] RDX: ffff8f487f61b8c0 RSI: 0000000000410001 RDI: ffff8f686e2b6b40 [ 8081.031088] RBP: ffff8f484fbb7b58 R08: ffff8f687f05b8c0 R09: 0000000000000000 [ 8081.031089] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.031089] R13: 0000000000000003 R14: 0000000000000013 R15: 000000002d9d0671 [ 8081.031090] FS: 0000000000000000(0000) GS:ffff8f687f040000(0000) knlGS:0000000000000000 [ 8081.031091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.031091] CR2: 00002aaaad64527d CR3: 0000003eef26a000 CR4: 00000000003607e0 [ 8081.031092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.031092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.031093] Call Trace: [ 8081.031095] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.031097] [] _raw_spin_lock+0x30/0x40 [ 8081.031103] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.031110] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.031135] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.031159] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.031182] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.031208] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.031211] [] ? wake_up_state+0x20/0x20 [ 8081.031236] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.031238] [] kthread+0xd1/0xe0 [ 8081.031240] [] ? insert_kthread_work+0x40/0x40 [ 8081.031241] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.031243] [] ? insert_kthread_work+0x40/0x40 [ 8081.031243] Code: [ 8081.031244] 13 [ 8081.031244] 48 [ 8081.031244] c1 [ 8081.031245] ea [ 8081.031245] 0d [ 8081.031245] 48 [ 8081.031245] 98 [ 8081.031246] 83 [ 8081.031246] e2 [ 8081.031246] 30 [ 8081.031247] 48 [ 8081.031247] 81 [ 8081.031247] c2 [ 8081.031248] c0 [ 8081.031248] b8 [ 8081.031248] 01 [ 8081.031248] 00 [ 8081.031249] 48 [ 8081.031249] 03 [ 8081.031249] 14 [ 8081.031249] c5 [ 8081.031250] e0 [ 8081.031250] 17 [ 8081.031250] 15 [ 8081.031250] 91 [ 8081.031251] 4c [ 8081.031251] 89 [ 8081.031251] 02 [ 8081.031251] 41 [ 8081.031252] 8b [ 8081.031252] 40 [ 8081.031252] 08 [ 8081.031252] 85 [ 8081.031253] c0 [ 8081.031253] 75 [ 8081.031253] 0f [ 8081.031253] 0f [ 8081.031254] 1f [ 8081.031254] 44 [ 8081.031254] 00 [ 8081.031254] 00 [ 8081.031255] f3 [ 8081.031255] 90 [ 8081.031256] <41> [ 8081.031256] 8b [ 8081.031256] 40 [ 8081.031256] 08 [ 8081.031257] 85 [ 8081.031257] c0 [ 8081.031257] 74 [ 8081.031257] f6 [ 8081.031257] 4d [ 8081.031258] 8b [ 8081.031258] 08 [ 8081.031258] 4d [ 8081.031259] 85 [ 8081.031259] c9 [ 8081.031259] 74 [ 8081.031259] 04 [ 8081.031260] 41 [ 8081.031260] 0f [ 8081.031260] 18 [ 8081.031260] 09 [ 8081.031261] 8b [ 8081.031261] [ 8081.041034] NMI watchdog: BUG: soft lockup - CPU#37 stuck for 23s! [ptlrpcd_00_02:16800] [ 8081.041035] Modules linked in: [ 8081.041036] mgc(OE) [ 8081.041036] lustre(OE) [ 8081.041037] lmv(OE) [ 8081.041037] mdc(OE) [ 8081.041038] osc(OE) [ 8081.041038] lov(OE) [ 8081.041039] fid(OE) [ 8081.041039] fld(OE) [ 8081.041040] ptlrpc(OE) [ 8081.041040] obdclass(OE) [ 8081.041041] ko2iblnd(OE) [ 8081.041041] lnet(OE) [ 8081.041042] libcfs(OE) [ 8081.041042] gdrdrv(POE) [ 8081.041043] iTCO_wdt [ 8081.041043] iTCO_vendor_support [ 8081.041044] rpcrdma [ 8081.041044] nvidia_drm(POE) [ 8081.041045] ib_iser [ 8081.041045] joydev [ 8081.041046] sb_edac [ 8081.041046] intel_powerclamp [ 8081.041046] coretemp [ 8081.041047] intel_rapl [ 8081.041047] iosf_mbi [ 8081.041048] kvm_intel [ 8081.041048] kvm [ 8081.041049] irqbypass [ 8081.041049] nvidia_modeset(POE) [ 8081.041050] sg [ 8081.041050] pcspkr [ 8081.041051] i2c_i801 [ 8081.041051] lpc_ich [ 8081.041051] nf_log_ipv4 [ 8081.041052] nf_log_common [ 8081.041052] xt_LOG [ 8081.041053] nf_conntrack_ipv4 [ 8081.041053] nf_defrag_ipv4 [ 8081.041054] xt_multiport [ 8081.041054] xt_owner [ 8081.041055] xt_conntrack [ 8081.041055] nf_conntrack [ 8081.041056] libcrc32c [ 8081.041056] iptable_filter [ 8081.041057] ipmi_si [ 8081.041057] ipmi_devintf [ 8081.041058] ipmi_msghandler [ 8081.041058] acpi_power_meter [ 8081.041059] ib_ipoib [ 8081.041059] rdma_ucm [ 8081.041059] ib_umad [ 8081.041060] iw_cxgb4 [ 8081.041060] rdma_cm [ 8081.041061] iw_cm [ 8081.041061] ib_cm [ 8081.041062] iw_cxgb3 [ 8081.041062] sch_fq_codel [ 8081.041063] binfmt_misc [ 8081.041063] msr_safe(OE) [ 8081.041064] ip_tables [ 8081.041064] nfsv3 [ 8081.041065] nfs_acl [ 8081.041065] rpcsec_gss_krb5 [ 8081.041065] auth_rpcgss [ 8081.041066] nfsv4 [ 8081.041066] dns_resolver [ 8081.041067] nfs [ 8081.041067] lockd [ 8081.041068] grace [ 8081.041068] fscache [ 8081.041069] overlay(T) [ 8081.041069] ext4 [ 8081.041070] mbcache [ 8081.041070] jbd2 [ 8081.041071] sd_mod [ 8081.041071] crc_t10dif [ 8081.041072] crct10dif_generic [ 8081.041072] nvidia_uvm(OE) [ 8081.041073] mlx5_ib [ 8081.041073] ib_uverbs [ 8081.041074] be2iscsi [ 8081.041074] ib_core [ 8081.041074] bnx2i [ 8081.041075] cnic [ 8081.041075] uio [ 8081.041076] cxgb4i [ 8081.041076] cxgb4 [ 8081.041077] cxgb3i [ 8081.041077] cxgb3 [ 8081.041078] mdio [ 8081.041078] libcxgbi [ 8081.041078] libcxgb [ 8081.041079] qla4xxx [ 8081.041079] iscsi_boot_sysfs [ 8081.041080] 8021q [ 8081.041080] garp [ 8081.041081] mrp [ 8081.041081] stp [ 8081.041082] llc [ 8081.041082] nvidia(POE) [ 8081.041082] ast [ 8081.041083] drm_kms_helper [ 8081.041083] crct10dif_pclmul [ 8081.041084] crct10dif_common [ 8081.041084] crc32_pclmul [ 8081.041085] crc32c_intel [ 8081.041085] syscopyarea [ 8081.041086] sysfillrect [ 8081.041086] sysimgblt [ 8081.041087] ghash_clmulni_intel [ 8081.041087] mlx5_core [ 8081.041087] fb_sys_fops [ 8081.041088] igb [ 8081.041088] ttm [ 8081.041089] aesni_intel [ 8081.041089] mlxfw [ 8081.041090] lrw [ 8081.041090] devlink [ 8081.041091] gf128mul [ 8081.041091] dca [ 8081.041091] glue_helper [ 8081.041092] ablk_helper [ 8081.041092] drm [ 8081.041093] dm_multipath [ 8081.041093] ptp [ 8081.041094] cryptd [ 8081.041094] i2c_algo_bit [ 8081.041095] pps_core [ 8081.041095] drm_panel_orientation_quirks [ 8081.041096] wmi [ 8081.041096] sunrpc [ 8081.041097] dm_mirror [ 8081.041097] dm_region_hash [ 8081.041097] dm_log [ 8081.041098] dm_mod [ 8081.041099] iscsi_tcp [ 8081.041099] libiscsi_tcp [ 8081.041099] libiscsi [ 8081.041100] scsi_transport_iscsi [ 8081.041100] fuse [ 8081.041101] [ 8081.041103] CPU: 37 PID: 16800 Comm: ptlrpcd_00_02 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.041104] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.041105] task: ffff8f484fb7b180 ti: ffff8f484c644000 task.ti: ffff8f484c644000 [ 8081.041106] RIP: 0010:[] [ 8081.041109] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8081.041110] RSP: 0018:ffff8f484c647b58 EFLAGS: 00000246 [ 8081.041111] RAX: 0000000000000000 RBX: ffff8f46d4019b00 RCX: 0000000001290000 [ 8081.041111] RDX: ffff8f687ee5b8c0 RSI: 0000000000d90001 RDI: ffff8f686e2b6b40 [ 8081.041112] RBP: ffff8f484c647b58 R08: ffff8f487f8db8c0 R09: 0000000000000000 [ 8081.041113] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.041114] R13: 0000000000000003 R14: 0000000000000013 R15: 000000001b5b3cfe [ 8081.041115] FS: 0000000000000000(0000) GS:ffff8f487f8c0000(0000) knlGS:0000000000000000 [ 8081.041116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.041117] CR2: 00007ffff7ff8000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.041118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.041119] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.041119] Call Trace: [ 8081.041122] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.041125] [] _raw_spin_lock+0x30/0x40 [ 8081.041133] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.041143] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.041178] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.041181] [] ? del_timer_sync+0x52/0x60 [ 8081.041213] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.041243] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.041278] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.041281] [] ? wake_up_state+0x20/0x20 [ 8081.041314] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.041317] [] kthread+0xd1/0xe0 [ 8081.041319] [] ? insert_kthread_work+0x40/0x40 [ 8081.041321] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.041323] [] ? insert_kthread_work+0x40/0x40 [ 8081.041324] Code: [ 8081.041325] 13 [ 8081.041325] 48 [ 8081.041326] c1 [ 8081.041326] ea [ 8081.041326] 0d [ 8081.041327] 48 [ 8081.041327] 98 [ 8081.041328] 83 [ 8081.041328] e2 [ 8081.041328] 30 [ 8081.041329] 48 [ 8081.041329] 81 [ 8081.041330] c2 [ 8081.041330] c0 [ 8081.041330] b8 [ 8081.041331] 01 [ 8081.041331] 00 [ 8081.041332] 48 [ 8081.041332] 03 [ 8081.041332] 14 [ 8081.041333] c5 [ 8081.041333] e0 [ 8081.041334] 17 [ 8081.041334] 15 [ 8081.041334] 91 [ 8081.041335] 4c [ 8081.041335] 89 [ 8081.041336] 02 [ 8081.041336] 41 [ 8081.041336] 8b [ 8081.041337] 40 [ 8081.041337] 08 [ 8081.041338] 85 [ 8081.041338] c0 [ 8081.041338] 75 [ 8081.041339] 0f [ 8081.041339] 0f [ 8081.041340] 1f [ 8081.041340] 44 [ 8081.041340] 00 [ 8081.041341] 00 [ 8081.041341] f3 [ 8081.041341] 90 [ 8081.041342] <41> [ 8081.041342] 8b [ 8081.041343] 40 [ 8081.041343] 08 [ 8081.041344] 85 [ 8081.041344] c0 [ 8081.041344] 74 [ 8081.041345] f6 [ 8081.041345] 4d [ 8081.041345] 8b [ 8081.041346] 08 [ 8081.041346] 4d [ 8081.041347] 85 [ 8081.041347] c9 [ 8081.041347] 74 [ 8081.041348] 04 [ 8081.041348] 41 [ 8081.041348] 0f [ 8081.041349] 18 [ 8081.041349] 09 [ 8081.041350] 8b [ 8081.041350] [ 8081.044034] NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [ptlrpcd_00_00:16798] [ 8081.044035] Modules linked in: [ 8081.044036] mgc(OE) [ 8081.044036] lustre(OE) [ 8081.044037] lmv(OE) [ 8081.044037] mdc(OE) [ 8081.044038] osc(OE) [ 8081.044038] lov(OE) [ 8081.044039] fid(OE) [ 8081.044039] fld(OE) [ 8081.044040] ptlrpc(OE) [ 8081.044040] obdclass(OE) [ 8081.044041] ko2iblnd(OE) [ 8081.044041] lnet(OE) [ 8081.044042] libcfs(OE) [ 8081.044042] gdrdrv(POE) [ 8081.044043] iTCO_wdt [ 8081.044043] iTCO_vendor_support [ 8081.044044] rpcrdma [ 8081.044044] nvidia_drm(POE) [ 8081.044045] ib_iser [ 8081.044045] joydev [ 8081.044046] sb_edac [ 8081.044046] intel_powerclamp [ 8081.044046] coretemp [ 8081.044047] intel_rapl [ 8081.044047] iosf_mbi [ 8081.044048] kvm_intel [ 8081.044048] kvm [ 8081.044049] irqbypass [ 8081.044049] nvidia_modeset(POE) [ 8081.044050] sg [ 8081.044050] pcspkr [ 8081.044051] i2c_i801 [ 8081.044051] lpc_ich [ 8081.044051] nf_log_ipv4 [ 8081.044052] nf_log_common [ 8081.044052] xt_LOG [ 8081.044053] nf_conntrack_ipv4 [ 8081.044053] nf_defrag_ipv4 [ 8081.044054] xt_multiport [ 8081.044054] xt_owner [ 8081.044055] xt_conntrack [ 8081.044055] nf_conntrack [ 8081.044055] libcrc32c [ 8081.044056] iptable_filter [ 8081.044056] ipmi_si [ 8081.044057] ipmi_devintf [ 8081.044057] ipmi_msghandler [ 8081.044058] acpi_power_meter [ 8081.044058] ib_ipoib [ 8081.044059] rdma_ucm [ 8081.044059] ib_umad [ 8081.044060] iw_cxgb4 [ 8081.044060] rdma_cm [ 8081.044060] iw_cm [ 8081.044061] ib_cm [ 8081.044061] iw_cxgb3 [ 8081.044062] sch_fq_codel [ 8081.044062] binfmt_misc [ 8081.044063] msr_safe(OE) [ 8081.044063] ip_tables [ 8081.044064] nfsv3 [ 8081.044064] nfs_acl [ 8081.044065] rpcsec_gss_krb5 [ 8081.044065] auth_rpcgss [ 8081.044065] nfsv4 [ 8081.044066] dns_resolver [ 8081.044066] nfs [ 8081.044067] lockd [ 8081.044067] grace [ 8081.044068] fscache [ 8081.044068] overlay(T) [ 8081.044069] ext4 [ 8081.044069] mbcache [ 8081.044070] jbd2 [ 8081.044070] sd_mod [ 8081.044070] crc_t10dif [ 8081.044071] crct10dif_generic [ 8081.044072] nvidia_uvm(OE) [ 8081.044072] mlx5_ib [ 8081.044072] ib_uverbs [ 8081.044073] be2iscsi [ 8081.044073] ib_core [ 8081.044074] bnx2i [ 8081.044074] cnic [ 8081.044075] uio [ 8081.044075] cxgb4i [ 8081.044075] cxgb4 [ 8081.044076] cxgb3i [ 8081.044076] cxgb3 [ 8081.044077] mdio [ 8081.044077] libcxgbi [ 8081.044078] libcxgb [ 8081.044078] qla4xxx [ 8081.044078] iscsi_boot_sysfs [ 8081.044079] 8021q [ 8081.044079] garp [ 8081.044080] mrp [ 8081.044080] stp [ 8081.044080] llc [ 8081.044081] nvidia(POE) [ 8081.044081] ast [ 8081.044082] drm_kms_helper [ 8081.044082] crct10dif_pclmul [ 8081.044083] crct10dif_common [ 8081.044083] crc32_pclmul [ 8081.044084] crc32c_intel [ 8081.044084] syscopyarea [ 8081.044085] sysfillrect [ 8081.044085] sysimgblt [ 8081.044086] ghash_clmulni_intel [ 8081.044086] mlx5_core [ 8081.044086] fb_sys_fops [ 8081.044087] igb [ 8081.044087] ttm [ 8081.044088] aesni_intel [ 8081.044088] mlxfw [ 8081.044089] lrw [ 8081.044089] devlink [ 8081.044089] gf128mul [ 8081.044090] dca [ 8081.044090] glue_helper [ 8081.044091] ablk_helper [ 8081.044091] drm [ 8081.044092] dm_multipath [ 8081.044092] ptp [ 8081.044092] cryptd [ 8081.044093] i2c_algo_bit [ 8081.044093] pps_core [ 8081.044094] drm_panel_orientation_quirks [ 8081.044095] wmi [ 8081.044095] sunrpc [ 8081.044096] dm_mirror [ 8081.044096] dm_region_hash [ 8081.044096] dm_log [ 8081.044097] dm_mod [ 8081.044097] iscsi_tcp [ 8081.044098] libiscsi_tcp [ 8081.044098] libiscsi [ 8081.044099] scsi_transport_iscsi [ 8081.044099] fuse [ 8081.044099] [ 8081.044102] CPU: 38 PID: 16798 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.044102] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.044104] task: ffff8f484fb79080 ti: ffff8f484c680000 task.ti: ffff8f484c680000 [ 8081.044104] RIP: 0010:[] [ 8081.044107] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8081.044108] RSP: 0018:ffff8f484c683b58 EFLAGS: 00000246 [ 8081.044109] RAX: 0000000000000000 RBX: ffff8f46d3ec2400 RCX: 0000000001310000 [ 8081.044110] RDX: ffff8f487f79b8c0 RSI: 0000000000710001 RDI: ffff8f686e2b6b40 [ 8081.044111] RBP: ffff8f484c683b58 R08: ffff8f487f91b8c0 R09: 0000000000000000 [ 8081.044111] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.044112] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000c8aa1eca [ 8081.044114] FS: 0000000000000000(0000) GS:ffff8f487f900000(0000) knlGS:0000000000000000 [ 8081.044115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.044115] CR2: 00002aaaab1139e5 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.044116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.044117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.044118] Call Trace: [ 8081.044121] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.044123] [] _raw_spin_lock+0x30/0x40 [ 8081.044131] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.044141] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.044176] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.044179] [] ? del_timer_sync+0x52/0x60 [ 8081.044210] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.044240] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.044275] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.044278] [] ? wake_up_state+0x20/0x20 [ 8081.044311] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.044314] [] kthread+0xd1/0xe0 [ 8081.044316] [] ? insert_kthread_work+0x40/0x40 [ 8081.044318] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.044320] [] ? insert_kthread_work+0x40/0x40 [ 8081.044321] Code: [ 8081.044321] 13 [ 8081.044322] 48 [ 8081.044322] c1 [ 8081.044323] ea [ 8081.044323] 0d [ 8081.044323] 48 [ 8081.044324] 98 [ 8081.044324] 83 [ 8081.044324] e2 [ 8081.044325] 30 [ 8081.044325] 48 [ 8081.044326] 81 [ 8081.044326] c2 [ 8081.044326] c0 [ 8081.044327] b8 [ 8081.044327] 01 [ 8081.044328] 00 [ 8081.044328] 48 [ 8081.044328] 03 [ 8081.044329] 14 [ 8081.044329] c5 [ 8081.044330] e0 [ 8081.044330] 17 [ 8081.044330] 15 [ 8081.044331] 91 [ 8081.044331] 4c [ 8081.044332] 89 [ 8081.044332] 02 [ 8081.044332] 41 [ 8081.044333] 8b [ 8081.044333] 40 [ 8081.044334] 08 [ 8081.044334] 85 [ 8081.044334] c0 [ 8081.044335] 75 [ 8081.044335] 0f [ 8081.044335] 0f [ 8081.044336] 1f [ 8081.044336] 44 [ 8081.044337] 00 [ 8081.044337] 00 [ 8081.044337] f3 [ 8081.044338] 90 [ 8081.044338] <41> [ 8081.044339] 8b [ 8081.044339] 40 [ 8081.044339] 08 [ 8081.044340] 85 [ 8081.044340] c0 [ 8081.044341] 74 [ 8081.044341] f6 [ 8081.044341] 4d [ 8081.044342] 8b [ 8081.044342] 08 [ 8081.044342] 4d [ 8081.044343] 85 [ 8081.044343] c9 [ 8081.044344] 74 [ 8081.044344] 04 [ 8081.044344] 41 [ 8081.044345] 0f [ 8081.044345] 18 [ 8081.044346] 09 [ 8081.044346] 8b [ 8081.044346] [ 8081.047034] NMI watchdog: BUG: soft lockup - CPU#39 stuck for 23s! [ptlrpcd_00_33:16831] [ 8081.047035] Modules linked in: [ 8081.047035] mgc(OE) [ 8081.047036] lustre(OE) [ 8081.047037] lmv(OE) [ 8081.047037] mdc(OE) [ 8081.047037] osc(OE) [ 8081.047038] lov(OE) [ 8081.047038] fid(OE) [ 8081.047038] fld(OE) [ 8081.047039] ptlrpc(OE) [ 8081.047039] obdclass(OE) [ 8081.047039] ko2iblnd(OE) [ 8081.047040] lnet(OE) [ 8081.047040] libcfs(OE) [ 8081.047040] gdrdrv(POE) [ 8081.047041] iTCO_wdt [ 8081.047041] iTCO_vendor_support [ 8081.047041] rpcrdma [ 8081.047042] nvidia_drm(POE) [ 8081.047042] ib_iser [ 8081.047042] joydev [ 8081.047043] sb_edac [ 8081.047043] intel_powerclamp [ 8081.047043] coretemp [ 8081.047044] intel_rapl [ 8081.047044] iosf_mbi [ 8081.047044] kvm_intel [ 8081.047045] kvm [ 8081.047045] irqbypass [ 8081.047045] nvidia_modeset(POE) [ 8081.047046] sg [ 8081.047046] pcspkr [ 8081.047046] i2c_i801 [ 8081.047047] lpc_ich [ 8081.047047] nf_log_ipv4 [ 8081.047047] nf_log_common [ 8081.047047] xt_LOG [ 8081.047048] nf_conntrack_ipv4 [ 8081.047048] nf_defrag_ipv4 [ 8081.047048] xt_multiport [ 8081.047049] xt_owner [ 8081.047049] xt_conntrack [ 8081.047049] nf_conntrack [ 8081.047050] libcrc32c [ 8081.047050] iptable_filter [ 8081.047050] ipmi_si [ 8081.047051] ipmi_devintf [ 8081.047051] ipmi_msghandler [ 8081.047051] acpi_power_meter [ 8081.047052] ib_ipoib [ 8081.047052] rdma_ucm [ 8081.047052] ib_umad [ 8081.047053] iw_cxgb4 [ 8081.047053] rdma_cm [ 8081.047053] iw_cm [ 8081.047053] ib_cm [ 8081.047054] iw_cxgb3 [ 8081.047054] sch_fq_codel [ 8081.047054] binfmt_misc [ 8081.047055] msr_safe(OE) [ 8081.047055] ip_tables [ 8081.047055] nfsv3 [ 8081.047056] nfs_acl [ 8081.047056] rpcsec_gss_krb5 [ 8081.047056] auth_rpcgss [ 8081.047057] nfsv4 [ 8081.047057] dns_resolver [ 8081.047057] nfs [ 8081.047057] lockd [ 8081.047058] grace [ 8081.047058] fscache [ 8081.047058] overlay(T) [ 8081.047059] ext4 [ 8081.047059] mbcache [ 8081.047059] jbd2 [ 8081.047060] sd_mod [ 8081.047060] crc_t10dif [ 8081.047060] crct10dif_generic [ 8081.047061] nvidia_uvm(OE) [ 8081.047061] mlx5_ib [ 8081.047061] ib_uverbs [ 8081.047061] be2iscsi [ 8081.047062] ib_core [ 8081.047062] bnx2i [ 8081.047062] cnic [ 8081.047063] uio [ 8081.047063] cxgb4i [ 8081.047063] cxgb4 [ 8081.047063] cxgb3i [ 8081.047064] cxgb3 [ 8081.047064] mdio [ 8081.047064] libcxgbi [ 8081.047065] libcxgb [ 8081.047065] qla4xxx [ 8081.047065] iscsi_boot_sysfs [ 8081.047065] 8021q [ 8081.047066] garp [ 8081.047066] mrp [ 8081.047066] stp [ 8081.047067] llc [ 8081.047067] nvidia(POE) [ 8081.047067] ast [ 8081.047068] drm_kms_helper [ 8081.047068] crct10dif_pclmul [ 8081.047068] crct10dif_common [ 8081.047068] crc32_pclmul [ 8081.047069] crc32c_intel [ 8081.047069] syscopyarea [ 8081.047069] sysfillrect [ 8081.047070] sysimgblt [ 8081.047070] ghash_clmulni_intel [ 8081.047070] mlx5_core [ 8081.047071] fb_sys_fops [ 8081.047071] igb [ 8081.047071] ttm [ 8081.047071] aesni_intel [ 8081.047072] mlxfw [ 8081.047072] lrw [ 8081.047072] devlink [ 8081.047073] gf128mul [ 8081.047073] dca [ 8081.047073] glue_helper [ 8081.047073] ablk_helper [ 8081.047074] drm [ 8081.047074] dm_multipath [ 8081.047074] ptp [ 8081.047075] cryptd [ 8081.047075] i2c_algo_bit [ 8081.047075] pps_core [ 8081.047076] drm_panel_orientation_quirks [ 8081.047076] wmi [ 8081.047076] sunrpc [ 8081.047076] dm_mirror [ 8081.047077] dm_region_hash [ 8081.047077] dm_log [ 8081.047077] dm_mod [ 8081.047078] iscsi_tcp [ 8081.047078] libiscsi_tcp [ 8081.047078] libiscsi [ 8081.047079] scsi_transport_iscsi [ 8081.047079] fuse [ 8081.047079] [ 8081.047081] CPU: 39 PID: 16831 Comm: ptlrpcd_00_33 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.047082] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.047082] task: ffff8f484f80e300 ti: ffff8f484f82c000 task.ti: ffff8f484f82c000 [ 8081.047083] RIP: 0010:[] [ 8081.047085] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.047086] RSP: 0018:ffff8f484f82fb58 EFLAGS: 00000246 [ 8081.047087] RAX: 0000000000000000 RBX: ffff8f46a8906300 RCX: 0000000001390000 [ 8081.047087] RDX: ffff8f487f55b8c0 RSI: 0000000000290001 RDI: ffff8f686e2b6b40 [ 8081.047088] RBP: ffff8f484f82fb58 R08: ffff8f487f95b8c0 R09: 0000000000000000 [ 8081.047088] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.047089] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d1e52d70 [ 8081.047090] FS: 0000000000000000(0000) GS:ffff8f487f940000(0000) knlGS:0000000000000000 [ 8081.047091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.047091] CR2: 00000000006bd400 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8081.047092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.047092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.047093] Call Trace: [ 8081.047096] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.047097] [] _raw_spin_lock+0x30/0x40 [ 8081.047104] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.047111] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.047136] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.047138] [] ? del_timer_sync+0x52/0x60 [ 8081.047162] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.047184] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.047210] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.047212] [] ? wake_up_state+0x20/0x20 [ 8081.047238] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.047240] [] kthread+0xd1/0xe0 [ 8081.047241] [] ? insert_kthread_work+0x40/0x40 [ 8081.047243] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.047244] [] ? insert_kthread_work+0x40/0x40 [ 8081.047245] Code: [ 8081.047245] 0d [ 8081.047245] 48 [ 8081.047246] 98 [ 8081.047246] 83 [ 8081.047246] e2 [ 8081.047246] 30 [ 8081.047247] 48 [ 8081.047247] 81 [ 8081.047247] c2 [ 8081.047247] c0 [ 8081.047248] b8 [ 8081.047248] 01 [ 8081.047248] 00 [ 8081.047248] 48 [ 8081.047249] 03 [ 8081.047249] 14 [ 8081.047249] c5 [ 8081.047249] e0 [ 8081.047250] 17 [ 8081.047250] 15 [ 8081.047250] 91 [ 8081.047250] 4c [ 8081.047251] 89 [ 8081.047251] 02 [ 8081.047251] 41 [ 8081.047251] 8b [ 8081.047252] 40 [ 8081.047252] 08 [ 8081.047252] 85 [ 8081.047252] c0 [ 8081.047253] 75 [ 8081.047253] 0f [ 8081.047253] 0f [ 8081.047253] 1f [ 8081.047254] 44 [ 8081.047254] 00 [ 8081.047254] 00 [ 8081.047254] f3 [ 8081.047255] 90 [ 8081.047255] 41 [ 8081.047255] 8b [ 8081.047255] 40 [ 8081.047256] 08 [ 8081.047256] <85> [ 8081.047256] c0 [ 8081.047256] 74 [ 8081.047257] f6 [ 8081.047257] 4d [ 8081.047257] 8b [ 8081.047257] 08 [ 8081.047258] 4d [ 8081.047258] 85 [ 8081.047258] c9 [ 8081.047258] 74 [ 8081.047258] 04 [ 8081.047259] 41 [ 8081.047259] 0f [ 8081.047259] 18 [ 8081.047259] 09 [ 8081.047260] 8b [ 8081.047260] 17 [ 8081.047260] 0f [ 8081.047260] b7 [ 8081.047261] c2 [ 8081.047261] [ 8081.086033] NMI watchdog: BUG: soft lockup - CPU#52 stuck for 23s! [ptlrpcd_00_19:16817] [ 8081.086034] Modules linked in: [ 8081.086034] mgc(OE) [ 8081.086035] lustre(OE) [ 8081.086035] lmv(OE) [ 8081.086036] mdc(OE) [ 8081.086036] osc(OE) [ 8081.086037] lov(OE) [ 8081.086037] fid(OE) [ 8081.086038] fld(OE) [ 8081.086038] ptlrpc(OE) [ 8081.086039] obdclass(OE) [ 8081.086039] ko2iblnd(OE) [ 8081.086040] lnet(OE) [ 8081.086040] libcfs(OE) [ 8081.086041] gdrdrv(POE) [ 8081.086041] iTCO_wdt [ 8081.086042] iTCO_vendor_support [ 8081.086042] rpcrdma [ 8081.086043] nvidia_drm(POE) [ 8081.086043] ib_iser [ 8081.086044] joydev [ 8081.086044] sb_edac [ 8081.086044] intel_powerclamp [ 8081.086045] coretemp [ 8081.086045] intel_rapl [ 8081.086046] iosf_mbi [ 8081.086046] kvm_intel [ 8081.086047] kvm [ 8081.086047] irqbypass [ 8081.086048] nvidia_modeset(POE) [ 8081.086048] sg [ 8081.086048] pcspkr [ 8081.086049] i2c_i801 [ 8081.086049] lpc_ich [ 8081.086050] nf_log_ipv4 [ 8081.086050] nf_log_common [ 8081.086051] xt_LOG [ 8081.086051] nf_conntrack_ipv4 [ 8081.086052] nf_defrag_ipv4 [ 8081.086052] xt_multiport [ 8081.086052] xt_owner [ 8081.086053] xt_conntrack [ 8081.086053] nf_conntrack [ 8081.086054] libcrc32c [ 8081.086054] iptable_filter [ 8081.086055] ipmi_si [ 8081.086055] ipmi_devintf [ 8081.086055] ipmi_msghandler [ 8081.086056] acpi_power_meter [ 8081.086056] ib_ipoib [ 8081.086057] rdma_ucm [ 8081.086057] ib_umad [ 8081.086058] iw_cxgb4 [ 8081.086058] rdma_cm [ 8081.086058] iw_cm [ 8081.086059] ib_cm [ 8081.086059] iw_cxgb3 [ 8081.086060] sch_fq_codel [ 8081.086060] binfmt_misc [ 8081.086061] msr_safe(OE) [ 8081.086061] ip_tables [ 8081.086062] nfsv3 [ 8081.086062] nfs_acl [ 8081.086063] rpcsec_gss_krb5 [ 8081.086063] auth_rpcgss [ 8081.086064] nfsv4 [ 8081.086064] dns_resolver [ 8081.086064] nfs [ 8081.086065] lockd [ 8081.086065] grace [ 8081.086066] fscache [ 8081.086066] overlay(T) [ 8081.086067] ext4 [ 8081.086067] mbcache [ 8081.086068] jbd2 [ 8081.086068] sd_mod [ 8081.086068] crc_t10dif [ 8081.086069] crct10dif_generic [ 8081.086069] nvidia_uvm(OE) [ 8081.086070] mlx5_ib [ 8081.086070] ib_uverbs [ 8081.086071] be2iscsi [ 8081.086071] ib_core [ 8081.086072] bnx2i [ 8081.086072] cnic [ 8081.086073] uio [ 8081.086073] cxgb4i [ 8081.086073] cxgb4 [ 8081.086074] cxgb3i [ 8081.086074] cxgb3 [ 8081.086075] mdio [ 8081.086075] libcxgbi [ 8081.086076] libcxgb [ 8081.086076] qla4xxx [ 8081.086077] iscsi_boot_sysfs [ 8081.086077] 8021q [ 8081.086077] garp [ 8081.086078] mrp [ 8081.086078] stp [ 8081.086079] llc [ 8081.086079] nvidia(POE) [ 8081.086080] ast [ 8081.086080] drm_kms_helper [ 8081.086080] crct10dif_pclmul [ 8081.086081] crct10dif_common [ 8081.086081] crc32_pclmul [ 8081.086082] crc32c_intel [ 8081.086082] syscopyarea [ 8081.086083] sysfillrect [ 8081.086083] sysimgblt [ 8081.086084] ghash_clmulni_intel [ 8081.086084] mlx5_core [ 8081.086085] fb_sys_fops [ 8081.086085] igb [ 8081.086085] ttm [ 8081.086086] aesni_intel [ 8081.086086] mlxfw [ 8081.086087] lrw [ 8081.086087] devlink [ 8081.086088] gf128mul [ 8081.086088] dca [ 8081.086088] glue_helper [ 8081.086089] ablk_helper [ 8081.086089] drm [ 8081.086090] dm_multipath [ 8081.086090] ptp [ 8081.086091] cryptd [ 8081.086091] i2c_algo_bit [ 8081.086092] pps_core [ 8081.086092] drm_panel_orientation_quirks [ 8081.086093] wmi [ 8081.086093] sunrpc [ 8081.086093] dm_mirror [ 8081.086094] dm_region_hash [ 8081.086094] dm_log [ 8081.086095] dm_mod [ 8081.086095] iscsi_tcp [ 8081.086096] libiscsi_tcp [ 8081.086096] libiscsi [ 8081.086097] scsi_transport_iscsi [ 8081.086097] fuse [ 8081.086097] [ 8081.086100] CPU: 52 PID: 16817 Comm: ptlrpcd_00_19 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.086100] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.086101] task: ffff8f484c626300 ti: ffff8f484d3cc000 task.ti: ffff8f484d3cc000 [ 8081.086102] RIP: 0010:[] [ 8081.086105] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.086106] RSP: 0018:ffff8f484d3cfb58 EFLAGS: 00000246 [ 8081.086107] RAX: 0000000000000000 RBX: ffff8f46cf3f3f00 RCX: 0000000001a10000 [ 8081.086108] RDX: ffff8f487f81b8c0 RSI: 0000000000810001 RDI: ffff8f686e2b6b40 [ 8081.086109] RBP: ffff8f484d3cfb58 R08: ffff8f487fc9b8c0 R09: 0000000000000000 [ 8081.086109] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.086110] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000af73bea2 [ 8081.086112] FS: 0000000000000000(0000) GS:ffff8f487fc80000(0000) knlGS:0000000000000000 [ 8081.086112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.086113] CR2: 00002aaaab1114b1 CR3: 0000003ff8218000 CR4: 00000000003607e0 [ 8081.086114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.086115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.086116] Call Trace: [ 8081.086119] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.086121] [] _raw_spin_lock+0x30/0x40 [ 8081.086129] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.086139] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.086170] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.086174] [] ? del_timer_sync+0x52/0x60 [ 8081.086205] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.086235] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.086269] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.086272] [] ? wake_up_state+0x20/0x20 [ 8081.086305] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.086307] [] kthread+0xd1/0xe0 [ 8081.086310] [] ? insert_kthread_work+0x40/0x40 [ 8081.086311] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.086313] [] ? insert_kthread_work+0x40/0x40 [ 8081.086314] Code: [ 8081.086315] 0d [ 8081.086316] 48 [ 8081.086316] 98 [ 8081.086316] 83 [ 8081.086317] e2 [ 8081.086317] 30 [ 8081.086317] 48 [ 8081.086318] 81 [ 8081.086318] c2 [ 8081.086319] c0 [ 8081.086319] b8 [ 8081.086319] 01 [ 8081.086320] 00 [ 8081.086320] 48 [ 8081.086321] 03 [ 8081.086321] 14 [ 8081.086321] c5 [ 8081.086322] e0 [ 8081.086322] 17 [ 8081.086323] 15 [ 8081.086323] 91 [ 8081.086323] 4c [ 8081.086324] 89 [ 8081.086324] 02 [ 8081.086325] 41 [ 8081.086325] 8b [ 8081.086325] 40 [ 8081.086326] 08 [ 8081.086326] 85 [ 8081.086326] c0 [ 8081.086327] 75 [ 8081.086327] 0f [ 8081.086328] 0f [ 8081.086328] 1f [ 8081.086328] 44 [ 8081.086329] 00 [ 8081.086329] 00 [ 8081.086330] f3 [ 8081.086330] 90 [ 8081.086331] 41 [ 8081.086331] 8b [ 8081.086332] 40 [ 8081.086332] 08 [ 8081.086332] <85> [ 8081.086333] c0 [ 8081.086333] 74 [ 8081.086334] f6 [ 8081.086334] 4d [ 8081.086334] 8b [ 8081.086335] 08 [ 8081.086335] 4d [ 8081.086335] 85 [ 8081.086336] c9 [ 8081.086336] 74 [ 8081.086337] 04 [ 8081.086337] 41 [ 8081.086337] 0f [ 8081.086338] 18 [ 8081.086338] 09 [ 8081.086338] 8b [ 8081.086339] 17 [ 8081.086339] 0f [ 8081.086340] b7 [ 8081.086340] c2 [ 8081.086340] [ 8081.094034] NMI watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [ptlrpcd_01_18:16853] [ 8081.094034] Modules linked in: [ 8081.094035] mgc(OE) [ 8081.094036] lustre(OE) [ 8081.094036] lmv(OE) [ 8081.094037] mdc(OE) [ 8081.094037] osc(OE) [ 8081.094038] lov(OE) [ 8081.094038] fid(OE) [ 8081.094038] fld(OE) [ 8081.094039] ptlrpc(OE) [ 8081.094040] obdclass(OE) [ 8081.094040] ko2iblnd(OE) [ 8081.094041] lnet(OE) [ 8081.094041] libcfs(OE) [ 8081.094042] gdrdrv(POE) [ 8081.094042] iTCO_wdt [ 8081.094043] iTCO_vendor_support [ 8081.094043] rpcrdma [ 8081.094044] nvidia_drm(POE) [ 8081.094044] ib_iser [ 8081.094044] joydev [ 8081.094045] sb_edac [ 8081.094045] intel_powerclamp [ 8081.094046] coretemp [ 8081.094046] intel_rapl [ 8081.094047] iosf_mbi [ 8081.094047] kvm_intel [ 8081.094047] kvm [ 8081.094048] irqbypass [ 8081.094049] nvidia_modeset(POE) [ 8081.094049] sg [ 8081.094049] pcspkr [ 8081.094050] i2c_i801 [ 8081.094050] lpc_ich [ 8081.094051] nf_log_ipv4 [ 8081.094051] nf_log_common [ 8081.094052] xt_LOG [ 8081.094052] nf_conntrack_ipv4 [ 8081.094052] nf_defrag_ipv4 [ 8081.094053] xt_multiport [ 8081.094053] xt_owner [ 8081.094054] xt_conntrack [ 8081.094054] nf_conntrack [ 8081.094055] libcrc32c [ 8081.094055] iptable_filter [ 8081.094055] ipmi_si [ 8081.094056] ipmi_devintf [ 8081.094056] ipmi_msghandler [ 8081.094057] acpi_power_meter [ 8081.094057] ib_ipoib [ 8081.094058] rdma_ucm [ 8081.094058] ib_umad [ 8081.094059] iw_cxgb4 [ 8081.094059] rdma_cm [ 8081.094060] iw_cm [ 8081.094060] ib_cm [ 8081.094061] iw_cxgb3 [ 8081.094061] sch_fq_codel [ 8081.094062] binfmt_misc [ 8081.094062] msr_safe(OE) [ 8081.094063] ip_tables [ 8081.094063] nfsv3 [ 8081.094063] nfs_acl [ 8081.094064] rpcsec_gss_krb5 [ 8081.094064] auth_rpcgss [ 8081.094065] nfsv4 [ 8081.094065] dns_resolver [ 8081.094066] nfs [ 8081.094066] lockd [ 8081.094067] grace [ 8081.094067] fscache [ 8081.094068] overlay(T) [ 8081.094068] ext4 [ 8081.094069] mbcache [ 8081.094069] jbd2 [ 8081.094070] sd_mod [ 8081.094070] crc_t10dif [ 8081.094071] crct10dif_generic [ 8081.094071] nvidia_uvm(OE) [ 8081.094072] mlx5_ib [ 8081.094072] ib_uverbs [ 8081.094073] be2iscsi [ 8081.094073] ib_core [ 8081.094073] bnx2i [ 8081.094074] cnic [ 8081.094074] uio [ 8081.094075] cxgb4i [ 8081.094076] cxgb4 [ 8081.094076] cxgb3i [ 8081.094077] cxgb3 [ 8081.094077] mdio [ 8081.094077] libcxgbi [ 8081.094078] libcxgb [ 8081.094078] qla4xxx [ 8081.094079] iscsi_boot_sysfs [ 8081.094079] 8021q [ 8081.094080] garp [ 8081.094080] mrp [ 8081.094081] stp [ 8081.094081] llc [ 8081.094082] nvidia(POE) [ 8081.094082] ast [ 8081.094083] drm_kms_helper [ 8081.094083] crct10dif_pclmul [ 8081.094084] crct10dif_common [ 8081.094084] crc32_pclmul [ 8081.094084] crc32c_intel [ 8081.094085] syscopyarea [ 8081.094085] sysfillrect [ 8081.094086] sysimgblt [ 8081.094086] ghash_clmulni_intel [ 8081.094087] mlx5_core [ 8081.094087] fb_sys_fops [ 8081.094088] igb [ 8081.094088] ttm [ 8081.094088] aesni_intel [ 8081.094089] mlxfw [ 8081.094089] lrw [ 8081.094090] devlink [ 8081.094090] gf128mul [ 8081.094091] dca [ 8081.094091] glue_helper [ 8081.094092] ablk_helper [ 8081.094092] drm [ 8081.094093] dm_multipath [ 8081.094093] ptp [ 8081.094093] cryptd [ 8081.094094] i2c_algo_bit [ 8081.094094] pps_core [ 8081.094095] drm_panel_orientation_quirks [ 8081.094095] wmi [ 8081.094096] sunrpc [ 8081.094096] dm_mirror [ 8081.094097] dm_region_hash [ 8081.094097] dm_log [ 8081.094098] dm_mod [ 8081.094098] iscsi_tcp [ 8081.094099] libiscsi_tcp [ 8081.094099] libiscsi [ 8081.094100] scsi_transport_iscsi [ 8081.094100] fuse [ 8081.094101] [ 8081.094103] CPU: 54 PID: 16853 Comm: ptlrpcd_01_18 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.094104] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.094105] task: ffff8f484fb9e300 ti: ffff8f484fbbc000 task.ti: ffff8f484fbbc000 [ 8081.094106] RIP: 0010:[] [ 8081.094109] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8081.094110] RSP: 0018:ffff8f484fbbfb58 EFLAGS: 00000246 [ 8081.094110] RAX: 0000000000000000 RBX: ffff8f65f9373600 RCX: 0000000001b10000 [ 8081.094111] RDX: ffff8f687ed5b8c0 RSI: 0000000000b90001 RDI: ffff8f686e2b6b40 [ 8081.094112] RBP: ffff8f484fbbfb58 R08: ffff8f687f09b8c0 R09: 0000000000000000 [ 8081.094113] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.094114] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003dee356c [ 8081.094116] FS: 0000000000000000(0000) GS:ffff8f687f080000(0000) knlGS:0000000000000000 [ 8081.094117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.094117] CR2: 0000000000640558 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.094118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.094119] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.094120] Call Trace: [ 8081.094123] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.094125] [] _raw_spin_lock+0x30/0x40 [ 8081.094133] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.094143] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.094178] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.094211] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.094241] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.094276] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.094279] [] ? wake_up_state+0x20/0x20 [ 8081.094326] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.094329] [] kthread+0xd1/0xe0 [ 8081.094331] [] ? insert_kthread_work+0x40/0x40 [ 8081.094333] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.094335] [] ? insert_kthread_work+0x40/0x40 [ 8081.094336] Code: [ 8081.094337] 13 [ 8081.094337] 48 [ 8081.094338] c1 [ 8081.094338] ea [ 8081.094339] 0d [ 8081.094339] 48 [ 8081.094339] 98 [ 8081.094340] 83 [ 8081.094340] e2 [ 8081.094341] 30 [ 8081.094341] 48 [ 8081.094341] 81 [ 8081.094342] c2 [ 8081.094342] c0 [ 8081.094343] b8 [ 8081.094343] 01 [ 8081.094344] 00 [ 8081.094344] 48 [ 8081.094344] 03 [ 8081.094345] 14 [ 8081.094345] c5 [ 8081.094345] e0 [ 8081.094346] 17 [ 8081.094346] 15 [ 8081.094347] 91 [ 8081.094347] 4c [ 8081.094348] 89 [ 8081.094348] 02 [ 8081.094348] 41 [ 8081.094349] 8b [ 8081.094349] 40 [ 8081.094350] 08 [ 8081.094350] 85 [ 8081.094350] c0 [ 8081.094351] 75 [ 8081.094351] 0f [ 8081.094352] 0f [ 8081.094352] 1f [ 8081.094352] 44 [ 8081.094353] 00 [ 8081.094353] 00 [ 8081.094354] f3 [ 8081.094354] 90 [ 8081.094354] <41> [ 8081.094355] 8b [ 8081.094355] 40 [ 8081.094356] 08 [ 8081.094356] 85 [ 8081.094356] c0 [ 8081.094357] 74 [ 8081.094357] f6 [ 8081.094358] 4d [ 8081.094358] 8b [ 8081.094358] 08 [ 8081.094359] 4d [ 8081.094359] 85 [ 8081.094359] c9 [ 8081.094360] 74 [ 8081.094360] 04 [ 8081.094361] 41 [ 8081.094361] 0f [ 8081.094361] 18 [ 8081.094362] 09 [ 8081.094362] 8b [ 8081.094362] [ 8081.097033] NMI watchdog: BUG: soft lockup - CPU#55 stuck for 23s! [ptlrpcd_01_26:16861] [ 8081.097033] Modules linked in: [ 8081.097034] mgc(OE) [ 8081.097034] lustre(OE) [ 8081.097035] lmv(OE) [ 8081.097035] mdc(OE) [ 8081.097036] osc(OE) [ 8081.097036] lov(OE) [ 8081.097037] fid(OE) [ 8081.097037] fld(OE) [ 8081.097038] ptlrpc(OE) [ 8081.097038] obdclass(OE) [ 8081.097039] ko2iblnd(OE) [ 8081.097039] lnet(OE) [ 8081.097040] libcfs(OE) [ 8081.097040] gdrdrv(POE) [ 8081.097041] iTCO_wdt [ 8081.097041] iTCO_vendor_support [ 8081.097042] rpcrdma [ 8081.097042] nvidia_drm(POE) [ 8081.097043] ib_iser [ 8081.097043] joydev [ 8081.097043] sb_edac [ 8081.097044] intel_powerclamp [ 8081.097044] coretemp [ 8081.097045] intel_rapl [ 8081.097045] iosf_mbi [ 8081.097046] kvm_intel [ 8081.097046] kvm [ 8081.097046] irqbypass [ 8081.097047] nvidia_modeset(POE) [ 8081.097047] sg [ 8081.097048] pcspkr [ 8081.097048] i2c_i801 [ 8081.097049] lpc_ich [ 8081.097049] nf_log_ipv4 [ 8081.097050] nf_log_common [ 8081.097050] xt_LOG [ 8081.097050] nf_conntrack_ipv4 [ 8081.097051] nf_defrag_ipv4 [ 8081.097051] xt_multiport [ 8081.097052] xt_owner [ 8081.097052] xt_conntrack [ 8081.097053] nf_conntrack [ 8081.097053] libcrc32c [ 8081.097054] iptable_filter [ 8081.097054] ipmi_si [ 8081.097055] ipmi_devintf [ 8081.097055] ipmi_msghandler [ 8081.097056] acpi_power_meter [ 8081.097056] ib_ipoib [ 8081.097057] rdma_ucm [ 8081.097057] ib_umad [ 8081.097058] iw_cxgb4 [ 8081.097058] rdma_cm [ 8081.097059] iw_cm [ 8081.097059] ib_cm [ 8081.097059] iw_cxgb3 [ 8081.097060] sch_fq_codel [ 8081.097060] binfmt_misc [ 8081.097061] msr_safe(OE) [ 8081.097061] ip_tables [ 8081.097062] nfsv3 [ 8081.097062] nfs_acl [ 8081.097063] rpcsec_gss_krb5 [ 8081.097063] auth_rpcgss [ 8081.097064] nfsv4 [ 8081.097064] dns_resolver [ 8081.097065] nfs [ 8081.097065] lockd [ 8081.097066] grace [ 8081.097066] fscache [ 8081.097067] overlay(T) [ 8081.097067] ext4 [ 8081.097068] mbcache [ 8081.097068] jbd2 [ 8081.097068] sd_mod [ 8081.097069] crc_t10dif [ 8081.097069] crct10dif_generic [ 8081.097070] nvidia_uvm(OE) [ 8081.097070] mlx5_ib [ 8081.097071] ib_uverbs [ 8081.097071] be2iscsi [ 8081.097072] ib_core [ 8081.097072] bnx2i [ 8081.097073] cnic [ 8081.097073] uio [ 8081.097074] cxgb4i [ 8081.097074] cxgb4 [ 8081.097075] cxgb3i [ 8081.097075] cxgb3 [ 8081.097075] mdio [ 8081.097076] libcxgbi [ 8081.097076] libcxgb [ 8081.097077] qla4xxx [ 8081.097077] iscsi_boot_sysfs [ 8081.097078] 8021q [ 8081.097078] garp [ 8081.097079] mrp [ 8081.097079] stp [ 8081.097079] llc [ 8081.097080] nvidia(POE) [ 8081.097080] ast [ 8081.097081] drm_kms_helper [ 8081.097082] crct10dif_pclmul [ 8081.097082] crct10dif_common [ 8081.097082] crc32_pclmul [ 8081.097083] crc32c_intel [ 8081.097083] syscopyarea [ 8081.097084] sysfillrect [ 8081.097084] sysimgblt [ 8081.097085] ghash_clmulni_intel [ 8081.097085] mlx5_core [ 8081.097086] fb_sys_fops [ 8081.097086] igb [ 8081.097087] ttm [ 8081.097087] aesni_intel [ 8081.097087] mlxfw [ 8081.097088] lrw [ 8081.097088] devlink [ 8081.097089] gf128mul [ 8081.097089] dca [ 8081.097090] glue_helper [ 8081.097090] ablk_helper [ 8081.097091] drm [ 8081.097091] dm_multipath [ 8081.097091] ptp [ 8081.097092] cryptd [ 8081.097092] i2c_algo_bit [ 8081.097093] pps_core [ 8081.097093] drm_panel_orientation_quirks [ 8081.097094] wmi [ 8081.097094] sunrpc [ 8081.097095] dm_mirror [ 8081.097095] dm_region_hash [ 8081.097096] dm_log [ 8081.097096] dm_mod [ 8081.097097] iscsi_tcp [ 8081.097097] libiscsi_tcp [ 8081.097098] libiscsi [ 8081.097098] scsi_transport_iscsi [ 8081.097099] fuse [ 8081.097099] [ 8081.097101] CPU: 55 PID: 16861 Comm: ptlrpcd_01_26 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.097102] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.097103] task: ffff8f484fbe8000 ti: ffff8f484fbe4000 task.ti: ffff8f484fbe4000 [ 8081.097103] RIP: 0010:[] [ 8081.097106] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.097106] RSP: 0018:ffff8f484fbe7b58 EFLAGS: 00000246 [ 8081.097107] RAX: 0000000000000000 RBX: ffff8f65fd49c380 RCX: 0000000001b90000 [ 8081.097108] RDX: ffff8f687f31b8c0 RSI: 0000000002010001 RDI: ffff8f686e2b6b40 [ 8081.097109] RBP: ffff8f484fbe7b58 R08: ffff8f687f0db8c0 R09: 0000000000000000 [ 8081.097110] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.097111] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000018ef9bd [ 8081.097112] FS: 0000000000000000(0000) GS:ffff8f687f0c0000(0000) knlGS:0000000000000000 [ 8081.097113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.097114] CR2: 00000000006e9360 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.097115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.097115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.097116] Call Trace: [ 8081.097119] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.097121] [] _raw_spin_lock+0x30/0x40 [ 8081.097129] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.097139] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.097171] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.097204] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.097235] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.097270] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.097273] [] ? wake_up_state+0x20/0x20 [ 8081.097306] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.097309] [] kthread+0xd1/0xe0 [ 8081.097311] [] ? insert_kthread_work+0x40/0x40 [ 8081.097313] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.097315] [] ? insert_kthread_work+0x40/0x40 [ 8081.097316] Code: [ 8081.097316] 0d [ 8081.097317] 48 [ 8081.097317] 98 [ 8081.097317] 83 [ 8081.097318] e2 [ 8081.097318] 30 [ 8081.097319] 48 [ 8081.097319] 81 [ 8081.097319] c2 [ 8081.097320] c0 [ 8081.097320] b8 [ 8081.097321] 01 [ 8081.097321] 00 [ 8081.097321] 48 [ 8081.097322] 03 [ 8081.097322] 14 [ 8081.097323] c5 [ 8081.097323] e0 [ 8081.097323] 17 [ 8081.097324] 15 [ 8081.097324] 91 [ 8081.097325] 4c [ 8081.097325] 89 [ 8081.097325] 02 [ 8081.097326] 41 [ 8081.097326] 8b [ 8081.097327] 40 [ 8081.097327] 08 [ 8081.097327] 85 [ 8081.097328] c0 [ 8081.097328] 75 [ 8081.097329] 0f [ 8081.097329] 0f [ 8081.097329] 1f [ 8081.097330] 44 [ 8081.097330] 00 [ 8081.097331] 00 [ 8081.097331] f3 [ 8081.097331] 90 [ 8081.097332] 41 [ 8081.097332] 8b [ 8081.097332] 40 [ 8081.097333] 08 [ 8081.097333] <85> [ 8081.097334] c0 [ 8081.097334] 74 [ 8081.097335] f6 [ 8081.097335] 4d [ 8081.097335] 8b [ 8081.097336] 08 [ 8081.097336] 4d [ 8081.097336] 85 [ 8081.097337] c9 [ 8081.097337] 74 [ 8081.097338] 04 [ 8081.097338] 41 [ 8081.097338] 0f [ 8081.097339] 18 [ 8081.097339] 09 [ 8081.097339] 8b [ 8081.097340] 17 [ 8081.097340] 0f [ 8081.097341] b7 [ 8081.097341] c2 [ 8081.097341] [ 8081.100034] NMI watchdog: BUG: soft lockup - CPU#56 stuck for 23s! [ptlrpcd_01_25:16860] [ 8081.100034] Modules linked in: [ 8081.100035] mgc(OE) [ 8081.100035] lustre(OE) [ 8081.100036] lmv(OE) [ 8081.100036] mdc(OE) [ 8081.100037] osc(OE) [ 8081.100037] lov(OE) [ 8081.100038] fid(OE) [ 8081.100038] fld(OE) [ 8081.100039] ptlrpc(OE) [ 8081.100039] obdclass(OE) [ 8081.100040] ko2iblnd(OE) [ 8081.100040] lnet(OE) [ 8081.100041] libcfs(OE) [ 8081.100041] gdrdrv(POE) [ 8081.100042] iTCO_wdt [ 8081.100042] iTCO_vendor_support [ 8081.100042] rpcrdma [ 8081.100043] nvidia_drm(POE) [ 8081.100043] ib_iser [ 8081.100044] joydev [ 8081.100044] sb_edac [ 8081.100045] intel_powerclamp [ 8081.100045] coretemp [ 8081.100046] intel_rapl [ 8081.100046] iosf_mbi [ 8081.100046] kvm_intel [ 8081.100047] kvm [ 8081.100047] irqbypass [ 8081.100048] nvidia_modeset(POE) [ 8081.100048] sg [ 8081.100049] pcspkr [ 8081.100049] i2c_i801 [ 8081.100050] lpc_ich [ 8081.100050] nf_log_ipv4 [ 8081.100050] nf_log_common [ 8081.100051] xt_LOG [ 8081.100051] nf_conntrack_ipv4 [ 8081.100052] nf_defrag_ipv4 [ 8081.100052] xt_multiport [ 8081.100053] xt_owner [ 8081.100053] xt_conntrack [ 8081.100053] nf_conntrack [ 8081.100054] libcrc32c [ 8081.100054] iptable_filter [ 8081.100055] ipmi_si [ 8081.100055] ipmi_devintf [ 8081.100056] ipmi_msghandler [ 8081.100056] acpi_power_meter [ 8081.100057] ib_ipoib [ 8081.100057] rdma_ucm [ 8081.100058] ib_umad [ 8081.100058] iw_cxgb4 [ 8081.100059] rdma_cm [ 8081.100059] iw_cm [ 8081.100059] ib_cm [ 8081.100060] iw_cxgb3 [ 8081.100060] sch_fq_codel [ 8081.100061] binfmt_misc [ 8081.100062] msr_safe(OE) [ 8081.100062] ip_tables [ 8081.100062] nfsv3 [ 8081.100063] nfs_acl [ 8081.100063] rpcsec_gss_krb5 [ 8081.100064] auth_rpcgss [ 8081.100064] nfsv4 [ 8081.100065] dns_resolver [ 8081.100065] nfs [ 8081.100066] lockd [ 8081.100066] grace [ 8081.100066] fscache [ 8081.100067] overlay(T) [ 8081.100067] ext4 [ 8081.100068] mbcache [ 8081.100068] jbd2 [ 8081.100069] sd_mod [ 8081.100069] crc_t10dif [ 8081.100070] crct10dif_generic [ 8081.100070] nvidia_uvm(OE) [ 8081.100071] mlx5_ib [ 8081.100071] ib_uverbs [ 8081.100072] be2iscsi [ 8081.100072] ib_core [ 8081.100073] bnx2i [ 8081.100073] cnic [ 8081.100074] uio [ 8081.100074] cxgb4i [ 8081.100074] cxgb4 [ 8081.100075] cxgb3i [ 8081.100075] cxgb3 [ 8081.100076] mdio [ 8081.100076] libcxgbi [ 8081.100077] libcxgb [ 8081.100077] qla4xxx [ 8081.100078] iscsi_boot_sysfs [ 8081.100078] 8021q [ 8081.100078] garp [ 8081.100079] mrp [ 8081.100079] stp [ 8081.100080] llc [ 8081.100080] nvidia(POE) [ 8081.100081] ast [ 8081.100081] drm_kms_helper [ 8081.100082] crct10dif_pclmul [ 8081.100082] crct10dif_common [ 8081.100083] crc32_pclmul [ 8081.100083] crc32c_intel [ 8081.100084] syscopyarea [ 8081.100084] sysfillrect [ 8081.100085] sysimgblt [ 8081.100085] ghash_clmulni_intel [ 8081.100085] mlx5_core [ 8081.100086] fb_sys_fops [ 8081.100086] igb [ 8081.100087] ttm [ 8081.100087] aesni_intel [ 8081.100088] mlxfw [ 8081.100088] lrw [ 8081.100089] devlink [ 8081.100089] gf128mul [ 8081.100089] dca [ 8081.100090] glue_helper [ 8081.100090] ablk_helper [ 8081.100091] drm [ 8081.100091] dm_multipath [ 8081.100092] ptp [ 8081.100092] cryptd [ 8081.100093] i2c_algo_bit [ 8081.100093] pps_core [ 8081.100094] drm_panel_orientation_quirks [ 8081.100094] wmi [ 8081.100094] sunrpc [ 8081.100095] dm_mirror [ 8081.100095] dm_region_hash [ 8081.100096] dm_log [ 8081.100096] dm_mod [ 8081.100097] iscsi_tcp [ 8081.100097] libiscsi_tcp [ 8081.100098] libiscsi [ 8081.100098] scsi_transport_iscsi [ 8081.100099] fuse [ 8081.100099] [ 8081.100101] CPU: 56 PID: 16860 Comm: ptlrpcd_01_25 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.100102] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.100103] task: ffff8f484fbc6300 ti: ffff8f484fbe0000 task.ti: ffff8f484fbe0000 [ 8081.100103] RIP: 0010:[] [ 8081.100105] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.100106] RSP: 0018:ffff8f484fbe3b58 EFLAGS: 00000246 [ 8081.100107] RAX: 0000000000000000 RBX: ffff8f65fa694c80 RCX: 0000000001c10000 [ 8081.100108] RDX: ffff8f487fa1b8c0 RSI: 0000000001510001 RDI: ffff8f686e2b6b40 [ 8081.100109] RBP: ffff8f484fbe3b58 R08: ffff8f687f11b8c0 R09: 0000000000000000 [ 8081.100109] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.100110] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000013a84c86 [ 8081.100112] FS: 0000000000000000(0000) GS:ffff8f687f100000(0000) knlGS:0000000000000000 [ 8081.100113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.100114] CR2: 00002aaaaafbbd70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.100114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.100115] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.100116] Call Trace: [ 8081.100118] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.100120] [] _raw_spin_lock+0x30/0x40 [ 8081.100128] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.100138] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.100169] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.100201] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.100231] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.100265] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.100268] [] ? wake_up_state+0x20/0x20 [ 8081.100300] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.100303] [] kthread+0xd1/0xe0 [ 8081.100305] [] ? insert_kthread_work+0x40/0x40 [ 8081.100307] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.100309] [] ? insert_kthread_work+0x40/0x40 [ 8081.100310] Code: [ 8081.100310] 0d [ 8081.100311] 48 [ 8081.100311] 98 [ 8081.100311] 83 [ 8081.100312] e2 [ 8081.100312] 30 [ 8081.100313] 48 [ 8081.100313] 81 [ 8081.100313] c2 [ 8081.100314] c0 [ 8081.100314] b8 [ 8081.100314] 01 [ 8081.100315] 00 [ 8081.100315] 48 [ 8081.100316] 03 [ 8081.100316] 14 [ 8081.100317] c5 [ 8081.100317] e0 [ 8081.100318] 17 [ 8081.100318] 15 [ 8081.100318] 91 [ 8081.100319] 4c [ 8081.100319] 89 [ 8081.100319] 02 [ 8081.100320] 41 [ 8081.100320] 8b [ 8081.100321] 40 [ 8081.100321] 08 [ 8081.100322] 85 [ 8081.100322] c0 [ 8081.100322] 75 [ 8081.100323] 0f [ 8081.100323] 0f [ 8081.100324] 1f [ 8081.100324] 44 [ 8081.100324] 00 [ 8081.100325] 00 [ 8081.100325] f3 [ 8081.100326] 90 [ 8081.100326] 41 [ 8081.100326] 8b [ 8081.100327] 40 [ 8081.100327] 08 [ 8081.100328] <85> [ 8081.100328] c0 [ 8081.100328] 74 [ 8081.100329] f6 [ 8081.100329] 4d [ 8081.100330] 8b [ 8081.100330] 08 [ 8081.100331] 4d [ 8081.100331] 85 [ 8081.100332] c9 [ 8081.100332] 74 [ 8081.100332] 04 [ 8081.100333] 41 [ 8081.100333] 0f [ 8081.100333] 18 [ 8081.100334] 09 [ 8081.100334] 8b [ 8081.100335] 17 [ 8081.100335] 0f [ 8081.100335] b7 [ 8081.100336] c2 [ 8081.100336] [ 8081.103033] NMI watchdog: BUG: soft lockup - CPU#57 stuck for 23s! [ptlrpcd_01_10:16845] [ 8081.103033] Modules linked in: [ 8081.103034] mgc(OE) [ 8081.103035] lustre(OE) [ 8081.103035] lmv(OE) [ 8081.103036] mdc(OE) [ 8081.103036] osc(OE) [ 8081.103037] lov(OE) [ 8081.103037] fid(OE) [ 8081.103038] fld(OE) [ 8081.103038] ptlrpc(OE) [ 8081.103039] obdclass(OE) [ 8081.103039] ko2iblnd(OE) [ 8081.103040] lnet(OE) [ 8081.103040] libcfs(OE) [ 8081.103041] gdrdrv(POE) [ 8081.103041] iTCO_wdt [ 8081.103042] iTCO_vendor_support [ 8081.103042] rpcrdma [ 8081.103043] nvidia_drm(POE) [ 8081.103043] ib_iser [ 8081.103044] joydev [ 8081.103044] sb_edac [ 8081.103045] intel_powerclamp [ 8081.103045] coretemp [ 8081.103046] intel_rapl [ 8081.103046] iosf_mbi [ 8081.103046] kvm_intel [ 8081.103047] kvm [ 8081.103047] irqbypass [ 8081.103048] nvidia_modeset(POE) [ 8081.103048] sg [ 8081.103049] pcspkr [ 8081.103049] i2c_i801 [ 8081.103050] lpc_ich [ 8081.103050] nf_log_ipv4 [ 8081.103051] nf_log_common [ 8081.103051] xt_LOG [ 8081.103052] nf_conntrack_ipv4 [ 8081.103052] nf_defrag_ipv4 [ 8081.103053] xt_multiport [ 8081.103053] xt_owner [ 8081.103054] xt_conntrack [ 8081.103054] nf_conntrack [ 8081.103055] libcrc32c [ 8081.103055] iptable_filter [ 8081.103056] ipmi_si [ 8081.103056] ipmi_devintf [ 8081.103057] ipmi_msghandler [ 8081.103057] acpi_power_meter [ 8081.103058] ib_ipoib [ 8081.103058] rdma_ucm [ 8081.103059] ib_umad [ 8081.103059] iw_cxgb4 [ 8081.103060] rdma_cm [ 8081.103060] iw_cm [ 8081.103060] ib_cm [ 8081.103061] iw_cxgb3 [ 8081.103061] sch_fq_codel [ 8081.103062] binfmt_misc [ 8081.103062] msr_safe(OE) [ 8081.103063] ip_tables [ 8081.103063] nfsv3 [ 8081.103064] nfs_acl [ 8081.103064] rpcsec_gss_krb5 [ 8081.103065] auth_rpcgss [ 8081.103065] nfsv4 [ 8081.103066] dns_resolver [ 8081.103066] nfs [ 8081.103066] lockd [ 8081.103067] grace [ 8081.103067] fscache [ 8081.103068] overlay(T) [ 8081.103068] ext4 [ 8081.103069] mbcache [ 8081.103069] jbd2 [ 8081.103070] sd_mod [ 8081.103070] crc_t10dif [ 8081.103071] crct10dif_generic [ 8081.103071] nvidia_uvm(OE) [ 8081.103072] mlx5_ib [ 8081.103072] ib_uverbs [ 8081.103073] be2iscsi [ 8081.103073] ib_core [ 8081.103074] bnx2i [ 8081.103074] cnic [ 8081.103074] uio [ 8081.103075] cxgb4i [ 8081.103075] cxgb4 [ 8081.103076] cxgb3i [ 8081.103076] cxgb3 [ 8081.103077] mdio [ 8081.103077] libcxgbi [ 8081.103078] libcxgb [ 8081.103078] qla4xxx [ 8081.103079] iscsi_boot_sysfs [ 8081.103079] 8021q [ 8081.103079] garp [ 8081.103080] mrp [ 8081.103080] stp [ 8081.103081] llc [ 8081.103081] nvidia(POE) [ 8081.103082] ast [ 8081.103082] drm_kms_helper [ 8081.103083] crct10dif_pclmul [ 8081.103083] crct10dif_common [ 8081.103084] crc32_pclmul [ 8081.103084] crc32c_intel [ 8081.103084] syscopyarea [ 8081.103085] sysfillrect [ 8081.103085] sysimgblt [ 8081.103086] ghash_clmulni_intel [ 8081.103086] mlx5_core [ 8081.103087] fb_sys_fops [ 8081.103087] igb [ 8081.103088] ttm [ 8081.103088] aesni_intel [ 8081.103088] mlxfw [ 8081.103089] lrw [ 8081.103089] devlink [ 8081.103090] gf128mul [ 8081.103090] dca [ 8081.103091] glue_helper [ 8081.103091] ablk_helper [ 8081.103092] drm [ 8081.103092] dm_multipath [ 8081.103093] ptp [ 8081.103093] cryptd [ 8081.103093] i2c_algo_bit [ 8081.103094] pps_core [ 8081.103095] drm_panel_orientation_quirks [ 8081.103095] wmi [ 8081.103095] sunrpc [ 8081.103096] dm_mirror [ 8081.103096] dm_region_hash [ 8081.103097] dm_log [ 8081.103097] dm_mod [ 8081.103098] iscsi_tcp [ 8081.103098] libiscsi_tcp [ 8081.103099] libiscsi [ 8081.103099] scsi_transport_iscsi [ 8081.103100] fuse [ 8081.103100] [ 8081.103102] CPU: 57 PID: 16845 Comm: ptlrpcd_01_10 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.103103] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.103104] task: ffff8f484fb2d280 ti: ffff8f484fb8c000 task.ti: ffff8f484fb8c000 [ 8081.103105] RIP: 0010:[] [ 8081.103108] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.103109] RSP: 0018:ffff8f484fb8fb58 EFLAGS: 00000246 [ 8081.103110] RAX: 0000000000000000 RBX: ffff8f65fe838900 RCX: 0000000001c90000 [ 8081.103110] RDX: ffff8f687f01b8c0 RSI: 0000000001110001 RDI: ffff8f686e2b6b40 [ 8081.103111] RBP: ffff8f484fb8fb58 R08: ffff8f687f15b8c0 R09: 0000000000000000 [ 8081.103112] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.103113] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000048d6b28f [ 8081.103114] FS: 0000000000000000(0000) GS:ffff8f687f140000(0000) knlGS:0000000000000000 [ 8081.103115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.103116] CR2: 00000000006d2a70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.103117] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.103118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.103118] Call Trace: [ 8081.103121] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.103123] [] _raw_spin_lock+0x30/0x40 [ 8081.103132] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.103142] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.103174] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.103207] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.103237] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.103272] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.103275] [] ? wake_up_state+0x20/0x20 [ 8081.103309] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.103311] [] kthread+0xd1/0xe0 [ 8081.103313] [] ? insert_kthread_work+0x40/0x40 [ 8081.103315] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.103317] [] ? insert_kthread_work+0x40/0x40 [ 8081.103318] Code: [ 8081.103319] 0d [ 8081.103319] 48 [ 8081.103319] 98 [ 8081.103320] 83 [ 8081.103320] e2 [ 8081.103321] 30 [ 8081.103321] 48 [ 8081.103321] 81 [ 8081.103322] c2 [ 8081.103322] c0 [ 8081.103323] b8 [ 8081.103323] 01 [ 8081.103323] 00 [ 8081.103324] 48 [ 8081.103324] 03 [ 8081.103325] 14 [ 8081.103325] c5 [ 8081.103325] e0 [ 8081.103326] 17 [ 8081.103326] 15 [ 8081.103326] 91 [ 8081.103327] 4c [ 8081.103327] 89 [ 8081.103328] 02 [ 8081.103328] 41 [ 8081.103328] 8b [ 8081.103329] 40 [ 8081.103329] 08 [ 8081.103330] 85 [ 8081.103330] c0 [ 8081.103330] 75 [ 8081.103331] 0f [ 8081.103331] 0f [ 8081.103332] 1f [ 8081.103332] 44 [ 8081.103332] 00 [ 8081.103333] 00 [ 8081.103333] f3 [ 8081.103334] 90 [ 8081.103334] 41 [ 8081.103335] 8b [ 8081.103335] 40 [ 8081.103335] 08 [ 8081.103336] <85> [ 8081.103336] c0 [ 8081.103337] 74 [ 8081.103337] f6 [ 8081.103338] 4d [ 8081.103338] 8b [ 8081.103338] 08 [ 8081.103339] 4d [ 8081.103339] 85 [ 8081.103340] c9 [ 8081.103340] 74 [ 8081.103340] 04 [ 8081.103341] 41 [ 8081.103341] 0f [ 8081.103342] 18 [ 8081.103342] 09 [ 8081.103343] 8b [ 8081.103343] 17 [ 8081.103343] 0f [ 8081.103344] b7 [ 8081.103344] c2 [ 8081.103345] [ 8081.124032] NMI watchdog: BUG: soft lockup - CPU#64 stuck for 23s! [ptlrpcd_01_09:16844] [ 8081.124033] Modules linked in: [ 8081.124033] mgc(OE) [ 8081.124034] lustre(OE) [ 8081.124035] lmv(OE) [ 8081.124035] mdc(OE) [ 8081.124035] osc(OE) [ 8081.124036] lov(OE) [ 8081.124036] fid(OE) [ 8081.124037] fld(OE) [ 8081.124037] ptlrpc(OE) [ 8081.124038] obdclass(OE) [ 8081.124038] ko2iblnd(OE) [ 8081.124039] lnet(OE) [ 8081.124039] libcfs(OE) [ 8081.124040] gdrdrv(POE) [ 8081.124040] iTCO_wdt [ 8081.124041] iTCO_vendor_support [ 8081.124041] rpcrdma [ 8081.124042] nvidia_drm(POE) [ 8081.124042] ib_iser [ 8081.124043] joydev [ 8081.124043] sb_edac [ 8081.124044] intel_powerclamp [ 8081.124044] coretemp [ 8081.124045] intel_rapl [ 8081.124045] iosf_mbi [ 8081.124045] kvm_intel [ 8081.124046] kvm [ 8081.124046] irqbypass [ 8081.124047] nvidia_modeset(POE) [ 8081.124047] sg [ 8081.124048] pcspkr [ 8081.124048] i2c_i801 [ 8081.124048] lpc_ich [ 8081.124049] nf_log_ipv4 [ 8081.124049] nf_log_common [ 8081.124050] xt_LOG [ 8081.124050] nf_conntrack_ipv4 [ 8081.124051] nf_defrag_ipv4 [ 8081.124051] xt_multiport [ 8081.124052] xt_owner [ 8081.124052] xt_conntrack [ 8081.124052] nf_conntrack [ 8081.124053] libcrc32c [ 8081.124053] iptable_filter [ 8081.124054] ipmi_si [ 8081.124054] ipmi_devintf [ 8081.124055] ipmi_msghandler [ 8081.124055] acpi_power_meter [ 8081.124056] ib_ipoib [ 8081.124056] rdma_ucm [ 8081.124057] ib_umad [ 8081.124057] iw_cxgb4 [ 8081.124058] rdma_cm [ 8081.124058] iw_cm [ 8081.124058] ib_cm [ 8081.124059] iw_cxgb3 [ 8081.124059] sch_fq_codel [ 8081.124060] binfmt_misc [ 8081.124060] msr_safe(OE) [ 8081.124061] ip_tables [ 8081.124061] nfsv3 [ 8081.124062] nfs_acl [ 8081.124062] rpcsec_gss_krb5 [ 8081.124063] auth_rpcgss [ 8081.124063] nfsv4 [ 8081.124064] dns_resolver [ 8081.124064] nfs [ 8081.124065] lockd [ 8081.124065] grace [ 8081.124065] fscache [ 8081.124066] overlay(T) [ 8081.124067] ext4 [ 8081.124067] mbcache [ 8081.124068] jbd2 [ 8081.124068] sd_mod [ 8081.124068] crc_t10dif [ 8081.124069] crct10dif_generic [ 8081.124070] nvidia_uvm(OE) [ 8081.124070] mlx5_ib [ 8081.124070] ib_uverbs [ 8081.124071] be2iscsi [ 8081.124071] ib_core [ 8081.124072] bnx2i [ 8081.124072] cnic [ 8081.124073] uio [ 8081.124073] cxgb4i [ 8081.124074] cxgb4 [ 8081.124074] cxgb3i [ 8081.124074] cxgb3 [ 8081.124075] mdio [ 8081.124075] libcxgbi [ 8081.124076] libcxgb [ 8081.124076] qla4xxx [ 8081.124077] iscsi_boot_sysfs [ 8081.124077] 8021q [ 8081.124078] garp [ 8081.124078] mrp [ 8081.124078] stp [ 8081.124079] llc [ 8081.124079] nvidia(POE) [ 8081.124080] ast [ 8081.124080] drm_kms_helper [ 8081.124081] crct10dif_pclmul [ 8081.124081] crct10dif_common [ 8081.124082] crc32_pclmul [ 8081.124082] crc32c_intel [ 8081.124083] syscopyarea [ 8081.124083] sysfillrect [ 8081.124084] sysimgblt [ 8081.124084] ghash_clmulni_intel [ 8081.124085] mlx5_core [ 8081.124085] fb_sys_fops [ 8081.124086] igb [ 8081.124086] ttm [ 8081.124087] aesni_intel [ 8081.124087] mlxfw [ 8081.124088] lrw [ 8081.124088] devlink [ 8081.124088] gf128mul [ 8081.124089] dca [ 8081.124089] glue_helper [ 8081.124090] ablk_helper [ 8081.124090] drm [ 8081.124091] dm_multipath [ 8081.124091] ptp [ 8081.124092] cryptd [ 8081.124092] i2c_algo_bit [ 8081.124092] pps_core [ 8081.124093] drm_panel_orientation_quirks [ 8081.124094] wmi [ 8081.124094] sunrpc [ 8081.124094] dm_mirror [ 8081.124095] dm_region_hash [ 8081.124095] dm_log [ 8081.124096] dm_mod [ 8081.124096] iscsi_tcp [ 8081.124097] libiscsi_tcp [ 8081.124097] libiscsi [ 8081.124098] scsi_transport_iscsi [ 8081.124098] fuse [ 8081.124099] [ 8081.124101] CPU: 64 PID: 16844 Comm: ptlrpcd_01_09 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.124102] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.124103] task: ffff8f484fb2c200 ti: ffff8f484fb88000 task.ti: ffff8f484fb88000 [ 8081.124104] RIP: 0010:[] [ 8081.124106] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8081.124107] RSP: 0018:ffff8f484fb8bb58 EFLAGS: 00000246 [ 8081.124108] RAX: 0000000000000000 RBX: ffff8f66f6d35580 RCX: 0000000002010000 [ 8081.124109] RDX: ffff8f687ec5b8c0 RSI: 0000000000990001 RDI: ffff8f686e2b6b40 [ 8081.124110] RBP: ffff8f484fb8bb58 R08: ffff8f687f31b8c0 R09: 0000000000000000 [ 8081.124111] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.124111] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000ac8b218f [ 8081.124113] FS: 0000000000000000(0000) GS:ffff8f687f300000(0000) knlGS:0000000000000000 [ 8081.124114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.124115] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.124116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.124116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.124117] Call Trace: [ 8081.124120] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.124122] [] _raw_spin_lock+0x30/0x40 [ 8081.124130] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.124140] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.124172] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.124204] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.124234] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.124269] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.124271] [] ? wake_up_state+0x20/0x20 [ 8081.124304] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.124307] [] kthread+0xd1/0xe0 [ 8081.124309] [] ? insert_kthread_work+0x40/0x40 [ 8081.124311] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.124313] [] ? insert_kthread_work+0x40/0x40 [ 8081.124314] Code: [ 8081.124314] 13 [ 8081.124315] 48 [ 8081.124315] c1 [ 8081.124315] ea [ 8081.124316] 0d [ 8081.124316] 48 [ 8081.124317] 98 [ 8081.124317] 83 [ 8081.124317] e2 [ 8081.124318] 30 [ 8081.124318] 48 [ 8081.124319] 81 [ 8081.124319] c2 [ 8081.124320] c0 [ 8081.124320] b8 [ 8081.124320] 01 [ 8081.124321] 00 [ 8081.124321] 48 [ 8081.124321] 03 [ 8081.124322] 14 [ 8081.124322] c5 [ 8081.124323] e0 [ 8081.124323] 17 [ 8081.124324] 15 [ 8081.124324] 91 [ 8081.124324] 4c [ 8081.124325] 89 [ 8081.124325] 02 [ 8081.124326] 41 [ 8081.124326] 8b [ 8081.124326] 40 [ 8081.124327] 08 [ 8081.124327] 85 [ 8081.124328] c0 [ 8081.124328] 75 [ 8081.124328] 0f [ 8081.124329] 0f [ 8081.124329] 1f [ 8081.124330] 44 [ 8081.124330] 00 [ 8081.124330] 00 [ 8081.124331] f3 [ 8081.124331] 90 [ 8081.124332] <41> [ 8081.124332] 8b [ 8081.124333] 40 [ 8081.124333] 08 [ 8081.124333] 85 [ 8081.124334] c0 [ 8081.124334] 74 [ 8081.124334] f6 [ 8081.124335] 4d [ 8081.124335] 8b [ 8081.124336] 08 [ 8081.124336] 4d [ 8081.124336] 85 [ 8081.124337] c9 [ 8081.124337] 74 [ 8081.124338] 04 [ 8081.124338] 41 [ 8081.124338] 0f [ 8081.124339] 18 [ 8081.124339] 09 [ 8081.124339] 8b [ 8081.124340] [ 8081.133033] NMI watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [ptlrpcd_01_33:16868] [ 8081.133034] Modules linked in: [ 8081.133034] mgc(OE) [ 8081.133035] lustre(OE) [ 8081.133035] lmv(OE) [ 8081.133036] mdc(OE) [ 8081.133036] osc(OE) [ 8081.133037] lov(OE) [ 8081.133037] fid(OE) [ 8081.133038] fld(OE) [ 8081.133038] ptlrpc(OE) [ 8081.133039] obdclass(OE) [ 8081.133039] ko2iblnd(OE) [ 8081.133040] lnet(OE) [ 8081.133040] libcfs(OE) [ 8081.133041] gdrdrv(POE) [ 8081.133041] iTCO_wdt [ 8081.133042] iTCO_vendor_support [ 8081.133042] rpcrdma [ 8081.133043] nvidia_drm(POE) [ 8081.133043] ib_iser [ 8081.133044] joydev [ 8081.133044] sb_edac [ 8081.133045] intel_powerclamp [ 8081.133045] coretemp [ 8081.133045] intel_rapl [ 8081.133046] iosf_mbi [ 8081.133046] kvm_intel [ 8081.133047] kvm [ 8081.133047] irqbypass [ 8081.133048] nvidia_modeset(POE) [ 8081.133048] sg [ 8081.133049] pcspkr [ 8081.133049] i2c_i801 [ 8081.133049] lpc_ich [ 8081.133050] nf_log_ipv4 [ 8081.133050] nf_log_common [ 8081.133051] xt_LOG [ 8081.133051] nf_conntrack_ipv4 [ 8081.133052] nf_defrag_ipv4 [ 8081.133052] xt_multiport [ 8081.133052] xt_owner [ 8081.133053] xt_conntrack [ 8081.133053] nf_conntrack [ 8081.133054] libcrc32c [ 8081.133054] iptable_filter [ 8081.133055] ipmi_si [ 8081.133055] ipmi_devintf [ 8081.133056] ipmi_msghandler [ 8081.133056] acpi_power_meter [ 8081.133056] ib_ipoib [ 8081.133057] rdma_ucm [ 8081.133057] ib_umad [ 8081.133058] iw_cxgb4 [ 8081.133058] rdma_cm [ 8081.133059] iw_cm [ 8081.133059] ib_cm [ 8081.133060] iw_cxgb3 [ 8081.133060] sch_fq_codel [ 8081.133061] binfmt_misc [ 8081.133061] msr_safe(OE) [ 8081.133062] ip_tables [ 8081.133062] nfsv3 [ 8081.133063] nfs_acl [ 8081.133063] rpcsec_gss_krb5 [ 8081.133064] auth_rpcgss [ 8081.133064] nfsv4 [ 8081.133064] dns_resolver [ 8081.133065] nfs [ 8081.133065] lockd [ 8081.133066] grace [ 8081.133066] fscache [ 8081.133067] overlay(T) [ 8081.133068] ext4 [ 8081.133068] mbcache [ 8081.133069] jbd2 [ 8081.133069] sd_mod [ 8081.133070] crc_t10dif [ 8081.133070] crct10dif_generic [ 8081.133071] nvidia_uvm(OE) [ 8081.133071] mlx5_ib [ 8081.133071] ib_uverbs [ 8081.133072] be2iscsi [ 8081.133072] ib_core [ 8081.133073] bnx2i [ 8081.133073] cnic [ 8081.133074] uio [ 8081.133074] cxgb4i [ 8081.133075] cxgb4 [ 8081.133075] cxgb3i [ 8081.133076] cxgb3 [ 8081.133076] mdio [ 8081.133076] libcxgbi [ 8081.133077] libcxgb [ 8081.133077] qla4xxx [ 8081.133078] iscsi_boot_sysfs [ 8081.133078] 8021q [ 8081.133079] garp [ 8081.133079] mrp [ 8081.133080] stp [ 8081.133080] llc [ 8081.133081] nvidia(POE) [ 8081.133081] ast [ 8081.133082] drm_kms_helper [ 8081.133082] crct10dif_pclmul [ 8081.133083] crct10dif_common [ 8081.133083] crc32_pclmul [ 8081.133084] crc32c_intel [ 8081.133084] syscopyarea [ 8081.133084] sysfillrect [ 8081.133085] sysimgblt [ 8081.133085] ghash_clmulni_intel [ 8081.133086] mlx5_core [ 8081.133086] fb_sys_fops [ 8081.133087] igb [ 8081.133087] ttm [ 8081.133088] aesni_intel [ 8081.133088] mlxfw [ 8081.133089] lrw [ 8081.133089] devlink [ 8081.133089] gf128mul [ 8081.133090] dca [ 8081.133090] glue_helper [ 8081.133091] ablk_helper [ 8081.133091] drm [ 8081.133092] dm_multipath [ 8081.133092] ptp [ 8081.133093] cryptd [ 8081.133093] i2c_algo_bit [ 8081.133094] pps_core [ 8081.133094] drm_panel_orientation_quirks [ 8081.133095] wmi [ 8081.133095] sunrpc [ 8081.133096] dm_mirror [ 8081.133096] dm_region_hash [ 8081.133096] dm_log [ 8081.133097] dm_mod [ 8081.133097] iscsi_tcp [ 8081.133098] libiscsi_tcp [ 8081.133098] libiscsi [ 8081.133099] scsi_transport_iscsi [ 8081.133099] fuse [ 8081.133100] [ 8081.133102] CPU: 67 PID: 16868 Comm: ptlrpcd_01_33 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8081.133103] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8081.133104] task: ffff8f484d290000 ti: ffff8f484d298000 task.ti: ffff8f484d298000 [ 8081.133105] RIP: 0010:[] [ 8081.133108] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8081.133109] RSP: 0018:ffff8f484d29bb58 EFLAGS: 00000246 [ 8081.133110] RAX: 0000000000000000 RBX: ffff8f6730f5f500 RCX: 0000000002190000 [ 8081.133110] RDX: ffff8f687f15b8c0 RSI: 0000000001c90001 RDI: ffff8f686e2b6b40 [ 8081.133111] RBP: ffff8f484d29bb58 R08: ffff8f687f3db8c0 R09: 0000000000000000 [ 8081.133112] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8081.133113] R13: 0000000000000003 R14: 0000000000000013 R15: 000000004ff7fd94 [ 8081.133114] FS: 0000000000000000(0000) GS:ffff8f687f3c0000(0000) knlGS:0000000000000000 [ 8081.133115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8081.133116] CR2: 0000000000630fb8 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8081.133117] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8081.133118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8081.133118] Call Trace: [ 8081.133122] [] queued_spin_lock_slowpath+0xb/0xf [ 8081.133124] [] _raw_spin_lock+0x30/0x40 [ 8081.133132] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8081.133142] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8081.133174] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8081.133209] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8081.133240] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8081.133275] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8081.133278] [] ? wake_up_state+0x20/0x20 [ 8081.133311] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8081.133314] [] kthread+0xd1/0xe0 [ 8081.133316] [] ? insert_kthread_work+0x40/0x40 [ 8081.133318] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8081.133320] [] ? insert_kthread_work+0x40/0x40 [ 8081.133321] Code: [ 8081.133322] 0d [ 8081.133322] 48 [ 8081.133323] 98 [ 8081.133323] 83 [ 8081.133323] e2 [ 8081.133324] 30 [ 8081.133324] 48 [ 8081.133325] 81 [ 8081.133325] c2 [ 8081.133325] c0 [ 8081.133326] b8 [ 8081.133326] 01 [ 8081.133327] 00 [ 8081.133327] 48 [ 8081.133328] 03 [ 8081.133328] 14 [ 8081.133328] c5 [ 8081.133329] e0 [ 8081.133329] 17 [ 8081.133330] 15 [ 8081.133330] 91 [ 8081.133330] 4c [ 8081.133331] 89 [ 8081.133331] 02 [ 8081.133332] 41 [ 8081.133332] 8b [ 8081.133332] 40 [ 8081.133333] 08 [ 8081.133333] 85 [ 8081.133334] c0 [ 8081.133334] 75 [ 8081.133334] 0f [ 8081.133335] 0f [ 8081.133335] 1f [ 8081.133336] 44 [ 8081.133336] 00 [ 8081.133336] 00 [ 8081.133337] f3 [ 8081.133337] 90 [ 8081.133338] 41 [ 8081.133338] 8b [ 8081.133338] 40 [ 8081.133339] 08 [ 8081.133339] <85> [ 8081.133340] c0 [ 8081.133340] 74 [ 8081.133341] f6 [ 8081.133341] 4d [ 8081.133341] 8b [ 8081.133342] 08 [ 8081.133342] 4d [ 8081.133342] 85 [ 8081.133343] c9 [ 8081.133343] 74 [ 8081.133344] 04 [ 8081.133344] 41 [ 8081.133344] 0f [ 8081.133345] 18 [ 8081.133345] 09 [ 8081.133345] 8b [ 8081.133346] 17 [ 8081.133346] 0f [ 8081.133347] b7 [ 8081.133347] c2 [ 8081.133347] [ 8084.799942] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ptlrpcd_00_24:16822] [ 8084.799943] Modules linked in: [ 8084.799943] mgc(OE) [ 8084.799944] lustre(OE) [ 8084.799945] lmv(OE) [ 8084.799945] mdc(OE) [ 8084.799946] osc(OE) [ 8084.799946] lov(OE) [ 8084.799947] fid(OE) [ 8084.799947] fld(OE) [ 8084.799947] ptlrpc(OE) [ 8084.799948] obdclass(OE) [ 8084.799948] ko2iblnd(OE) [ 8084.799949] lnet(OE) [ 8084.799949] libcfs(OE) [ 8084.799950] gdrdrv(POE) [ 8084.799950] iTCO_wdt [ 8084.799951] iTCO_vendor_support [ 8084.799951] rpcrdma [ 8084.799952] nvidia_drm(POE) [ 8084.799952] ib_iser [ 8084.799953] joydev [ 8084.799953] sb_edac [ 8084.799954] intel_powerclamp [ 8084.799954] coretemp [ 8084.799955] intel_rapl [ 8084.799955] iosf_mbi [ 8084.799955] kvm_intel [ 8084.799956] kvm [ 8084.799956] irqbypass [ 8084.799957] nvidia_modeset(POE) [ 8084.799957] sg [ 8084.799958] pcspkr [ 8084.799958] i2c_i801 [ 8084.799959] lpc_ich [ 8084.799959] nf_log_ipv4 [ 8084.799959] nf_log_common [ 8084.799960] xt_LOG [ 8084.799960] nf_conntrack_ipv4 [ 8084.799961] nf_defrag_ipv4 [ 8084.799961] xt_multiport [ 8084.799962] xt_owner [ 8084.799962] xt_conntrack [ 8084.799963] nf_conntrack [ 8084.799963] libcrc32c [ 8084.799964] iptable_filter [ 8084.799964] ipmi_si [ 8084.799964] ipmi_devintf [ 8084.799965] ipmi_msghandler [ 8084.799965] acpi_power_meter [ 8084.799966] ib_ipoib [ 8084.799966] rdma_ucm [ 8084.799967] ib_umad [ 8084.799967] iw_cxgb4 [ 8084.799968] rdma_cm [ 8084.799968] iw_cm [ 8084.799968] ib_cm [ 8084.799969] iw_cxgb3 [ 8084.799969] sch_fq_codel [ 8084.799970] binfmt_misc [ 8084.799970] msr_safe(OE) [ 8084.799971] ip_tables [ 8084.799971] nfsv3 [ 8084.799972] nfs_acl [ 8084.799972] rpcsec_gss_krb5 [ 8084.799973] auth_rpcgss [ 8084.799973] nfsv4 [ 8084.799974] dns_resolver [ 8084.799974] nfs [ 8084.799974] lockd [ 8084.799975] grace [ 8084.799975] fscache [ 8084.799976] overlay(T) [ 8084.799976] ext4 [ 8084.799977] mbcache [ 8084.799977] jbd2 [ 8084.799978] sd_mod [ 8084.799978] crc_t10dif [ 8084.799979] crct10dif_generic [ 8084.799979] nvidia_uvm(OE) [ 8084.799980] mlx5_ib [ 8084.799980] ib_uverbs [ 8084.799981] be2iscsi [ 8084.799981] ib_core [ 8084.799982] bnx2i [ 8084.799982] cnic [ 8084.799982] uio [ 8084.799983] cxgb4i [ 8084.799983] cxgb4 [ 8084.799984] cxgb3i [ 8084.799984] cxgb3 [ 8084.799985] mdio [ 8084.799985] libcxgbi [ 8084.799986] libcxgb [ 8084.799986] qla4xxx [ 8084.799986] iscsi_boot_sysfs [ 8084.799987] 8021q [ 8084.799987] garp [ 8084.799988] mrp [ 8084.799988] stp [ 8084.799988] llc [ 8084.799989] nvidia(POE) [ 8084.799989] ast [ 8084.799990] drm_kms_helper [ 8084.799990] crct10dif_pclmul [ 8084.799991] crct10dif_common [ 8084.799991] crc32_pclmul [ 8084.799992] crc32c_intel [ 8084.799992] syscopyarea [ 8084.799993] sysfillrect [ 8084.799993] sysimgblt [ 8084.799993] ghash_clmulni_intel [ 8084.799994] mlx5_core [ 8084.799994] fb_sys_fops [ 8084.799995] igb [ 8084.799995] ttm [ 8084.799996] aesni_intel [ 8084.799996] mlxfw [ 8084.799996] lrw [ 8084.799997] devlink [ 8084.799997] gf128mul [ 8084.799998] dca [ 8084.799998] glue_helper [ 8084.799999] ablk_helper [ 8084.799999] drm [ 8084.800000] dm_multipath [ 8084.800000] ptp [ 8084.800000] cryptd [ 8084.800001] i2c_algo_bit [ 8084.800001] pps_core [ 8084.800002] drm_panel_orientation_quirks [ 8084.800002] wmi [ 8084.800003] sunrpc [ 8084.800003] dm_mirror [ 8084.800004] dm_region_hash [ 8084.800004] dm_log [ 8084.800004] dm_mod [ 8084.800005] iscsi_tcp [ 8084.800005] libiscsi_tcp [ 8084.800006] libiscsi [ 8084.800006] scsi_transport_iscsi [ 8084.800007] fuse [ 8084.800007] [ 8084.800009] CPU: 9 PID: 16822 Comm: ptlrpcd_00_24 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.800010] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.800011] task: ffff8f484d3dc200 ti: ffff8f484d3f8000 task.ti: ffff8f484d3f8000 [ 8084.800012] RIP: 0010:[] [ 8084.800014] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8084.800015] RSP: 0018:ffff8f484d3fbb58 EFLAGS: 00000246 [ 8084.800016] RAX: 0000000000000000 RBX: ffff8f465d969200 RCX: 0000000000490000 [ 8084.800017] RDX: ffff8f687ec9b8c0 RSI: 0000000000a10001 RDI: ffff8f686e2b6b40 [ 8084.800017] RBP: ffff8f484d3fbb58 R08: ffff8f487f65b8c0 R09: 0000000000000000 [ 8084.800018] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.800019] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000b3f23bae [ 8084.800020] FS: 0000000000000000(0000) GS:ffff8f487f640000(0000) knlGS:0000000000000000 [ 8084.800021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.800022] CR2: 00002aaaabaa0aa0 CR3: 0000001f090d8000 CR4: 00000000003607e0 [ 8084.800023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.800024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.800024] Call Trace: [ 8084.800027] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.800029] [] _raw_spin_lock+0x30/0x40 [ 8084.800037] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.800047] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.800080] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.800112] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.800142] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.800176] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.800179] [] ? wake_up_state+0x20/0x20 [ 8084.800212] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.800214] [] kthread+0xd1/0xe0 [ 8084.800216] [] ? insert_kthread_work+0x40/0x40 [ 8084.800218] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8084.800220] [] ? insert_kthread_work+0x40/0x40 [ 8084.800221] Code: [ 8084.800222] 0d [ 8084.800222] 48 [ 8084.800222] 98 [ 8084.800223] 83 [ 8084.800223] e2 [ 8084.800224] 30 [ 8084.800224] 48 [ 8084.800224] 81 [ 8084.800225] c2 [ 8084.800225] c0 [ 8084.800225] b8 [ 8084.800226] 01 [ 8084.800226] 00 [ 8084.800227] 48 [ 8084.800227] 03 [ 8084.800227] 14 [ 8084.800228] c5 [ 8084.800228] e0 [ 8084.800229] 17 [ 8084.800229] 15 [ 8084.800229] 91 [ 8084.800230] 4c [ 8084.800230] 89 [ 8084.800231] 02 [ 8084.800231] 41 [ 8084.800231] 8b [ 8084.800232] 40 [ 8084.800232] 08 [ 8084.800233] 85 [ 8084.800233] c0 [ 8084.800233] 75 [ 8084.800234] 0f [ 8084.800234] 0f [ 8084.800235] 1f [ 8084.800235] 44 [ 8084.800235] 00 [ 8084.800236] 00 [ 8084.800236] f3 [ 8084.800236] 90 [ 8084.800237] 41 [ 8084.800237] 8b [ 8084.800238] 40 [ 8084.800238] 08 [ 8084.800239] <85> [ 8084.800239] c0 [ 8084.800239] 74 [ 8084.800240] f6 [ 8084.800240] 4d [ 8084.800241] 8b [ 8084.800241] 08 [ 8084.800241] 4d [ 8084.800242] 85 [ 8084.800242] c9 [ 8084.800242] 74 [ 8084.800243] 04 [ 8084.800243] 41 [ 8084.800244] 0f [ 8084.800244] 18 [ 8084.800244] 09 [ 8084.800245] 8b [ 8084.800245] 17 [ 8084.800245] 0f [ 8084.800246] b7 [ 8084.800246] c2 [ 8084.800247] [ 8084.824942] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [ptlrpcd_00_08:16806] [ 8084.824943] Modules linked in: [ 8084.824943] mgc(OE) [ 8084.824944] lustre(OE) [ 8084.824944] lmv(OE) [ 8084.824945] mdc(OE) [ 8084.824945] osc(OE) [ 8084.824946] lov(OE) [ 8084.824946] fid(OE) [ 8084.824947] fld(OE) [ 8084.824947] ptlrpc(OE) [ 8084.824948] obdclass(OE) [ 8084.824948] ko2iblnd(OE) [ 8084.824949] lnet(OE) [ 8084.824949] libcfs(OE) [ 8084.824950] gdrdrv(POE) [ 8084.824950] iTCO_wdt [ 8084.824951] iTCO_vendor_support [ 8084.824951] rpcrdma [ 8084.824952] nvidia_drm(POE) [ 8084.824952] ib_iser [ 8084.824953] joydev [ 8084.824953] sb_edac [ 8084.824954] intel_powerclamp [ 8084.824954] coretemp [ 8084.824955] intel_rapl [ 8084.824955] iosf_mbi [ 8084.824955] kvm_intel [ 8084.824956] kvm [ 8084.824956] irqbypass [ 8084.824957] nvidia_modeset(POE) [ 8084.824957] sg [ 8084.824958] pcspkr [ 8084.824958] i2c_i801 [ 8084.824959] lpc_ich [ 8084.824959] nf_log_ipv4 [ 8084.824960] nf_log_common [ 8084.824960] xt_LOG [ 8084.824960] nf_conntrack_ipv4 [ 8084.824961] nf_defrag_ipv4 [ 8084.824961] xt_multiport [ 8084.824962] xt_owner [ 8084.824962] xt_conntrack [ 8084.824963] nf_conntrack [ 8084.824963] libcrc32c [ 8084.824964] iptable_filter [ 8084.824964] ipmi_si [ 8084.824964] ipmi_devintf [ 8084.824965] ipmi_msghandler [ 8084.824965] acpi_power_meter [ 8084.824966] ib_ipoib [ 8084.824966] rdma_ucm [ 8084.824967] ib_umad [ 8084.824967] iw_cxgb4 [ 8084.824968] rdma_cm [ 8084.824968] iw_cm [ 8084.824968] ib_cm [ 8084.824969] iw_cxgb3 [ 8084.824969] sch_fq_codel [ 8084.824970] binfmt_misc [ 8084.824970] msr_safe(OE) [ 8084.824971] ip_tables [ 8084.824971] nfsv3 [ 8084.824972] nfs_acl [ 8084.824972] rpcsec_gss_krb5 [ 8084.824973] auth_rpcgss [ 8084.824973] nfsv4 [ 8084.824974] dns_resolver [ 8084.824974] nfs [ 8084.824974] lockd [ 8084.824975] grace [ 8084.824975] fscache [ 8084.824976] overlay(T) [ 8084.824976] ext4 [ 8084.824977] mbcache [ 8084.824977] jbd2 [ 8084.824978] sd_mod [ 8084.824978] crc_t10dif [ 8084.824979] crct10dif_generic [ 8084.824979] nvidia_uvm(OE) [ 8084.824980] mlx5_ib [ 8084.824980] ib_uverbs [ 8084.824980] be2iscsi [ 8084.824981] ib_core [ 8084.824981] bnx2i [ 8084.824982] cnic [ 8084.824982] uio [ 8084.824983] cxgb4i [ 8084.824983] cxgb4 [ 8084.824984] cxgb3i [ 8084.824984] cxgb3 [ 8084.824984] mdio [ 8084.824985] libcxgbi [ 8084.824985] libcxgb [ 8084.824986] qla4xxx [ 8084.824986] iscsi_boot_sysfs [ 8084.824987] 8021q [ 8084.824987] garp [ 8084.824987] mrp [ 8084.824988] stp [ 8084.824988] llc [ 8084.824989] nvidia(POE) [ 8084.824989] ast [ 8084.824990] drm_kms_helper [ 8084.824990] crct10dif_pclmul [ 8084.824991] crct10dif_common [ 8084.824991] crc32_pclmul [ 8084.824992] crc32c_intel [ 8084.824992] syscopyarea [ 8084.824992] sysfillrect [ 8084.824993] sysimgblt [ 8084.824993] ghash_clmulni_intel [ 8084.824994] mlx5_core [ 8084.824994] fb_sys_fops [ 8084.824995] igb [ 8084.824995] ttm [ 8084.824995] aesni_intel [ 8084.824996] mlxfw [ 8084.824996] lrw [ 8084.824997] devlink [ 8084.824997] gf128mul [ 8084.824998] dca [ 8084.824998] glue_helper [ 8084.824999] ablk_helper [ 8084.824999] drm [ 8084.824999] dm_multipath [ 8084.825000] ptp [ 8084.825000] cryptd [ 8084.825001] i2c_algo_bit [ 8084.825001] pps_core [ 8084.825002] drm_panel_orientation_quirks [ 8084.825002] wmi [ 8084.825003] sunrpc [ 8084.825003] dm_mirror [ 8084.825004] dm_region_hash [ 8084.825004] dm_log [ 8084.825005] dm_mod [ 8084.825005] iscsi_tcp [ 8084.825006] libiscsi_tcp [ 8084.825006] libiscsi [ 8084.825007] scsi_transport_iscsi [ 8084.825007] fuse [ 8084.825007] [ 8084.825009] CPU: 13 PID: 16806 Comm: ptlrpcd_00_08 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.825010] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.825011] task: ffff8f484c67a100 ti: ffff8f484c600000 task.ti: ffff8f484c600000 [ 8084.825012] RIP: 0010:[] [ 8084.825015] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8084.825016] RSP: 0018:ffff8f484c603b58 EFLAGS: 00000246 [ 8084.825017] RAX: 0000000000000000 RBX: ffff8f475076c380 RCX: 0000000000690000 [ 8084.825018] RDX: ffff8f487f6db8c0 RSI: 0000000000590001 RDI: ffff8f686e2b6b40 [ 8084.825019] RBP: ffff8f484c603b58 R08: ffff8f487f75b8c0 R09: 0000000000000000 [ 8084.825020] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.825020] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e69de9aa [ 8084.825022] FS: 0000000000000000(0000) GS:ffff8f487f740000(0000) knlGS:0000000000000000 [ 8084.825023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.825023] CR2: 00007ffff7ff8000 CR3: 0000001ff8618000 CR4: 00000000003607e0 [ 8084.825024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.825025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.825026] Call Trace: [ 8084.825029] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.825031] [] _raw_spin_lock+0x30/0x40 [ 8084.825039] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.825050] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.825084] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.825120] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.825152] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.825187] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.825190] [] ? wake_up_state+0x20/0x20 [ 8084.825223] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.825226] [] kthread+0xd1/0xe0 [ 8084.825228] [] ? insert_kthread_work+0x40/0x40 [ 8084.825230] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8084.825232] [] ? insert_kthread_work+0x40/0x40 [ 8084.825233] Code: [ 8084.825234] 13 [ 8084.825234] 48 [ 8084.825234] c1 [ 8084.825235] ea [ 8084.825235] 0d [ 8084.825236] 48 [ 8084.825236] 98 [ 8084.825236] 83 [ 8084.825237] e2 [ 8084.825237] 30 [ 8084.825237] 48 [ 8084.825238] 81 [ 8084.825238] c2 [ 8084.825239] c0 [ 8084.825239] b8 [ 8084.825239] 01 [ 8084.825240] 00 [ 8084.825240] 48 [ 8084.825241] 03 [ 8084.825241] 14 [ 8084.825241] c5 [ 8084.825242] e0 [ 8084.825242] 17 [ 8084.825243] 15 [ 8084.825243] 91 [ 8084.825243] 4c [ 8084.825244] 89 [ 8084.825244] 02 [ 8084.825245] 41 [ 8084.825245] 8b [ 8084.825245] 40 [ 8084.825246] 08 [ 8084.825246] 85 [ 8084.825247] c0 [ 8084.825247] 75 [ 8084.825247] 0f [ 8084.825248] 0f [ 8084.825248] 1f [ 8084.825248] 44 [ 8084.825249] 00 [ 8084.825249] 00 [ 8084.825250] f3 [ 8084.825250] 90 [ 8084.825251] <41> [ 8084.825251] 8b [ 8084.825251] 40 [ 8084.825252] 08 [ 8084.825252] 85 [ 8084.825253] c0 [ 8084.825253] 74 [ 8084.825253] f6 [ 8084.825254] 4d [ 8084.825254] 8b [ 8084.825255] 08 [ 8084.825255] 4d [ 8084.825255] 85 [ 8084.825256] c9 [ 8084.825256] 74 [ 8084.825257] 04 [ 8084.825257] 41 [ 8084.825257] 0f [ 8084.825258] 18 [ 8084.825258] 09 [ 8084.825258] 8b [ 8084.825259] [ 8084.849941] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [ptlrpcd_00_10:16808] [ 8084.849942] Modules linked in: [ 8084.849942] mgc(OE) [ 8084.849943] lustre(OE) [ 8084.849943] lmv(OE) [ 8084.849944] mdc(OE) [ 8084.849944] osc(OE) [ 8084.849945] lov(OE) [ 8084.849945] fid(OE) [ 8084.849946] fld(OE) [ 8084.849946] ptlrpc(OE) [ 8084.849947] obdclass(OE) [ 8084.849947] ko2iblnd(OE) [ 8084.849948] lnet(OE) [ 8084.849948] libcfs(OE) [ 8084.849949] gdrdrv(POE) [ 8084.849949] iTCO_wdt [ 8084.849950] iTCO_vendor_support [ 8084.849950] rpcrdma [ 8084.849951] nvidia_drm(POE) [ 8084.849951] ib_iser [ 8084.849952] joydev [ 8084.849952] sb_edac [ 8084.849953] intel_powerclamp [ 8084.849953] coretemp [ 8084.849954] intel_rapl [ 8084.849954] iosf_mbi [ 8084.849955] kvm_intel [ 8084.849955] kvm [ 8084.849955] irqbypass [ 8084.849956] nvidia_modeset(POE) [ 8084.849956] sg [ 8084.849957] pcspkr [ 8084.849957] i2c_i801 [ 8084.849958] lpc_ich [ 8084.849958] nf_log_ipv4 [ 8084.849959] nf_log_common [ 8084.849959] xt_LOG [ 8084.849960] nf_conntrack_ipv4 [ 8084.849960] nf_defrag_ipv4 [ 8084.849961] xt_multiport [ 8084.849961] xt_owner [ 8084.849962] xt_conntrack [ 8084.849962] nf_conntrack [ 8084.849963] libcrc32c [ 8084.849963] iptable_filter [ 8084.849963] ipmi_si [ 8084.849964] ipmi_devintf [ 8084.849964] ipmi_msghandler [ 8084.849965] acpi_power_meter [ 8084.849965] ib_ipoib [ 8084.849966] rdma_ucm [ 8084.849966] ib_umad [ 8084.849967] iw_cxgb4 [ 8084.849967] rdma_cm [ 8084.849968] iw_cm [ 8084.849968] ib_cm [ 8084.849968] iw_cxgb3 [ 8084.849969] sch_fq_codel [ 8084.849969] binfmt_misc [ 8084.849970] msr_safe(OE) [ 8084.849970] ip_tables [ 8084.849971] nfsv3 [ 8084.849971] nfs_acl [ 8084.849972] rpcsec_gss_krb5 [ 8084.849972] auth_rpcgss [ 8084.849973] nfsv4 [ 8084.849973] dns_resolver [ 8084.849974] nfs [ 8084.849974] lockd [ 8084.849974] grace [ 8084.849975] fscache [ 8084.849975] overlay(T) [ 8084.849976] ext4 [ 8084.849976] mbcache [ 8084.849977] jbd2 [ 8084.849977] sd_mod [ 8084.849978] crc_t10dif [ 8084.849978] crct10dif_generic [ 8084.849979] nvidia_uvm(OE) [ 8084.849979] mlx5_ib [ 8084.849980] ib_uverbs [ 8084.849980] be2iscsi [ 8084.849981] ib_core [ 8084.849981] bnx2i [ 8084.849981] cnic [ 8084.849982] uio [ 8084.849982] cxgb4i [ 8084.849983] cxgb4 [ 8084.849983] cxgb3i [ 8084.849984] cxgb3 [ 8084.849984] mdio [ 8084.849984] libcxgbi [ 8084.849985] libcxgb [ 8084.849985] qla4xxx [ 8084.849986] iscsi_boot_sysfs [ 8084.849986] 8021q [ 8084.849987] garp [ 8084.849987] mrp [ 8084.849987] stp [ 8084.849988] llc [ 8084.849988] nvidia(POE) [ 8084.849989] ast [ 8084.849989] drm_kms_helper [ 8084.849990] crct10dif_pclmul [ 8084.849990] crct10dif_common [ 8084.849991] crc32_pclmul [ 8084.849991] crc32c_intel [ 8084.849992] syscopyarea [ 8084.849992] sysfillrect [ 8084.849993] sysimgblt [ 8084.849993] ghash_clmulni_intel [ 8084.849994] mlx5_core [ 8084.849994] fb_sys_fops [ 8084.849995] igb [ 8084.849995] ttm [ 8084.849995] aesni_intel [ 8084.849996] mlxfw [ 8084.849996] lrw [ 8084.849997] devlink [ 8084.849997] gf128mul [ 8084.849998] dca [ 8084.849998] glue_helper [ 8084.849999] ablk_helper [ 8084.849999] drm [ 8084.849999] dm_multipath [ 8084.850000] ptp [ 8084.850000] cryptd [ 8084.850001] i2c_algo_bit [ 8084.850001] pps_core [ 8084.850002] drm_panel_orientation_quirks [ 8084.850002] wmi [ 8084.850003] sunrpc [ 8084.850003] dm_mirror [ 8084.850004] dm_region_hash [ 8084.850004] dm_log [ 8084.850004] dm_mod [ 8084.850005] iscsi_tcp [ 8084.850005] libiscsi_tcp [ 8084.850006] libiscsi [ 8084.850006] scsi_transport_iscsi [ 8084.850007] fuse [ 8084.850007] [ 8084.850009] CPU: 17 PID: 16808 Comm: ptlrpcd_00_10 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.850010] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.850011] task: ffff8f484c67c200 ti: ffff8f484c608000 task.ti: ffff8f484c608000 [ 8084.850012] RIP: 0010:[] [ 8084.850014] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8084.850015] RSP: 0018:ffff8f484c60bb58 EFLAGS: 00000246 [ 8084.850016] RAX: 0000000000000000 RBX: ffff8f476daa3f00 RCX: 0000000000890000 [ 8084.850017] RDX: ffff8f687f21b8c0 RSI: 0000000001e10001 RDI: ffff8f686e2b6b40 [ 8084.850018] RBP: ffff8f484c60bb58 R08: ffff8f487f85b8c0 R09: 0000000000000000 [ 8084.850019] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.850019] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000bb7c4242 [ 8084.850021] FS: 0000000000000000(0000) GS:ffff8f487f840000(0000) knlGS:0000000000000000 [ 8084.850021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.850022] CR2: 00002aaaad64527d CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8084.850023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.850024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.850024] Call Trace: [ 8084.850027] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.850029] [] _raw_spin_lock+0x30/0x40 [ 8084.850037] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.850047] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.850080] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.850115] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.850147] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.850182] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.850185] [] ? wake_up_state+0x20/0x20 [ 8084.850219] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.850221] [] kthread+0xd1/0xe0 [ 8084.850223] [] ? insert_kthread_work+0x40/0x40 [ 8084.850225] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8084.850227] [] ? insert_kthread_work+0x40/0x40 [ 8084.850228] Code: [ 8084.850229] 0d [ 8084.850229] 48 [ 8084.850229] 98 [ 8084.850230] 83 [ 8084.850230] e2 [ 8084.850231] 30 [ 8084.850231] 48 [ 8084.850231] 81 [ 8084.850232] c2 [ 8084.850232] c0 [ 8084.850232] b8 [ 8084.850233] 01 [ 8084.850233] 00 [ 8084.850234] 48 [ 8084.850234] 03 [ 8084.850235] 14 [ 8084.850235] c5 [ 8084.850235] e0 [ 8084.850236] 17 [ 8084.850236] 15 [ 8084.850236] 91 [ 8084.850237] 4c [ 8084.850237] 89 [ 8084.850238] 02 [ 8084.850238] 41 [ 8084.850238] 8b [ 8084.850239] 40 [ 8084.850239] 08 [ 8084.850240] 85 [ 8084.850240] c0 [ 8084.850240] 75 [ 8084.850241] 0f [ 8084.850241] 0f [ 8084.850242] 1f [ 8084.850242] 44 [ 8084.850242] 00 [ 8084.850243] 00 [ 8084.850243] f3 [ 8084.850244] 90 [ 8084.850244] 41 [ 8084.850244] 8b [ 8084.850245] 40 [ 8084.850245] 08 [ 8084.850245] <85> [ 8084.850246] c0 [ 8084.850246] 74 [ 8084.850247] f6 [ 8084.850247] 4d [ 8084.850247] 8b [ 8084.850248] 08 [ 8084.850248] 4d [ 8084.850248] 85 [ 8084.850249] c9 [ 8084.850249] 74 [ 8084.850250] 04 [ 8084.850250] 41 [ 8084.850250] 0f [ 8084.850251] 18 [ 8084.850251] 09 [ 8084.850252] 8b [ 8084.850252] 17 [ 8084.850252] 0f [ 8084.850253] b7 [ 8084.850253] c2 [ 8084.850253] [ 8084.956939] NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [ptlrpcd_01_24:16859] [ 8084.956939] Modules linked in: [ 8084.956940] mgc(OE) [ 8084.956941] lustre(OE) [ 8084.956941] lmv(OE) [ 8084.956942] mdc(OE) [ 8084.956942] osc(OE) [ 8084.956943] lov(OE) [ 8084.956943] fid(OE) [ 8084.956943] fld(OE) [ 8084.956944] ptlrpc(OE) [ 8084.956945] obdclass(OE) [ 8084.956945] ko2iblnd(OE) [ 8084.956946] lnet(OE) [ 8084.956946] libcfs(OE) [ 8084.956947] gdrdrv(POE) [ 8084.956947] iTCO_wdt [ 8084.956947] iTCO_vendor_support [ 8084.956948] rpcrdma [ 8084.956949] nvidia_drm(POE) [ 8084.956949] ib_iser [ 8084.956949] joydev [ 8084.956950] sb_edac [ 8084.956950] intel_powerclamp [ 8084.956951] coretemp [ 8084.956951] intel_rapl [ 8084.956952] iosf_mbi [ 8084.956952] kvm_intel [ 8084.956952] kvm [ 8084.956953] irqbypass [ 8084.956953] nvidia_modeset(POE) [ 8084.956954] sg [ 8084.956954] pcspkr [ 8084.956955] i2c_i801 [ 8084.956955] lpc_ich [ 8084.956956] nf_log_ipv4 [ 8084.956956] nf_log_common [ 8084.956956] xt_LOG [ 8084.956957] nf_conntrack_ipv4 [ 8084.956957] nf_defrag_ipv4 [ 8084.956958] xt_multiport [ 8084.956958] xt_owner [ 8084.956959] xt_conntrack [ 8084.956959] nf_conntrack [ 8084.956960] libcrc32c [ 8084.956960] iptable_filter [ 8084.956961] ipmi_si [ 8084.956961] ipmi_devintf [ 8084.956962] ipmi_msghandler [ 8084.956962] acpi_power_meter [ 8084.956963] ib_ipoib [ 8084.956963] rdma_ucm [ 8084.956964] ib_umad [ 8084.956964] iw_cxgb4 [ 8084.956965] rdma_cm [ 8084.956965] iw_cm [ 8084.956965] ib_cm [ 8084.956966] iw_cxgb3 [ 8084.956966] sch_fq_codel [ 8084.956967] binfmt_misc [ 8084.956967] msr_safe(OE) [ 8084.956968] ip_tables [ 8084.956969] nfsv3 [ 8084.956969] nfs_acl [ 8084.956969] rpcsec_gss_krb5 [ 8084.956970] auth_rpcgss [ 8084.956970] nfsv4 [ 8084.956971] dns_resolver [ 8084.956971] nfs [ 8084.956972] lockd [ 8084.956972] grace [ 8084.956973] fscache [ 8084.956973] overlay(T) [ 8084.956974] ext4 [ 8084.956974] mbcache [ 8084.956975] jbd2 [ 8084.956975] sd_mod [ 8084.956975] crc_t10dif [ 8084.956976] crct10dif_generic [ 8084.956977] nvidia_uvm(OE) [ 8084.956977] mlx5_ib [ 8084.956977] ib_uverbs [ 8084.956978] be2iscsi [ 8084.956978] ib_core [ 8084.956979] bnx2i [ 8084.956979] cnic [ 8084.956980] uio [ 8084.956980] cxgb4i [ 8084.956981] cxgb4 [ 8084.956981] cxgb3i [ 8084.956982] cxgb3 [ 8084.956982] mdio [ 8084.956983] libcxgbi [ 8084.956983] libcxgb [ 8084.956983] qla4xxx [ 8084.956984] iscsi_boot_sysfs [ 8084.956985] 8021q [ 8084.956985] garp [ 8084.956986] mrp [ 8084.956986] stp [ 8084.956986] llc [ 8084.956987] nvidia(POE) [ 8084.956988] ast [ 8084.956988] drm_kms_helper [ 8084.956988] crct10dif_pclmul [ 8084.956989] crct10dif_common [ 8084.956989] crc32_pclmul [ 8084.956990] crc32c_intel [ 8084.956990] syscopyarea [ 8084.956991] sysfillrect [ 8084.956992] sysimgblt [ 8084.956992] ghash_clmulni_intel [ 8084.956993] mlx5_core [ 8084.956993] fb_sys_fops [ 8084.956994] igb [ 8084.956994] ttm [ 8084.956995] aesni_intel [ 8084.956995] mlxfw [ 8084.956995] lrw [ 8084.956996] devlink [ 8084.956996] gf128mul [ 8084.956997] dca [ 8084.956997] glue_helper [ 8084.956998] ablk_helper [ 8084.956998] drm [ 8084.956999] dm_multipath [ 8084.956999] ptp [ 8084.956999] cryptd [ 8084.957000] i2c_algo_bit [ 8084.957000] pps_core [ 8084.957001] drm_panel_orientation_quirks [ 8084.957001] wmi [ 8084.957002] sunrpc [ 8084.957002] dm_mirror [ 8084.957003] dm_region_hash [ 8084.957003] dm_log [ 8084.957004] dm_mod [ 8084.957004] iscsi_tcp [ 8084.957005] libiscsi_tcp [ 8084.957005] libiscsi [ 8084.957006] scsi_transport_iscsi [ 8084.957006] fuse [ 8084.957006] [ 8084.957008] CPU: 23 PID: 16859 Comm: ptlrpcd_01_24 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.957009] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.957010] task: ffff8f484fbc5280 ti: ffff8f484fbdc000 task.ti: ffff8f484fbdc000 [ 8084.957011] RIP: 0010:[] [ 8084.957014] [] native_queued_spin_lock_slowpath+0x120/0x200 [ 8084.957015] RSP: 0018:ffff8f484fbdfb58 EFLAGS: 00000246 [ 8084.957016] RAX: 0000000000000000 RBX: ffff8f471c358d80 RCX: 0000000000b90000 [ 8084.957017] RDX: ffff8f687f2db8c0 RSI: 0000000001f90001 RDI: ffff8f686e2b6b40 [ 8084.957018] RBP: ffff8f484fbdfb58 R08: ffff8f687ed5b8c0 R09: 0000000000000000 [ 8084.957018] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.957019] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a400a58d [ 8084.957021] FS: 0000000000000000(0000) GS:ffff8f687ed40000(0000) knlGS:0000000000000000 [ 8084.957022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.957023] CR2: 00002aaaad64527d CR3: 0000003ee53fa000 CR4: 00000000003607e0 [ 8084.957024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.957025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.957025] Call Trace: [ 8084.957028] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.957030] [] _raw_spin_lock+0x30/0x40 [ 8084.957038] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.957048] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.957080] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.957113] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.957144] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.957179] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.957181] [] ? wake_up_state+0x20/0x20 [ 8084.957232] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.957235] [] kthread+0xd1/0xe0 [ 8084.957336] sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8084.957339] CPU: 0 PID: 16803 Comm: ptlrpcd_00_05 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.957340] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.957341] task: ffff8f484fb7e300 ti: ffff8f484c64c000 task.ti: ffff8f484c64c000 [ 8084.957345] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8084.957346] RSP: 0018:ffff8f484c64fb58 EFLAGS: 00000246 [ 8084.957347] RAX: 0000000000000000 RBX: ffff8f46d3e40000 RCX: 0000000000010000 [ 8084.957348] RDX: ffff8f487f7db8c0 RSI: 0000000000790001 RDI: ffff8f686e2b6b40 [ 8084.957348] RBP: ffff8f484c64fb58 R08: ffff8f487f41b8c0 R09: 0000000000000000 [ 8084.957349] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.957350] R13: 0000000000000003 R14: 0000000000000013 R15: 000000000cc63187 [ 8084.957351] FS: 0000000000000000(0000) GS:ffff8f487f400000(0000) knlGS:0000000000000000 [ 8084.957352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.957353] CR2: 00002aaaabaa0aa0 CR3: 0000001dfc636000 CR4: 00000000003607f0 [ 8084.957354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.957355] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.957355] Call Trace: [ 8084.957358] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.957361] [] _raw_spin_lock+0x30/0x40 [ 8084.957369] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.957379] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.957412] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.957415] [] ? del_timer_sync+0x52/0x60 [ 8084.957449] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.957481] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.957517] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.957519] [] ? wake_up_state+0x20/0x20 [ 8084.957553] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.957555] [] kthread+0xd1/0xe0 [ 8084.957558] [] ? insert_kthread_work+0x40/0x40 [ 8084.957560] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8084.957562] [] ? insert_kthread_work+0x40/0x40 [ 8084.957583] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8084.962939] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [ptlrpcd_01_17:16852] [ 8084.962969] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8084.962992] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8084.962994] CPU: 24 PID: 16852 Comm: ptlrpcd_01_17 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8084.962995] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8084.962996] task: ffff8f484fb9d280 ti: ffff8f484fbb8000 task.ti: ffff8f484fbb8000 [ 8084.963000] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8084.963001] RSP: 0018:ffff8f484fbbbb58 EFLAGS: 00000246 [ 8084.963002] RAX: 0000000000000000 RBX: ffff8f65f9cf8000 RCX: 0000000000c10000 [ 8084.963003] RDX: ffff8f487f59b8c0 RSI: 0000000000310001 RDI: ffff8f686e2b6b40 [ 8084.963003] RBP: ffff8f484fbbbb58 R08: ffff8f687ed9b8c0 R09: 0000000000000000 [ 8084.963004] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8084.963005] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000099bba806 [ 8084.963006] FS: 0000000000000000(0000) GS:ffff8f687ed80000(0000) knlGS:0000000000000000 [ 8084.963007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.963008] CR2: 00002aaaad64527d CR3: 0000003e741f4000 CR4: 00000000003607e0 [ 8084.963009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8084.963010] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8084.963010] Call Trace: [ 8084.963013] [] queued_spin_lock_slowpath+0xb/0xf [ 8084.963015] [] _raw_spin_lock+0x30/0x40 [ 8084.963024] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8084.963033] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8084.963066] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8084.963098] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8084.963129] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8084.963164] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8084.963167] [] ? wake_up_state+0x20/0x20 [ 8084.963200] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8084.963202] [] kthread+0xd1/0xe0 [ 8084.963205] [] ? insert_kthread_work+0x40/0x40 [ 8084.963207] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8084.963209] [] ? insert_kthread_work+0x40/0x40 [ 8084.963229] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8085.011937] NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [ptlrpcd_01_35:16870] [ 8085.011966] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.011988] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.011990] CPU: 32 PID: 16870 Comm: ptlrpcd_01_35 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.011991] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.011992] task: ffff8f484d292100 ti: ffff8f484d2a0000 task.ti: ffff8f484d2a0000 [ 8085.011995] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8085.011996] RSP: 0018:ffff8f484d2a3b58 EFLAGS: 00000246 [ 8085.011997] RAX: 0000000000000000 RBX: ffff8f65fc36da00 RCX: 0000000001010000 [ 8085.011998] RDX: ffff8f687ee1b8c0 RSI: 0000000000d10001 RDI: ffff8f686e2b6b40 [ 8085.011999] RBP: ffff8f484d2a3b58 R08: ffff8f687ef9b8c0 R09: 0000000000000000 [ 8085.012000] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.012001] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000d0b041d1 [ 8085.012002] FS: 0000000000000000(0000) GS:ffff8f687ef80000(0000) knlGS:0000000000000000 [ 8085.012003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.012004] CR2: 00002aaaad64527d CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8085.012004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.012005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.012006] Call Trace: [ 8085.012008] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.012010] [] _raw_spin_lock+0x30/0x40 [ 8085.012019] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.012028] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.012060] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.012095] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.012125] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.012160] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.012163] [] ? wake_up_state+0x20/0x20 [ 8085.012196] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.012198] [] kthread+0xd1/0xe0 [ 8085.012200] [] ? insert_kthread_work+0x40/0x40 [ 8085.012202] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.012204] [] ? insert_kthread_work+0x40/0x40 [ 8085.012225] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8085.055936] NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [ptlrpcd_00_12:16810] [ 8085.055966] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.055988] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.055990] CPU: 42 PID: 16810 Comm: ptlrpcd_00_12 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.055991] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.055992] task: ffff8f484c67e300 ti: ffff8f484c61c000 task.ti: ffff8f484c61c000 [ 8085.055995] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8085.055996] RSP: 0018:ffff8f484c61fb58 EFLAGS: 00000246 [ 8085.055997] RAX: 0000000000000000 RBX: ffff8f4764bc5a00 RCX: 0000000001510000 [ 8085.055998] RDX: ffff8f487f45b8c0 RSI: 0000000000090001 RDI: ffff8f686e2b6b40 [ 8085.055999] RBP: ffff8f484c61fb58 R08: ffff8f487fa1b8c0 R09: 0000000000000000 [ 8085.056000] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.056000] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000b73f4650 [ 8085.056002] FS: 0000000000000000(0000) GS:ffff8f487fa00000(0000) knlGS:0000000000000000 [ 8085.056002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.056003] CR2: 00002aaaaad94d70 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8085.056004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.056005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.056005] Call Trace: [ 8085.056008] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.056010] [] _raw_spin_lock+0x30/0x40 [ 8085.056018] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.056028] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.056060] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.056092] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.056122] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.056157] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.056160] [] ? wake_up_state+0x20/0x20 [ 8085.056193] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.056195] [] kthread+0xd1/0xe0 [ 8085.056198] [] ? insert_kthread_work+0x40/0x40 [ 8085.056200] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.056202] [] ? insert_kthread_work+0x40/0x40 [ 8085.056222] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8085.064935] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [ptlrpcd_00_27:16825] [ 8085.064964] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.064987] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.064989] CPU: 45 PID: 16825 Comm: ptlrpcd_00_27 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.064990] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.064991] task: ffff8f484f808000 ti: ffff8f484f804000 task.ti: ffff8f484f804000 [ 8085.064994] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8085.064995] RSP: 0018:ffff8f484f807b58 EFLAGS: 00000246 [ 8085.064996] RAX: 0000000000000000 RBX: ffff8f474b33ad00 RCX: 0000000001690000 [ 8085.064997] RDX: ffff8f487f49b8c0 RSI: 0000000000110001 RDI: ffff8f686e2b6b40 [ 8085.064997] RBP: ffff8f484f807b58 R08: ffff8f487fadb8c0 R09: 0000000000000000 [ 8085.064998] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.064999] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000003f4d10b [ 8085.065000] FS: 0000000000000000(0000) GS:ffff8f487fac0000(0000) knlGS:0000000000000000 [ 8085.065001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.065002] CR2: 00002aaaab1114b1 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8085.065002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.065003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.065004] Call Trace: [ 8085.065006] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.065008] [] _raw_spin_lock+0x30/0x40 [ 8085.065016] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.065026] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.065057] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.065089] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.065121] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.065157] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.065159] [] ? wake_up_state+0x20/0x20 [ 8085.065192] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.065195] [] kthread+0xd1/0xe0 [ 8085.065197] [] ? insert_kthread_work+0x40/0x40 [ 8085.065199] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.065201] [] ? insert_kthread_work+0x40/0x40 [ 8085.065221] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8085.070936] NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [ptlrpcd_00_32:16830] [ 8085.070965] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.070987] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.070989] CPU: 47 PID: 16830 Comm: ptlrpcd_00_32 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.070990] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.070991] task: ffff8f484f80d280 ti: ffff8f484f828000 task.ti: ffff8f484f828000 [ 8085.070994] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8085.070996] RSP: 0018:ffff8f484f82bb58 EFLAGS: 00000246 [ 8085.070996] RAX: 0000000000000000 RBX: ffff8f4761490480 RCX: 0000000001790000 [ 8085.070997] RDX: ffff8f487f9db8c0 RSI: 0000000001490001 RDI: ffff8f686e2b6b40 [ 8085.070998] RBP: ffff8f484f82bb58 R08: ffff8f487fb5b8c0 R09: 0000000000000000 [ 8085.070999] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.071000] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000ab54e064 [ 8085.071001] FS: 0000000000000000(0000) GS:ffff8f487fb40000(0000) knlGS:0000000000000000 [ 8085.071002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.071003] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8085.071003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.071004] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.071005] Call Trace: [ 8085.071007] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.071009] [] _raw_spin_lock+0x30/0x40 [ 8085.071018] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.071027] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.071059] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.071091] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.071121] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.071156] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.071158] [] ? wake_up_state+0x20/0x20 [ 8085.071191] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.071194] [] kthread+0xd1/0xe0 [ 8085.071196] [] ? insert_kthread_work+0x40/0x40 [ 8085.071198] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.071200] [] ? insert_kthread_work+0x40/0x40 [ 8085.071220] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8085.108935] NMI watchdog: BUG: soft lockup - CPU#59 stuck for 22s! [ptlrpcd_01_32:16867] [ 8085.108961] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.108980] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.108982] CPU: 59 PID: 16867 Comm: ptlrpcd_01_32 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.108983] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.108984] task: ffff8f484fbee300 ti: ffff8f484d28c000 task.ti: ffff8f484d28c000 [ 8085.108986] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8085.108987] RSP: 0018:ffff8f484d28fb58 EFLAGS: 00000246 [ 8085.108988] RAX: 0000000000000000 RBX: ffff8f6735cfe780 RCX: 0000000001d90000 [ 8085.108989] RDX: ffff8f687efdb8c0 RSI: 0000000001090001 RDI: ffff8f686e2b6b40 [ 8085.108989] RBP: ffff8f484d28fb58 R08: ffff8f687f1db8c0 R09: 0000000000000000 [ 8085.108990] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.108991] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a91fcf32 [ 8085.108992] FS: 0000000000000000(0000) GS:ffff8f687f1c0000(0000) knlGS:0000000000000000 [ 8085.108992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.108993] CR2: 00002aaaaaad6f58 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8085.108994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.108994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.108995] Call Trace: [ 8085.108997] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.108999] [] _raw_spin_lock+0x30/0x40 [ 8085.109007] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.109016] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.109045] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.109076] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.109104] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.109137] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.109140] [] ? wake_up_state+0x20/0x20 [ 8085.109171] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.109173] [] kthread+0xd1/0xe0 [ 8085.109175] [] ? insert_kthread_work+0x40/0x40 [ 8085.109177] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.109179] [] ? insert_kthread_work+0x40/0x40 [ 8085.109196] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8085.120934] NMI watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [ptlrpcd_01_00:16834] [ 8085.120964] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.120986] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.120989] CPU: 63 PID: 16834 Comm: ptlrpcd_01_00 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.120990] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.120991] task: ffff8f484f83a100 ti: ffff8f484fb0c000 task.ti: ffff8f484fb0c000 [ 8085.120995] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8085.120996] RSP: 0018:ffff8f484fb0fb58 EFLAGS: 00000246 [ 8085.120997] RAX: 0000000000000000 RBX: ffff8f6601229680 RCX: 0000000001f90000 [ 8085.120998] RDX: ffff8f687f0db8c0 RSI: 0000000001b90000 RDI: ffff8f686e2b6b40 [ 8085.120998] RBP: ffff8f484fb0fb58 R08: ffff8f687f2db8c0 R09: 0000000000000000 [ 8085.120999] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.121000] R13: 0000000000000003 R14: 0000000000000013 R15: 000000003fb7af09 [ 8085.121001] FS: 0000000000000000(0000) GS:ffff8f687f2c0000(0000) knlGS:0000000000000000 [ 8085.121002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.121003] CR2: 00002aaaab0fc0a0 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8085.121004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.121005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.121005] Call Trace: [ 8085.121008] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.121010] [] _raw_spin_lock+0x30/0x40 [ 8085.121019] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.121029] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.121061] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.121094] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.121124] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.121159] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.121161] [] ? wake_up_state+0x20/0x20 [ 8085.121195] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.121197] [] kthread+0xd1/0xe0 [ 8085.121199] [] ? insert_kthread_work+0x40/0x40 [ 8085.121201] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.121203] [] ? insert_kthread_work+0x40/0x40 [ 8085.121224] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8085.129934] NMI watchdog: BUG: soft lockup - CPU#66 stuck for 22s! [ptlrpcd_01_08:16843] [ 8085.129964] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.129986] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.129989] CPU: 66 PID: 16843 Comm: ptlrpcd_01_08 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.129990] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.129991] task: ffff8f484fb2b180 ti: ffff8f484fb3c000 task.ti: ffff8f484fb3c000 [ 8085.129994] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8085.129996] RSP: 0018:ffff8f484fb3fb58 EFLAGS: 00000246 [ 8085.129997] RAX: 0000000000000000 RBX: ffff8f6629668000 RCX: 0000000002110000 [ 8085.129998] RDX: ffff8f687f3db8c0 RSI: 0000000002190001 RDI: ffff8f686e2b6b40 [ 8085.129998] RBP: ffff8f484fb3fb58 R08: ffff8f687f39b8c0 R09: 0000000000000000 [ 8085.129999] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.130000] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000138cb389 [ 8085.130001] FS: 0000000000000000(0000) GS:ffff8f687f380000(0000) knlGS:0000000000000000 [ 8085.130002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.130003] CR2: 00002aaab8007088 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8085.130004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.130005] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.130005] Call Trace: [ 8085.130008] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.130010] [] _raw_spin_lock+0x30/0x40 [ 8085.130019] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.130029] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.130061] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.130094] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.130125] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.130160] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.130162] [] ? wake_up_state+0x20/0x20 [ 8085.130196] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.130199] [] kthread+0xd1/0xe0 [ 8085.130201] [] ? insert_kthread_work+0x40/0x40 [ 8085.130203] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.130205] [] ? insert_kthread_work+0x40/0x40 [ 8085.130226] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8085.135934] NMI watchdog: BUG: soft lockup - CPU#68 stuck for 22s! [ptlrpcd_01_22:16857] [ 8085.135963] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8085.135985] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8085.135987] CPU: 68 PID: 16857 Comm: ptlrpcd_01_22 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8085.135988] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8085.135989] task: ffff8f484fbc3180 ti: ffff8f484fbd4000 task.ti: ffff8f484fbd4000 [ 8085.135992] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8085.135993] RSP: 0018:ffff8f484fbd7b58 EFLAGS: 00000246 [ 8085.135994] RAX: 0000000000000000 RBX: ffff8f65feecf980 RCX: 0000000002210000 [ 8085.135995] RDX: ffff8f687f05b8c0 RSI: 0000000001190001 RDI: ffff8f686e2b6b40 [ 8085.135996] RBP: ffff8f484fbd7b58 R08: ffff8f687f41b8c0 R09: 0000000000000000 [ 8085.135996] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8085.135997] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000162586b5 [ 8085.135998] FS: 0000000000000000(0000) GS:ffff8f687f400000(0000) knlGS:0000000000000000 [ 8085.135999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8085.136000] CR2: 00002aaaaafbbd70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8085.136001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8085.136002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8085.136002] Call Trace: [ 8085.136005] [] queued_spin_lock_slowpath+0xb/0xf [ 8085.136006] [] _raw_spin_lock+0x30/0x40 [ 8085.136015] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8085.136024] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8085.136056] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8085.136059] [] ? del_timer_sync+0x52/0x60 [ 8085.136090] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8085.136120] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8085.136153] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8085.136156] [] ? wake_up_state+0x20/0x20 [ 8085.136189] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8085.136191] [] kthread+0xd1/0xe0 [ 8085.136193] [] ? insert_kthread_work+0x40/0x40 [ 8085.136195] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8085.136197] [] ? insert_kthread_work+0x40/0x40 [ 8085.136218] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8088.757846] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_00_13:16811] [ 8088.757875] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8088.757897] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8088.757899] CPU: 2 PID: 16811 Comm: ptlrpcd_00_13 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8088.757900] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8088.757901] task: ffff8f484c620000 ti: ffff8f484c628000 task.ti: ffff8f484c628000 [ 8088.757904] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8088.757906] RSP: 0018:ffff8f484c62bb58 EFLAGS: 00000246 [ 8088.757906] RAX: 0000000000000000 RBX: ffff8f475fcee300 RCX: 0000000000110000 [ 8088.757907] RDX: ffff8f487fc5b8c0 RSI: 0000000001990001 RDI: ffff8f686e2b6b40 [ 8088.757908] RBP: ffff8f484c62bb58 R08: ffff8f487f49b8c0 R09: 0000000000000000 [ 8088.757909] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8088.757910] R13: 0000000000000003 R14: 0000000000000013 R15: 000000005957ed38 [ 8088.757911] FS: 0000000000000000(0000) GS:ffff8f487f480000(0000) knlGS:0000000000000000 [ 8088.757912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8088.757912] CR2: 00007ffff7f84330 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8088.757913] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8088.757914] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8088.757914] Call Trace: [ 8088.757917] [] queued_spin_lock_slowpath+0xb/0xf [ 8088.757919] [] _raw_spin_lock+0x30/0x40 [ 8088.757927] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8088.757937] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8088.757968] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8088.757972] [] ? del_timer_sync+0x52/0x60 [ 8088.758003] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8088.758033] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8088.758067] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8088.758070] [] ? wake_up_state+0x20/0x20 [ 8088.758103] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8088.758105] [] kthread+0xd1/0xe0 [ 8088.758107] [] ? insert_kthread_work+0x40/0x40 [ 8088.758109] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8088.758111] [] ? insert_kthread_work+0x40/0x40 [ 8088.758132] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8088.781846] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [ptlrpcd_00_15:16813] [ 8088.781874] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8088.781896] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8088.781898] CPU: 6 PID: 16813 Comm: ptlrpcd_00_15 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8088.781899] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8088.781900] task: ffff8f484c622100 ti: ffff8f484c630000 task.ti: ffff8f484c630000 [ 8088.781903] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8088.781903] RSP: 0018:ffff8f484c633b58 EFLAGS: 00000246 [ 8088.781904] RAX: 0000000000000000 RBX: ffff8f475fd97500 RCX: 0000000000310000 [ 8088.781905] RDX: ffff8f687f11b8c0 RSI: 0000000001c10001 RDI: ffff8f686e2b6b40 [ 8088.781906] RBP: ffff8f484c633b58 R08: ffff8f487f59b8c0 R09: 0000000000000000 [ 8088.781907] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8088.781907] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000471bd2fd [ 8088.781909] FS: 0000000000000000(0000) GS:ffff8f487f580000(0000) knlGS:0000000000000000 [ 8088.781909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8088.781910] CR2: 00002aaaabaa0aa0 CR3: 0000001dfe0ea000 CR4: 00000000003607e0 [ 8088.781911] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8088.781912] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8088.781912] Call Trace: [ 8088.781915] [] queued_spin_lock_slowpath+0xb/0xf [ 8088.781916] [] _raw_spin_lock+0x30/0x40 [ 8088.781924] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8088.781934] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8088.781966] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8088.781997] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8088.782027] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8088.782061] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8088.782064] [] ? wake_up_state+0x20/0x20 [ 8088.782097] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8088.782099] [] kthread+0xd1/0xe0 [ 8088.782101] [] ? insert_kthread_work+0x40/0x40 [ 8088.782103] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8088.782105] [] ? insert_kthread_work+0x40/0x40 [ 8088.782126] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8088.811845] NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [ptlrpcd_00_23:16821] [ 8088.811873] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8088.811895] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8088.811898] CPU: 11 PID: 16821 Comm: ptlrpcd_00_23 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8088.811898] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8088.811899] task: ffff8f484d3db180 ti: ffff8f484d3ec000 task.ti: ffff8f484d3ec000 [ 8088.811902] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8088.811903] RSP: 0018:ffff8f484d3efb58 EFLAGS: 00000246 [ 8088.811904] RAX: 0000000000000000 RBX: ffff8f4659830900 RCX: 0000000000590000 [ 8088.811905] RDX: ffff8f687ed9b8c0 RSI: 0000000000c10001 RDI: ffff8f686e2b6b40 [ 8088.811906] RBP: ffff8f484d3efb58 R08: ffff8f487f6db8c0 R09: 0000000000000000 [ 8088.811907] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8088.811907] R13: 0000000000000003 R14: 0000000000000013 R15: 000000006d460fff [ 8088.811909] FS: 0000000000000000(0000) GS:ffff8f487f6c0000(0000) knlGS:0000000000000000 [ 8088.811909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8088.811910] CR2: 00002aaaabaa0aa0 CR3: 0000001f25f2e000 CR4: 00000000003607e0 [ 8088.811911] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8088.811912] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8088.811912] Call Trace: [ 8088.811915] [] queued_spin_lock_slowpath+0xb/0xf [ 8088.811917] [] _raw_spin_lock+0x30/0x40 [ 8088.811924] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8088.811934] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8088.811966] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8088.811969] [] ? del_timer_sync+0x52/0x60 [ 8088.812000] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8088.812030] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8088.812064] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8088.812066] [] ? wake_up_state+0x20/0x20 [ 8088.812099] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8088.812102] [] kthread+0xd1/0xe0 [ 8088.812104] [] ? insert_kthread_work+0x40/0x40 [ 8088.812106] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8088.812108] [] ? insert_kthread_work+0x40/0x40 [ 8088.812128] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8089.088838] NMI watchdog: BUG: soft lockup - CPU#53 stuck for 22s! [ptlrpcd_00_21:16819] [ 8089.088868] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8089.088891] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8089.088893] CPU: 53 PID: 16819 Comm: ptlrpcd_00_21 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8089.088893] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8089.088894] task: ffff8f484d3d9080 ti: ffff8f484d3e4000 task.ti: ffff8f484d3e4000 [ 8089.088897] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.088898] RSP: 0018:ffff8f484d3e7b58 EFLAGS: 00000246 [ 8089.088899] RAX: 0000000000000000 RBX: ffff8f475f935e80 RCX: 0000000001a90000 [ 8089.088900] RDX: ffff8f687f41b8c0 RSI: 0000000002210001 RDI: ffff8f686e2b6b40 [ 8089.088901] RBP: ffff8f484d3e7b58 R08: ffff8f487fcdb8c0 R09: 0000000000000000 [ 8089.088902] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8089.088902] R13: 0000000000000003 R14: 0000000000000013 R15: 000000009abfa125 [ 8089.088904] FS: 0000000000000000(0000) GS:ffff8f487fcc0000(0000) knlGS:0000000000000000 [ 8089.088904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8089.088905] CR2: 00002aaaab0fc0a0 CR3: 0000003ffd858000 CR4: 00000000003607e0 [ 8089.088906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8089.088907] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8089.088907] Call Trace: [ 8089.088910] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.088912] [] _raw_spin_lock+0x30/0x40 [ 8089.088920] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.088930] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.088962] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.088965] [] ? del_timer_sync+0x52/0x60 [ 8089.088997] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.089028] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.089062] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.089065] [] ? wake_up_state+0x20/0x20 [ 8089.089099] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.089101] [] kthread+0xd1/0xe0 [ 8089.089104] [] ? insert_kthread_work+0x40/0x40 [ 8089.089105] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.089107] [] ? insert_kthread_work+0x40/0x40 [ 8089.089128] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8089.109843] INFO: rcu_sched self-detected stall on CPU { 32} (t=60000 jiffies g=193015 c=193014 q=15521) [ 8089.109844] Task dump for CPU 19: [ 8089.109846] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.109847] Call Trace: [ 8089.109851] [] ? del_timer_sync+0x52/0x60 [ 8089.109882] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.109913] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.109946] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.109949] [] ? wake_up_state+0x20/0x20 [ 8089.109982] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.109984] [] ? kthread+0xd1/0xe0 [ 8089.109986] [] ? insert_kthread_work+0x40/0x40 [ 8089.109988] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.109990] [] ? insert_kthread_work+0x40/0x40 [ 8089.109991] Task dump for CPU 20: [ 8089.109992] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.109993] Call Trace: [ 8089.110024] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110055] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110087] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110090] [] ? wake_up_state+0x20/0x20 [ 8089.110122] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110124] [] ? kthread+0xd1/0xe0 [ 8089.110127] [] ? insert_kthread_work+0x40/0x40 [ 8089.110128] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110130] [] ? insert_kthread_work+0x40/0x40 [ 8089.110131] Task dump for CPU 24: [ 8089.110133] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.110133] Call Trace: [ 8089.110164] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110194] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110227] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110229] [] ? wake_up_state+0x20/0x20 [ 8089.110262] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110264] [] ? kthread+0xd1/0xe0 [ 8089.110266] [] ? insert_kthread_work+0x40/0x40 [ 8089.110268] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110269] [] ? insert_kthread_work+0x40/0x40 [ 8089.110271] Task dump for CPU 32: [ 8089.110272] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.110272] Call Trace: [ 8089.110276] [] sched_show_task+0xbf/0x120 [ 8089.110278] [] dump_cpu_task+0x39/0x70 [ 8089.110283] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.110287] [] rcu_check_callbacks+0x482/0x770 [ 8089.110289] [] update_process_times+0x46/0x80 [ 8089.110295] [] tick_sched_handle+0x30/0x70 [ 8089.110297] [] tick_sched_timer+0x39/0x80 [ 8089.110299] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.110301] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.110303] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.110307] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.110310] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.110312] [] apic_timer_interrupt+0x16a/0x170 [ 8089.110315] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.110317] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.110319] [] _raw_spin_lock+0x30/0x40 [ 8089.110327] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.110337] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.110368] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.110399] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110431] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110463] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110466] [] ? wake_up_state+0x20/0x20 [ 8089.110498] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110500] [] kthread+0xd1/0xe0 [ 8089.110502] [] ? insert_kthread_work+0x40/0x40 [ 8089.110504] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110506] [] ? insert_kthread_work+0x40/0x40 [ 8089.110507] Task dump for CPU 33: [ 8089.110509] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.110509] Call Trace: [ 8089.110540] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110570] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110601] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110604] [] ? wake_up_state+0x20/0x20 [ 8089.110635] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110638] [] ? kthread+0xd1/0xe0 [ 8089.110640] [] ? insert_kthread_work+0x40/0x40 [ 8089.110641] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110643] [] ? insert_kthread_work+0x40/0x40 [ 8089.110644] Task dump for CPU 34: [ 8089.110646] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.110646] Call Trace: [ 8089.110677] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110707] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110738] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110741] [] ? wake_up_state+0x20/0x20 [ 8089.110772] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110775] [] ? kthread+0xd1/0xe0 [ 8089.110777] [] ? insert_kthread_work+0x40/0x40 [ 8089.110778] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110780] [] ? insert_kthread_work+0x40/0x40 [ 8089.110781] Task dump for CPU 35: [ 8089.110783] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.110783] Call Trace: [ 8089.110814] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.110844] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.110846] INFO: rcu_sched self-detected stall on CPU [ 8089.110847] INFO: rcu_sched self-detected stall on CPU [ 8089.110848] INFO: rcu_sched self-detected stall on CPU [ 8089.110849] INFO: rcu_sched self-detected stall on CPU [ 8089.110850] INFO: rcu_sched self-detected stall on CPU [ 8089.110851] INFO: rcu_sched self-detected stall on CPU [ 8089.110852] INFO: rcu_sched self-detected stall on CPU [ 8089.110853] INFO: rcu_sched self-detected stall on CPU [ 8089.110854] INFO: rcu_sched self-detected stall on CPU [ 8089.110855] INFO: rcu_sched self-detected stall on CPU [ 8089.110856] INFO: rcu_sched self-detected stall on CPU [ 8089.110887] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.110888] INFO: rcu_sched self-detected stall on CPU [ 8089.110888] { [ 8089.110889] { [ 8089.110889] { [ 8089.110890] { [ 8089.110890] { [ 8089.110891] { [ 8089.110891] { [ 8089.110892] { [ 8089.110893] { [ 8089.110893] { [ 8089.110894] { [ 8089.110896] [] ? wake_up_state+0x20/0x20 [ 8089.110897] { [ 8089.110897] 56 [ 8089.110898] 34 [ 8089.110899] 33 [ 8089.110899] 19 [ 8089.110900] 55 [ 8089.110901] 57 [ 8089.110901] 54 [ 8089.110902] 20 [ 8089.110903] 24 [ 8089.110904] 67 [ 8089.110904] 64 [ 8089.110935] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.110936] 35 [ 8089.110936] } [ 8089.110937] } [ 8089.110937] } [ 8089.110938] } [ 8089.110938] } [ 8089.110939] } [ 8089.110940] } [ 8089.110940] } [ 8089.110941] } [ 8089.110942] } [ 8089.110942] } [ 8089.110944] [] ? kthread+0xd1/0xe0 [ 8089.110946] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110947] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110948] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110950] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110951] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110952] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110953] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110954] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110956] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110957] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110958] (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110960] } (t=60001 jiffies g=193015 c=193014 q=15521) [ 8089.110962] [] ? insert_kthread_work+0x40/0x40 [ 8089.110963] Task dump for CPU 19: [ 8089.110965] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.110967] [] ? insert_kthread_work+0x40/0x40 [ 8089.110970] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.110970] Task dump for CPU 54: [ 8089.110971] Call Trace: [ 8089.110973] ptlrpcd_01_18 R running task 0 16853 2 0x00000088 [ 8089.110975] [] ? del_timer_sync+0x52/0x60 [ 8089.110976] Call Trace: [ 8089.111007] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111036] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111066] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111095] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111128] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111159] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111162] [] ? wake_up_state+0x20/0x20 [ 8089.111164] [] ? wake_up_state+0x20/0x20 [ 8089.111196] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111227] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111229] [] ? kthread+0xd1/0xe0 [ 8089.111231] [] ? kthread+0xd1/0xe0 [ 8089.111233] [] ? insert_kthread_work+0x40/0x40 [ 8089.111235] [] ? insert_kthread_work+0x40/0x40 [ 8089.111237] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111239] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111241] [] ? insert_kthread_work+0x40/0x40 [ 8089.111243] [] ? insert_kthread_work+0x40/0x40 [ 8089.111243] Task dump for CPU 20: [ 8089.111244] Task dump for CPU 55: [ 8089.111245] ptlrpcd_01_19 R running task [ 8089.111246] 0 16854 2 0x00000088 [ 8089.111247] Call Trace: [ 8089.111248] ptlrpcd_01_26 R running task 0 16861 2 0x00000088 [ 8089.111248] Call Trace: [ 8089.111279] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111309] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111338] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111367] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111417] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111448] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111451] [] ? wake_up_state+0x20/0x20 [ 8089.111453] [] ? wake_up_state+0x20/0x20 [ 8089.111503] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111535] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111537] [] ? kthread+0xd1/0xe0 [ 8089.111540] [] ? kthread+0xd1/0xe0 [ 8089.111542] [] ? insert_kthread_work+0x40/0x40 [ 8089.111543] [] ? insert_kthread_work+0x40/0x40 [ 8089.111545] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111547] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111549] [] ? insert_kthread_work+0x40/0x40 [ 8089.111551] [] ? insert_kthread_work+0x40/0x40 [ 8089.111551] Task dump for CPU 24: [ 8089.111552] Task dump for CPU 56: [ 8089.111553] ptlrpcd_01_17 R [ 8089.111554] ptlrpcd_01_25 R [ 8089.111554] running task [ 8089.111556] 0 16852 2 0x00000088 [ 8089.111557] running task 0 16860 2 0x00000088 [ 8089.111557] Call Trace: [ 8089.111558] Call Trace: [ 8089.111606] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111636] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111683] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111712] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.111763] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111794] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.111797] [] ? wake_up_state+0x20/0x20 [ 8089.111799] [] ? wake_up_state+0x20/0x20 [ 8089.111849] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111881] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.111884] [] ? kthread+0xd1/0xe0 [ 8089.111887] [] ? kthread+0xd1/0xe0 [ 8089.111889] [] ? insert_kthread_work+0x40/0x40 [ 8089.111891] [] ? insert_kthread_work+0x40/0x40 [ 8089.111893] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111895] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.111897] [] ? insert_kthread_work+0x40/0x40 [ 8089.111898] [] ? insert_kthread_work+0x40/0x40 [ 8089.111899] Task dump for CPU 32: [ 8089.111900] Task dump for CPU 19: [ 8089.111901] Task dump for CPU 57: [ 8089.111902] INFO: rcu_sched detected stalls on CPUs/tasks: { [ 8089.111903] ptlrpcd_01_35 R [ 8089.111903] ptlrpcd_01_31 R [ 8089.111904] ptlrpcd_01_10 R [ 8089.111905] running task [ 8089.111905] running task [ 8089.111906] 0 16870 2 0x00000088 [ 8089.111907] 0 16866 2 0x00000088 [ 8089.111908] running task 0 16845 2 0x00000088 [ 8089.111909] Call Trace: [ 8089.111910] Call Trace: [ 8089.111910] Call Trace: [ 8089.111958] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.111960] [] ? del_timer_sync+0x52/0x60 [ 8089.111990] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112038] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112070] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112100] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112150] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112180] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112212] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112214] [] ? wake_up_state+0x20/0x20 [ 8089.112249] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112252] [] ? wake_up_state+0x20/0x20 [ 8089.112302] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112304] [] ? wake_up_state+0x20/0x20 [ 8089.112335] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112338] [] ? kthread+0xd1/0xe0 [ 8089.112371] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112373] [] ? kthread+0xd1/0xe0 [ 8089.112375] [] ? insert_kthread_work+0x40/0x40 [ 8089.112377] [] ? kthread+0xd1/0xe0 [ 8089.112379] [] ? insert_kthread_work+0x40/0x40 [ 8089.112381] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112383] [] ? insert_kthread_work+0x40/0x40 [ 8089.112385] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112387] [] ? insert_kthread_work+0x40/0x40 [ 8089.112389] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112391] [] ? insert_kthread_work+0x40/0x40 [ 8089.112392] Task dump for CPU 33: [ 8089.112394] [] ? insert_kthread_work+0x40/0x40 [ 8089.112395] Task dump for CPU 64: [ 8089.112396] Task dump for CPU 20: [ 8089.112397] ptlrpcd_01_05 R [ 8089.112397] ptlrpcd_01_09 R [ 8089.112398] running task [ 8089.112399] ptlrpcd_01_19 R [ 8089.112400] 0 16840 2 0x00000088 [ 8089.112400] running task [ 8089.112401] 0 16844 2 0x00000088 [ 8089.112402] Call Trace: [ 8089.112403] running task 0 16854 2 0x00000088 [ 8089.112404] Call Trace: [ 8089.112404] Call Trace: [ 8089.112452] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112484] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112515] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112563] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112592] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112625] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.112675] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112707] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112740] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.112742] [] ? wake_up_state+0x20/0x20 [ 8089.112744] [] ? wake_up_state+0x20/0x20 [ 8089.112747] [] ? wake_up_state+0x20/0x20 [ 8089.112797] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112828] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112860] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.112862] [] ? kthread+0xd1/0xe0 [ 8089.112865] [] ? kthread+0xd1/0xe0 [ 8089.112867] [] ? kthread+0xd1/0xe0 [ 8089.112869] [] ? insert_kthread_work+0x40/0x40 [ 8089.112871] [] ? insert_kthread_work+0x40/0x40 [ 8089.112873] [] ? insert_kthread_work+0x40/0x40 [ 8089.112875] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112877] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112879] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.112881] [] ? insert_kthread_work+0x40/0x40 [ 8089.112883] [] ? insert_kthread_work+0x40/0x40 [ 8089.112884] [] ? insert_kthread_work+0x40/0x40 [ 8089.112885] Task dump for CPU 34: [ 8089.112886] Task dump for CPU 67: [ 8089.112886] Task dump for CPU 24: [ 8089.112887] ptlrpcd_01_04 R [ 8089.112888] ptlrpcd_01_33 R [ 8089.112889] ptlrpcd_01_17 R [ 8089.112889] running task [ 8089.112890] running task [ 8089.112891] 0 16839 2 0x00000088 [ 8089.112892] 0 16868 2 0x00000088 [ 8089.112893] running task 0 16852 2 0x00000088 [ 8089.112894] Call Trace: [ 8089.112894] Call Trace: [ 8089.112895] Call Trace: [ 8089.112942] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.112973] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113004] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113051] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113081] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113113] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113163] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.113196] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.113228] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.113231] [] ? wake_up_state+0x20/0x20 [ 8089.113233] [] ? wake_up_state+0x20/0x20 [ 8089.113235] [] ? wake_up_state+0x20/0x20 [ 8089.113286] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.113317] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.113349] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.113352] [] ? kthread+0xd1/0xe0 [ 8089.113354] [] ? kthread+0xd1/0xe0 [ 8089.113356] [] ? kthread+0xd1/0xe0 [ 8089.113358] [] ? insert_kthread_work+0x40/0x40 [ 8089.113360] [] ? insert_kthread_work+0x40/0x40 [ 8089.113362] [] ? insert_kthread_work+0x40/0x40 [ 8089.113364] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.113366] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.113368] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.113370] [] ? insert_kthread_work+0x40/0x40 [ 8089.113371] [] ? insert_kthread_work+0x40/0x40 [ 8089.113373] [] ? insert_kthread_work+0x40/0x40 [ 8089.113374] Task dump for CPU 35: [ 8089.113375] Task dump for CPU 19: [ 8089.113376] ptlrpcd_01_16 R running task [ 8089.113377] 0 16851 2 0x00000088 [ 8089.113378] Call Trace: [ 8089.113380] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.113427] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113428] Call Trace: [ 8089.113475] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113477] [] ? del_timer_sync+0x52/0x60 [ 8089.113527] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.113559] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113562] [] ? wake_up_state+0x20/0x20 [ 8089.113592] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113642] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.113676] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.113678] [] ? kthread+0xd1/0xe0 [ 8089.113681] [] ? wake_up_state+0x20/0x20 [ 8089.113683] [] ? insert_kthread_work+0x40/0x40 [ 8089.113715] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.113717] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.113720] [] ? kthread+0xd1/0xe0 [ 8089.113721] [] ? insert_kthread_work+0x40/0x40 [ 8089.113723] [] ? insert_kthread_work+0x40/0x40 [ 8089.113724] Task dump for CPU 32: [ 8089.113725] Task dump for CPU 54: [ 8089.113727] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.113728] ptlrpcd_01_35 R [ 8089.113730] [] ? insert_kthread_work+0x40/0x40 [ 8089.113731] ptlrpcd_01_18 R [ 8089.113731] running task [ 8089.113732] Task dump for CPU 20: [ 8089.113733] 0 16870 2 0x00000088 [ 8089.113734] running task 0 16853 2 0x00000088 [ 8089.113735] Call Trace: [ 8089.113736] Call Trace: [ 8089.113737] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.113768] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113816] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113816] Call Trace: [ 8089.113847] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113894] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.113925] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.113958] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114008] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114038] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.114040] [] ? wake_up_state+0x20/0x20 [ 8089.114043] [] ? wake_up_state+0x20/0x20 [ 8089.114076] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114108] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114158] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114160] [] ? wake_up_state+0x20/0x20 [ 8089.114163] [] ? kthread+0xd1/0xe0 [ 8089.114165] [] ? kthread+0xd1/0xe0 [ 8089.114197] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114199] [] ? insert_kthread_work+0x40/0x40 [ 8089.114201] [] ? insert_kthread_work+0x40/0x40 [ 8089.114203] [] ? kthread+0xd1/0xe0 [ 8089.114205] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114207] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114209] [] ? insert_kthread_work+0x40/0x40 [ 8089.114211] [] ? insert_kthread_work+0x40/0x40 [ 8089.114213] [] ? insert_kthread_work+0x40/0x40 [ 8089.114215] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114216] Task dump for CPU 33: [ 8089.114217] Task dump for CPU 55: [ 8089.114219] [] ? insert_kthread_work+0x40/0x40 [ 8089.114220] ptlrpcd_01_05 R [ 8089.114221] ptlrpcd_01_26 R [ 8089.114222] Task dump for CPU 24: [ 8089.114222] running task [ 8089.114223] 0 16840 2 0x00000088 [ 8089.114224] running task [ 8089.114225] 0 16861 2 0x00000088 [ 8089.114225] Call Trace: [ 8089.114226] Call Trace: [ 8089.114227] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.114258] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.114258] Call Trace: [ 8089.114305] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.114336] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.114368] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.114415] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.114448] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114478] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.114528] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114530] [] ? wake_up_state+0x20/0x20 [ 8089.114563] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.114565] [] ? wake_up_state+0x20/0x20 [ 8089.114597] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114599] [] ? wake_up_state+0x20/0x20 [ 8089.114648] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114650] [] ? kthread+0xd1/0xe0 [ 8089.114682] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.114684] [] ? kthread+0xd1/0xe0 [ 8089.114686] [] ? insert_kthread_work+0x40/0x40 [ 8089.114688] [] ? kthread+0xd1/0xe0 [ 8089.114690] [] ? insert_kthread_work+0x40/0x40 [ 8089.114692] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114694] [] ? insert_kthread_work+0x40/0x40 [ 8089.114696] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114698] [] ? insert_kthread_work+0x40/0x40 [ 8089.114701] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.114703] [] ? insert_kthread_work+0x40/0x40 [ 8089.114703] Task dump for CPU 34: [ 8089.114705] [] ? insert_kthread_work+0x40/0x40 [ 8089.114706] Task dump for CPU 56: [ 8089.114707] ptlrpcd_01_04 R [ 8089.114708] Task dump for CPU 19: [ 8089.114709] ptlrpcd_01_25 R [ 8089.114709] running task [ 8089.114710] running task [ 8089.114711] 0 16839 2 0x00000088 [ 8089.114713] 0 16860 2 0x00000088 [ 8089.114713] Call Trace: [ 8089.114714] Call Trace: [ 8089.114716] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.114717] [ 8089.114718] Call Trace: [ 8089.114720] [] sched_show_task+0xbf/0x120 [ 8089.114723] [] sched_show_task+0xbf/0x120 [ 8089.114726] [] dump_cpu_task+0x39/0x70 [ 8089.114728] [] dump_cpu_task+0x39/0x70 [ 8089.114731] [] sched_show_task+0xbf/0x120 [ 8089.114733] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.114736] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.114738] [] dump_cpu_task+0x39/0x70 [ 8089.114740] [] rcu_check_callbacks+0x482/0x770 [ 8089.114743] [] rcu_check_callbacks+0x482/0x770 [ 8089.114745] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.114747] [] update_process_times+0x46/0x80 [ 8089.114750] [] update_process_times+0x46/0x80 [ 8089.114752] [] rcu_check_callbacks+0x482/0x770 [ 8089.114755] [] tick_sched_handle+0x30/0x70 [ 8089.114757] [] tick_sched_handle+0x30/0x70 [ 8089.114760] [] update_process_times+0x46/0x80 [ 8089.114762] [] tick_sched_timer+0x39/0x80 [ 8089.114764] [] tick_sched_timer+0x39/0x80 [ 8089.114767] [] tick_sched_handle+0x30/0x70 [ 8089.114769] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.114770] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.114773] [] tick_sched_timer+0x39/0x80 [ 8089.114775] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.114777] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.114779] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.114781] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.114782] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.114784] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.114786] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.114788] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.114790] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.114792] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.114795] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.114797] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.114798] [] apic_timer_interrupt+0x16a/0x170 [ 8089.114800] [] apic_timer_interrupt+0x16a/0x170 [ 8089.114803] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.114804] [ 8089.114806] [] apic_timer_interrupt+0x16a/0x170 [ 8089.114808] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.114810] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.114813] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.114816] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.114818] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.114820] [] _raw_spin_lock+0x30/0x40 [ 8089.114822] [] _raw_spin_lock+0x30/0x40 [ 8089.114824] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.114832] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.114842] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.114843] [] _raw_spin_lock+0x30/0x40 [ 8089.114853] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.114866] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.114873] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.114903] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.114950] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.114959] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.114991] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115038] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115071] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.115100] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115147] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115150] [] ? del_timer_sync+0x52/0x60 [ 8089.115183] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115233] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115266] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115269] [] ? wake_up_state+0x20/0x20 [ 8089.115271] [] ? wake_up_state+0x20/0x20 [ 8089.115303] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115335] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115386] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115421] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115423] [] kthread+0xd1/0xe0 [ 8089.115426] [] kthread+0xd1/0xe0 [ 8089.115428] [] ? wake_up_state+0x20/0x20 [ 8089.115430] [] ? insert_kthread_work+0x40/0x40 [ 8089.115432] [] ? insert_kthread_work+0x40/0x40 [ 8089.115466] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115468] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115470] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115472] [] kthread+0xd1/0xe0 [ 8089.115474] [] ? insert_kthread_work+0x40/0x40 [ 8089.115476] [] ? insert_kthread_work+0x40/0x40 [ 8089.115479] [] ? insert_kthread_work+0x40/0x40 [ 8089.115479] Task dump for CPU 35: [ 8089.115480] Task dump for CPU 57: [ 8089.115482] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115483] ptlrpcd_01_16 R [ 8089.115485] [] ? insert_kthread_work+0x40/0x40 [ 8089.115486] ptlrpcd_01_10 R [ 8089.115487] running task [ 8089.115488] 0 16851 2 0x00000088 [ 8089.115488] Task dump for CPU 20: [ 8089.115489] running task 0 16845 2 0x00000088 [ 8089.115490] Call Trace: [ 8089.115491] Call Trace: [ 8089.115522] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115524] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.115571] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115601] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115602] Call Trace: [ 8089.115649] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115682] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115714] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.115765] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115767] [] ? wake_up_state+0x20/0x20 [ 8089.115799] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.115801] [] ? wake_up_state+0x20/0x20 [ 8089.115833] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115867] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.115917] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115920] [] ? kthread+0xd1/0xe0 [ 8089.115922] [] ? wake_up_state+0x20/0x20 [ 8089.115925] [] ? kthread+0xd1/0xe0 [ 8089.115927] [] ? insert_kthread_work+0x40/0x40 [ 8089.115959] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.115961] [] ? insert_kthread_work+0x40/0x40 [ 8089.115964] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115966] [] ? kthread+0xd1/0xe0 [ 8089.115968] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115970] [] ? insert_kthread_work+0x40/0x40 [ 8089.115972] [] ? insert_kthread_work+0x40/0x40 [ 8089.115974] [] ? insert_kthread_work+0x40/0x40 [ 8089.115975] Task dump for CPU 32: [ 8089.115977] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.115977] Task dump for CPU 64: [ 8089.115979] Task dump for CPU 54: [ 8089.115981] [] ? insert_kthread_work+0x40/0x40 [ 8089.115981] ptlrpcd_01_35 R [ 8089.115982] ptlrpcd_01_09 R [ 8089.115983] running task [ 8089.115983] Task dump for CPU 24: [ 8089.115984] ptlrpcd_01_18 R [ 8089.115985] 0 16870 2 0x00000088 [ 8089.115986] running task [ 8089.115986] running task [ 8089.115988] 0 16844 2 0x00000088 [ 8089.115988] Call Trace: [ 8089.115989] 0 16853 2 0x00000088 [ 8089.115990] Call Trace: [ 8089.115991] Call Trace: [ 8089.116021] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116022] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.116069] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116100] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116129] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116130] Call Trace: [ 8089.116177] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116207] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116239] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116270] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116321] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116353] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116355] [] ? wake_up_state+0x20/0x20 [ 8089.116387] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116389] [] ? wake_up_state+0x20/0x20 [ 8089.116392] [] ? wake_up_state+0x20/0x20 [ 8089.116423] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.116457] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116507] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.116540] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.116542] [] ? kthread+0xd1/0xe0 [ 8089.116544] [] ? wake_up_state+0x20/0x20 [ 8089.116547] [] ? kthread+0xd1/0xe0 [ 8089.116549] [] ? kthread+0xd1/0xe0 [ 8089.116551] [] ? insert_kthread_work+0x40/0x40 [ 8089.116584] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.116586] [] ? insert_kthread_work+0x40/0x40 [ 8089.116588] [] ? insert_kthread_work+0x40/0x40 [ 8089.116590] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.116593] [] ? kthread+0xd1/0xe0 [ 8089.116595] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.116597] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.116599] [] ? insert_kthread_work+0x40/0x40 [ 8089.116601] [] ? insert_kthread_work+0x40/0x40 [ 8089.116603] [] ? insert_kthread_work+0x40/0x40 [ 8089.116605] [] ? insert_kthread_work+0x40/0x40 [ 8089.116605] Task dump for CPU 33: [ 8089.116608] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.116608] Task dump for CPU 67: [ 8089.116609] Task dump for CPU 55: [ 8089.116611] [] ? insert_kthread_work+0x40/0x40 [ 8089.116612] ptlrpcd_01_05 R [ 8089.116613] ptlrpcd_01_33 R [ 8089.116614] ptlrpcd_01_26 R [ 8089.116615] Task dump for CPU 19: [ 8089.116615] running task [ 8089.116616] running task [ 8089.116617] 0 16840 2 0x00000088 [ 8089.116617] running task [ 8089.116619] 0 16868 2 0x00000088 [ 8089.116620] 0 16861 2 0x00000088 [ 8089.116620] Call Trace: [ 8089.116621] Call Trace: [ 8089.116622] Call Trace: [ 8089.116623] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.116671] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116671] Call Trace: [ 8089.116702] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116704] [] sched_show_task+0xbf/0x120 [ 8089.116751] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116754] [] ? del_timer_sync+0x52/0x60 [ 8089.116784] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116786] [] dump_cpu_task+0x39/0x70 [ 8089.116836] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116869] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.116902] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.116905] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.116907] [] ? wake_up_state+0x20/0x20 [ 8089.116938] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.116940] [] ? wake_up_state+0x20/0x20 [ 8089.116942] [] rcu_check_callbacks+0x482/0x770 [ 8089.116993] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117027] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117060] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117063] [] update_process_times+0x46/0x80 [ 8089.117065] [] ? kthread+0xd1/0xe0 [ 8089.117067] [] ? wake_up_state+0x20/0x20 [ 8089.117070] [] ? kthread+0xd1/0xe0 [ 8089.117072] [] tick_sched_handle+0x30/0x70 [ 8089.117074] [] ? insert_kthread_work+0x40/0x40 [ 8089.117107] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117109] [] ? insert_kthread_work+0x40/0x40 [ 8089.117111] [] tick_sched_timer+0x39/0x80 [ 8089.117113] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117116] [] ? kthread+0xd1/0xe0 [ 8089.117118] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117119] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.117121] [] ? insert_kthread_work+0x40/0x40 [ 8089.117123] [] ? insert_kthread_work+0x40/0x40 [ 8089.117125] [] ? insert_kthread_work+0x40/0x40 [ 8089.117128] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.117130] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117130] Task dump for CPU 56: [ 8089.117132] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.117134] [] ? insert_kthread_work+0x40/0x40 [ 8089.117136] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.117137] Task dump for CPU 20: [ 8089.117140] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.117140] ptlrpcd_01_25 R running task [ 8089.117142] 0 16860 2 0x00000088 [ 8089.117143] [] apic_timer_interrupt+0x16a/0x170 [ 8089.117144] Call Trace: [ 8089.117145] ptlrpcd_01_19 R running task [ 8089.117146] 0 16854 2 0x00000088 [ 8089.117148] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.117179] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117180] Call Trace: [ 8089.117182] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.117212] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117244] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117246] [] _raw_spin_lock+0x30/0x40 [ 8089.117278] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117309] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117317] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.117320] [] ? wake_up_state+0x20/0x20 [ 8089.117353] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117362] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.117394] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117396] [] ? wake_up_state+0x20/0x20 [ 8089.117426] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.117429] [] ? kthread+0xd1/0xe0 [ 8089.117462] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117492] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117494] [] ? insert_kthread_work+0x40/0x40 [ 8089.117496] [] ? kthread+0xd1/0xe0 [ 8089.117526] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117528] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117530] [] ? insert_kthread_work+0x40/0x40 [ 8089.117563] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117565] [] ? insert_kthread_work+0x40/0x40 [ 8089.117567] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117570] [] ? wake_up_state+0x20/0x20 [ 8089.117570] Task dump for CPU 57: [ 8089.117572] [] ? insert_kthread_work+0x40/0x40 [ 8089.117604] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117606] Task dump for CPU 24: [ 8089.117608] [] kthread+0xd1/0xe0 [ 8089.117609] ptlrpcd_01_10 R running task [ 8089.117610] 0 16845 2 0x00000088 [ 8089.117612] [] ? insert_kthread_work+0x40/0x40 [ 8089.117613] Call Trace: [ 8089.117615] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117616] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.117649] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117651] [] ? insert_kthread_work+0x40/0x40 [ 8089.117651] Call Trace: [ 8089.117682] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117682] Task dump for CPU 34: [ 8089.117714] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117747] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117778] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117781] [] ? wake_up_state+0x20/0x20 [ 8089.117814] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117815] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.117847] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117849] [] ? wake_up_state+0x20/0x20 [ 8089.117850] Call Trace: [ 8089.117852] [] ? kthread+0xd1/0xe0 [ 8089.117885] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.117916] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.117918] [] ? insert_kthread_work+0x40/0x40 [ 8089.117920] [] ? kthread+0xd1/0xe0 [ 8089.117950] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.117952] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117954] [] ? insert_kthread_work+0x40/0x40 [ 8089.117987] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.117988] [] ? insert_kthread_work+0x40/0x40 [ 8089.117991] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.117993] [] ? wake_up_state+0x20/0x20 [ 8089.117994] Task dump for CPU 64: [ 8089.117996] [] ? insert_kthread_work+0x40/0x40 [ 8089.118028] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118029] Task dump for CPU 19: [ 8089.118031] [] ? kthread+0xd1/0xe0 [ 8089.118032] ptlrpcd_01_09 R running task 0 16844 2 0x00000088 [ 8089.118035] [] ? insert_kthread_work+0x40/0x40 [ 8089.118036] Call Trace: [ 8089.118038] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118040] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.118070] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118072] [] ? insert_kthread_work+0x40/0x40 [ 8089.118072] Call Trace: [ 8089.118103] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118103] Task dump for CPU 35: [ 8089.118106] [] ? del_timer_sync+0x52/0x60 [ 8089.118138] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118170] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118172] [] ? wake_up_state+0x20/0x20 [ 8089.118203] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118204] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.118236] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118269] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118270] Call Trace: [ 8089.118272] [] ? kthread+0xd1/0xe0 [ 8089.118275] [] ? wake_up_state+0x20/0x20 [ 8089.118305] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118307] [] ? insert_kthread_work+0x40/0x40 [ 8089.118339] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118371] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118373] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118375] [] ? kthread+0xd1/0xe0 [ 8089.118407] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118409] [] ? insert_kthread_work+0x40/0x40 [ 8089.118411] [] ? insert_kthread_work+0x40/0x40 [ 8089.118413] [] ? wake_up_state+0x20/0x20 [ 8089.118413] Task dump for CPU 67: [ 8089.118415] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118447] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118449] [] ? insert_kthread_work+0x40/0x40 [ 8089.118452] [] ? kthread+0xd1/0xe0 [ 8089.118453] ptlrpcd_01_33 R running task 0 16868 2 0x00000088 [ 8089.118453] Task dump for CPU 20: [ 8089.118455] [] ? insert_kthread_work+0x40/0x40 [ 8089.118456] Call Trace: [ 8089.118459] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118489] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118491] [] ? insert_kthread_work+0x40/0x40 [ 8089.118492] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.118523] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118523] Task dump for CPU 54: [ 8089.118524] Task dump for CPU 32: [ 8089.118524] Call Trace: [ 8089.118556] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118558] ptlrpcd_01_18 R [ 8089.118589] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118591] [] ? wake_up_state+0x20/0x20 [ 8089.118592] ptlrpcd_01_35 R [ 8089.118592] running task [ 8089.118622] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118623] 0 16853 2 0x00000088 [ 8089.118654] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118656] running task 0 16870 2 0x00000088 [ 8089.118688] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118689] Call Trace: [ 8089.118691] [] ? kthread+0xd1/0xe0 [ 8089.118692] Call Trace: [ 8089.118694] [] ? wake_up_state+0x20/0x20 [ 8089.118724] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118726] [] ? insert_kthread_work+0x40/0x40 [ 8089.118757] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.118789] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118819] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118821] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118852] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.118855] [] ? kthread+0xd1/0xe0 [ 8089.118887] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118889] [] ? insert_kthread_work+0x40/0x40 [ 8089.118923] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.118925] [] ? insert_kthread_work+0x40/0x40 [ 8089.118927] [] ? wake_up_state+0x20/0x20 [ 8089.118929] [] ? wake_up_state+0x20/0x20 [ 8089.118931] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.118962] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118995] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.118997] [] ? insert_kthread_work+0x40/0x40 [ 8089.118999] [] ? kthread+0xd1/0xe0 [ 8089.119001] [] ? kthread+0xd1/0xe0 [ 8089.119002] Task dump for CPU 24: [ 8089.119004] [] ? insert_kthread_work+0x40/0x40 [ 8089.119006] [] ? insert_kthread_work+0x40/0x40 [ 8089.119009] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119011] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119013] [] ? insert_kthread_work+0x40/0x40 [ 8089.119015] [] ? insert_kthread_work+0x40/0x40 [ 8089.119016] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.119017] Task dump for CPU 55: [ 8089.119018] Task dump for CPU 33: [ 8089.119018] Call Trace: [ 8089.119019] ptlrpcd_01_26 R [ 8089.119020] ptlrpcd_01_05 R [ 8089.119051] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119052] running task [ 8089.119053] 0 16861 2 0x00000088 [ 8089.119082] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119083] running task 0 16840 2 0x00000088 [ 8089.119084] Call Trace: [ 8089.119116] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119116] Call Trace: [ 8089.119146] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119149] [] ? wake_up_state+0x20/0x20 [ 8089.119181] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119210] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119241] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119273] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119305] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119307] [] ? kthread+0xd1/0xe0 [ 8089.119340] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119342] [] ? wake_up_state+0x20/0x20 [ 8089.119344] [] ? insert_kthread_work+0x40/0x40 [ 8089.119347] [] ? wake_up_state+0x20/0x20 [ 8089.119378] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119380] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119413] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119415] [] ? kthread+0xd1/0xe0 [ 8089.119417] [] ? insert_kthread_work+0x40/0x40 [ 8089.119420] [] ? kthread+0xd1/0xe0 [ 8089.119422] [] ? insert_kthread_work+0x40/0x40 [ 8089.119422] Task dump for CPU 19: [ 8089.119424] [] ? insert_kthread_work+0x40/0x40 [ 8089.119427] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119430] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119431] [] ? insert_kthread_work+0x40/0x40 [ 8089.119434] [] ? insert_kthread_work+0x40/0x40 [ 8089.119434] Task dump for CPU 56: [ 8089.119436] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.119437] Task dump for CPU 34: [ 8089.119438] Call Trace: [ 8089.119438] ptlrpcd_01_25 R [ 8089.119439] ptlrpcd_01_04 R [ 8089.119441] running task [ 8089.119443] [] ? del_timer_sync+0x52/0x60 [ 8089.119444] 0 16860 2 0x00000088 [ 8089.119446] running task 0 16839 2 0x00000088 [ 8089.119477] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119478] Call Trace: [ 8089.119478] Call Trace: [ 8089.119511] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119541] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119573] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119607] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119636] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119667] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119670] [] ? wake_up_state+0x20/0x20 [ 8089.119701] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119735] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.119767] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119769] [] ? wake_up_state+0x20/0x20 [ 8089.119772] [] ? wake_up_state+0x20/0x20 [ 8089.119774] [] ? kthread+0xd1/0xe0 [ 8089.119805] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119838] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.119840] [] ? insert_kthread_work+0x40/0x40 [ 8089.119842] [] ? kthread+0xd1/0xe0 [ 8089.119845] [] ? kthread+0xd1/0xe0 [ 8089.119847] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119849] [] ? insert_kthread_work+0x40/0x40 [ 8089.119851] [] ? insert_kthread_work+0x40/0x40 [ 8089.119853] [] ? insert_kthread_work+0x40/0x40 [ 8089.119855] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119857] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.119858] Task dump for CPU 20: [ 8089.119860] [] ? insert_kthread_work+0x40/0x40 [ 8089.119862] [] ? insert_kthread_work+0x40/0x40 [ 8089.119864] Task dump for CPU 57: [ 8089.119864] Task dump for CPU 35: [ 8089.119866] ptlrpcd_01_19 R running task [ 8089.119866] ptlrpcd_01_10 R [ 8089.119867] 0 16854 2 0x00000088 [ 8089.119868] ptlrpcd_01_16 R [ 8089.119869] running task [ 8089.119869] Call Trace: [ 8089.119870] 0 16845 2 0x00000088 [ 8089.119871] running task 0 16851 2 0x00000088 [ 8089.119872] Call Trace: [ 8089.119903] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119903] Call Trace: [ 8089.119933] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.119963] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.119995] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120025] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120057] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120089] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120120] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120123] [] ? wake_up_state+0x20/0x20 [ 8089.120156] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120158] [] ? wake_up_state+0x20/0x20 [ 8089.120191] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120193] [] ? wake_up_state+0x20/0x20 [ 8089.120224] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120226] [] ? kthread+0xd1/0xe0 [ 8089.120259] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120261] [] ? kthread+0xd1/0xe0 [ 8089.120263] [] ? insert_kthread_work+0x40/0x40 [ 8089.120266] [] ? kthread+0xd1/0xe0 [ 8089.120268] [] ? insert_kthread_work+0x40/0x40 [ 8089.120270] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120272] [] ? insert_kthread_work+0x40/0x40 [ 8089.120273] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120275] [] ? insert_kthread_work+0x40/0x40 [ 8089.120277] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120279] [] ? insert_kthread_work+0x40/0x40 [ 8089.120280] Task dump for CPU 24: [ 8089.120282] [] ? insert_kthread_work+0x40/0x40 [ 8089.120283] Task dump for CPU 64: [ 8089.120284] ptlrpcd_01_17 R [ 8089.120285] Task dump for CPU 54: [ 8089.120285] Task dump for CPU 32: [ 8089.120286] ptlrpcd_01_09 R [ 8089.120286] running task [ 8089.120287] running task [ 8089.120288] ptlrpcd_01_18 R [ 8089.120289] 0 16852 2 0x00000088 [ 8089.120290] 0 16844 2 0x00000088 [ 8089.120290] ptlrpcd_01_35 R [ 8089.120290] running task [ 8089.120291] Call Trace: [ 8089.120292] Call Trace: [ 8089.120293] 0 16853 2 0x00000088 [ 8089.120293] running task 0 16870 2 0x00000088 [ 8089.120324] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120324] Call Trace: [ 8089.120354] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120355] Call Trace: [ 8089.120384] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120415] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120445] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120475] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120508] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120540] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120571] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120601] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.120604] [] ? wake_up_state+0x20/0x20 [ 8089.120636] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120639] [] ? wake_up_state+0x20/0x20 [ 8089.120671] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.120703] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120706] [] ? wake_up_state+0x20/0x20 [ 8089.120737] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120739] [] ? wake_up_state+0x20/0x20 [ 8089.120741] [] ? kthread+0xd1/0xe0 [ 8089.120773] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120775] [] ? kthread+0xd1/0xe0 [ 8089.120806] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.120808] [] ? insert_kthread_work+0x40/0x40 [ 8089.120811] [] ? kthread+0xd1/0xe0 [ 8089.120813] [] ? insert_kthread_work+0x40/0x40 [ 8089.120815] [] ? kthread+0xd1/0xe0 [ 8089.120817] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120819] [] ? insert_kthread_work+0x40/0x40 [ 8089.120821] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120823] [] ? insert_kthread_work+0x40/0x40 [ 8089.120825] [] ? insert_kthread_work+0x40/0x40 [ 8089.120827] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120829] [] ? insert_kthread_work+0x40/0x40 [ 8089.120831] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.120831] Task dump for CPU 19: [ 8089.120833] [] ? insert_kthread_work+0x40/0x40 [ 8089.120834] Task dump for CPU 67: [ 8089.120836] [] ? insert_kthread_work+0x40/0x40 [ 8089.120837] Task dump for CPU 55: [ 8089.120838] ptlrpcd_01_31 R [ 8089.120838] Task dump for CPU 33: [ 8089.120839] ptlrpcd_01_33 R [ 8089.120840] running task [ 8089.120840] ptlrpcd_01_26 R [ 8089.120841] running task [ 8089.120842] 0 16866 2 0x00000088 [ 8089.120842] ptlrpcd_01_05 R [ 8089.120843] 0 16868 2 0x00000088 [ 8089.120844] running task [ 8089.120844] Call Trace: [ 8089.120845] 0 16861 2 0x00000088 [ 8089.120846] Call Trace: [ 8089.120847] running task 0 16840 2 0x00000088 [ 8089.120847] Call Trace: [ 8089.120849] [] ? del_timer_sync+0x52/0x60 [ 8089.120850] Call Trace: [ 8089.120880] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120910] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120942] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.120972] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.121002] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121032] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121063] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121092] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121124] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121157] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121191] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121224] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121226] [] ? wake_up_state+0x20/0x20 [ 8089.121229] [] ? wake_up_state+0x20/0x20 [ 8089.121231] [] ? wake_up_state+0x20/0x20 [ 8089.121233] [] ? wake_up_state+0x20/0x20 [ 8089.121265] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121296] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121329] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121360] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121363] [] ? kthread+0xd1/0xe0 [ 8089.121365] [] ? kthread+0xd1/0xe0 [ 8089.121367] [] ? kthread+0xd1/0xe0 [ 8089.121369] [] ? kthread+0xd1/0xe0 [ 8089.121371] [] ? insert_kthread_work+0x40/0x40 [ 8089.121373] [] ? insert_kthread_work+0x40/0x40 [ 8089.121375] [] ? insert_kthread_work+0x40/0x40 [ 8089.121377] [] ? insert_kthread_work+0x40/0x40 [ 8089.121379] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121381] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121383] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121385] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121387] [] ? insert_kthread_work+0x40/0x40 [ 8089.121388] [] ? insert_kthread_work+0x40/0x40 [ 8089.121390] [] ? insert_kthread_work+0x40/0x40 [ 8089.121392] [] ? insert_kthread_work+0x40/0x40 [ 8089.121393] Task dump for CPU 56: [ 8089.121393] Task dump for CPU 20: [ 8089.121394] Task dump for CPU 34: [ 8089.121395] ptlrpcd_01_25 R [ 8089.121396] ptlrpcd_01_19 R [ 8089.121397] ptlrpcd_01_04 R [ 8089.121397] running task [ 8089.121398] running task [ 8089.121399] 0 16860 2 0x00000088 [ 8089.121400] 0 16854 2 0x00000088 [ 8089.121401] running task 0 16839 2 0x00000088 [ 8089.121401] Call Trace: [ 8089.121402] Call Trace: [ 8089.121403] Call Trace: [ 8089.121434] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.121467] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.121499] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121501] [] sched_show_task+0xbf/0x120 [ 8089.121532] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121564] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121566] [] dump_cpu_task+0x39/0x70 [ 8089.121598] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121601] [] ? wake_up_state+0x20/0x20 [ 8089.121603] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.121605] [] ? wake_up_state+0x20/0x20 [ 8089.121636] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121638] [] rcu_check_callbacks+0x482/0x770 [ 8089.121670] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121673] [] ? kthread+0xd1/0xe0 [ 8089.121675] [] update_process_times+0x46/0x80 [ 8089.121676] [] ? kthread+0xd1/0xe0 [ 8089.121678] [] ? insert_kthread_work+0x40/0x40 [ 8089.121681] [] tick_sched_handle+0x30/0x70 [ 8089.121682] [] ? insert_kthread_work+0x40/0x40 [ 8089.121684] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121686] [] tick_sched_timer+0x39/0x80 [ 8089.121688] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.121690] [] ? insert_kthread_work+0x40/0x40 [ 8089.121691] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.121693] [] ? insert_kthread_work+0x40/0x40 [ 8089.121694] Task dump for CPU 57: [ 8089.121696] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.121697] Task dump for CPU 35: [ 8089.121699] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.121699] ptlrpcd_01_10 R [ 8089.121700] ptlrpcd_01_16 R [ 8089.121702] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.121702] running task [ 8089.121703] 0 16845 2 0x00000088 [ 8089.121705] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.121706] running task 0 16851 2 0x00000088 [ 8089.121707] Call Trace: [ 8089.121709] [] apic_timer_interrupt+0x16a/0x170 [ 8089.121709] Call Trace: [ 8089.121740] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.121772] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.121802] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121804] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.121835] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.121867] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121869] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.121902] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.121904] [] ? wake_up_state+0x20/0x20 [ 8089.121906] [] _raw_spin_lock+0x30/0x40 [ 8089.121908] [] ? wake_up_state+0x20/0x20 [ 8089.121939] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121947] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.121980] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.121982] [] ? kthread+0xd1/0xe0 [ 8089.121991] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.121993] [] ? kthread+0xd1/0xe0 [ 8089.121995] [] ? insert_kthread_work+0x40/0x40 [ 8089.122025] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.122027] [] ? insert_kthread_work+0x40/0x40 [ 8089.122029] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122060] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122062] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122064] [] ? insert_kthread_work+0x40/0x40 [ 8089.122094] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122096] [] ? insert_kthread_work+0x40/0x40 [ 8089.122096] Task dump for CPU 64: [ 8089.122130] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122131] Task dump for CPU 54: [ 8089.122132] Task dump for CPU 32: [ 8089.122134] [] ? wake_up_state+0x20/0x20 [ 8089.122135] ptlrpcd_01_09 R running task [ 8089.122136] ptlrpcd_01_18 R [ 8089.122137] 0 16844 2 0x00000088 [ 8089.122170] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122170] ptlrpcd_01_35 R [ 8089.122171] running task [ 8089.122171] Call Trace: [ 8089.122174] [] kthread+0xd1/0xe0 [ 8089.122175] 0 16853 2 0x00000088 [ 8089.122176] running task 0 16870 2 0x00000088 [ 8089.122206] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122208] [] ? insert_kthread_work+0x40/0x40 [ 8089.122208] Call Trace: [ 8089.122209] Call Trace: [ 8089.122238] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122240] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122270] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122300] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122332] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122334] [] ? insert_kthread_work+0x40/0x40 [ 8089.122365] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122397] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122400] [] ? wake_up_state+0x20/0x20 [ 8089.122400] Task dump for CPU 24: [ 8089.122432] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122464] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122496] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122499] [] ? wake_up_state+0x20/0x20 [ 8089.122502] [] ? wake_up_state+0x20/0x20 [ 8089.122504] [] ? kthread+0xd1/0xe0 [ 8089.122535] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122567] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122569] [] ? insert_kthread_work+0x40/0x40 [ 8089.122570] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.122572] [] ? kthread+0xd1/0xe0 [ 8089.122575] [] ? kthread+0xd1/0xe0 [ 8089.122577] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122577] Call Trace: [ 8089.122579] [] ? insert_kthread_work+0x40/0x40 [ 8089.122581] [] ? insert_kthread_work+0x40/0x40 [ 8089.122583] [] ? insert_kthread_work+0x40/0x40 [ 8089.122613] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122615] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122617] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122618] Task dump for CPU 67: [ 8089.122648] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122650] [] ? insert_kthread_work+0x40/0x40 [ 8089.122652] [] ? insert_kthread_work+0x40/0x40 [ 8089.122685] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122686] Task dump for CPU 55: [ 8089.122687] Task dump for CPU 33: [ 8089.122689] [] ? wake_up_state+0x20/0x20 [ 8089.122690] ptlrpcd_01_33 R running task [ 8089.122690] 0 16868 2 0x00000088 [ 8089.122691] ptlrpcd_01_26 R [ 8089.122692] ptlrpcd_01_05 R [ 8089.122723] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122724] Call Trace: [ 8089.122724] running task [ 8089.122726] 0 16861 2 0x00000088 [ 8089.122728] [] ? kthread+0xd1/0xe0 [ 8089.122729] running task 0 16840 2 0x00000088 [ 8089.122760] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122761] Call Trace: [ 8089.122763] [] ? insert_kthread_work+0x40/0x40 [ 8089.122764] Call Trace: [ 8089.122796] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122798] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.122829] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.122862] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122864] [] sched_show_task+0xbf/0x120 [ 8089.122866] [] ? insert_kthread_work+0x40/0x40 [ 8089.122896] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.122898] [] ? wake_up_state+0x20/0x20 [ 8089.122900] [] dump_cpu_task+0x39/0x70 [ 8089.122901] Task dump for CPU 19: [ 8089.122933] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.122965] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.122968] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.122971] [] ? wake_up_state+0x20/0x20 [ 8089.122973] [] ? kthread+0xd1/0xe0 [ 8089.122975] [] rcu_check_callbacks+0x482/0x770 [ 8089.123007] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123009] [] ? insert_kthread_work+0x40/0x40 [ 8089.123011] [] update_process_times+0x46/0x80 [ 8089.123013] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.123015] [] ? kthread+0xd1/0xe0 [ 8089.123017] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123020] [] tick_sched_handle+0x30/0x70 [ 8089.123020] Call Trace: [ 8089.123022] [] ? insert_kthread_work+0x40/0x40 [ 8089.123024] [] ? insert_kthread_work+0x40/0x40 [ 8089.123026] [] tick_sched_timer+0x39/0x80 [ 8089.123029] [] ? del_timer_sync+0x52/0x60 [ 8089.123031] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123032] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.123064] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123066] [] ? insert_kthread_work+0x40/0x40 [ 8089.123068] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.123099] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123099] Task dump for CPU 34: [ 8089.123101] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.123135] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123137] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.123140] [] ? wake_up_state+0x20/0x20 [ 8089.123142] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.123143] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.123176] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123178] [] apic_timer_interrupt+0x16a/0x170 [ 8089.123179] Call Trace: [ 8089.123181] [] ? kthread+0xd1/0xe0 [ 8089.123212] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123214] [] ? insert_kthread_work+0x40/0x40 [ 8089.123216] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.123246] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123248] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123250] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.123282] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123284] [] ? insert_kthread_work+0x40/0x40 [ 8089.123286] [] _raw_spin_lock+0x30/0x40 [ 8089.123288] [] ? wake_up_state+0x20/0x20 [ 8089.123289] Task dump for CPU 20: [ 8089.123297] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.123328] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123338] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.123341] [] ? kthread+0xd1/0xe0 [ 8089.123371] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.123373] [] ? insert_kthread_work+0x40/0x40 [ 8089.123374] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.123405] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123407] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123407] Call Trace: [ 8089.123437] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123439] [] ? insert_kthread_work+0x40/0x40 [ 8089.123470] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123503] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123504] Task dump for CPU 35: [ 8089.123533] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123536] [] ? wake_up_state+0x20/0x20 [ 8089.123568] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123601] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123603] [] ? wake_up_state+0x20/0x20 [ 8089.123604] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.123607] [] kthread+0xd1/0xe0 [ 8089.123639] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123640] Call Trace: [ 8089.123642] [] ? insert_kthread_work+0x40/0x40 [ 8089.123644] [] ? kthread+0xd1/0xe0 [ 8089.123674] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123676] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123678] [] ? insert_kthread_work+0x40/0x40 [ 8089.123707] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123709] [] ? insert_kthread_work+0x40/0x40 [ 8089.123711] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123743] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123744] Task dump for CPU 56: [ 8089.123746] [] ? insert_kthread_work+0x40/0x40 [ 8089.123749] [] ? wake_up_state+0x20/0x20 [ 8089.123750] Task dump for CPU 24: [ 8089.123782] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123783] ptlrpcd_01_25 R running task [ 8089.123784] 0 16860 2 0x00000088 [ 8089.123786] [] ? kthread+0xd1/0xe0 [ 8089.123787] Call Trace: [ 8089.123788] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.123790] [] ? insert_kthread_work+0x40/0x40 [ 8089.123791] Call Trace: [ 8089.123821] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123823] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123854] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.123856] [] ? insert_kthread_work+0x40/0x40 [ 8089.123858] [] sched_show_task+0xbf/0x120 [ 8089.123891] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.123892] Task dump for CPU 32: [ 8089.123894] [] dump_cpu_task+0x39/0x70 [ 8089.123896] [] ? wake_up_state+0x20/0x20 [ 8089.123899] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.123932] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.123934] [] rcu_check_callbacks+0x482/0x770 [ 8089.123935] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.123937] [] ? kthread+0xd1/0xe0 [ 8089.123940] [] update_process_times+0x46/0x80 [ 8089.123940] Call Trace: [ 8089.123942] [] ? insert_kthread_work+0x40/0x40 [ 8089.123945] [] tick_sched_handle+0x30/0x70 [ 8089.123976] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.123978] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.123980] [] tick_sched_timer+0x39/0x80 [ 8089.124012] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124014] [] ? insert_kthread_work+0x40/0x40 [ 8089.124016] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.124048] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124049] Task dump for CPU 57: [ 8089.124051] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.124054] [] ? wake_up_state+0x20/0x20 [ 8089.124056] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.124088] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124090] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.124091] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.124094] [] ? kthread+0xd1/0xe0 [ 8089.124096] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.124097] Call Trace: [ 8089.124098] [] ? insert_kthread_work+0x40/0x40 [ 8089.124100] [] apic_timer_interrupt+0x16a/0x170 [ 8089.124131] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124133] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124164] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124165] [] ? insert_kthread_work+0x40/0x40 [ 8089.124168] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.124200] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124201] Task dump for CPU 33: [ 8089.124203] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.124205] [] ? wake_up_state+0x20/0x20 [ 8089.124208] [] _raw_spin_lock+0x30/0x40 [ 8089.124240] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124241] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.124249] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.124251] [] ? kthread+0xd1/0xe0 [ 8089.124252] Call Trace: [ 8089.124261] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.124263] [] ? insert_kthread_work+0x40/0x40 [ 8089.124294] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124323] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.124326] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124355] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124386] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124388] [] ? insert_kthread_work+0x40/0x40 [ 8089.124420] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124450] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124451] Task dump for CPU 64: [ 8089.124452] Task dump for CPU 54: [ 8089.124454] [] ? wake_up_state+0x20/0x20 [ 8089.124488] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124489] ptlrpcd_01_09 R [ 8089.124521] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124523] [] ? wake_up_state+0x20/0x20 [ 8089.124524] ptlrpcd_01_18 R [ 8089.124524] running task [ 8089.124527] [] ? kthread+0xd1/0xe0 [ 8089.124528] 0 16844 2 0x00000088 [ 8089.124560] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124561] running task 0 16853 2 0x00000088 [ 8089.124563] [] ? insert_kthread_work+0x40/0x40 [ 8089.124563] Call Trace: [ 8089.124565] [] kthread+0xd1/0xe0 [ 8089.124566] Call Trace: [ 8089.124568] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124598] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124601] [] ? insert_kthread_work+0x40/0x40 [ 8089.124631] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124633] [] ? insert_kthread_work+0x40/0x40 [ 8089.124662] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124665] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124694] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124695] Task dump for CPU 34: [ 8089.124727] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124729] [] ? insert_kthread_work+0x40/0x40 [ 8089.124762] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124765] [] ? wake_up_state+0x20/0x20 [ 8089.124765] Task dump for CPU 19: [ 8089.124768] [] ? wake_up_state+0x20/0x20 [ 8089.124800] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124801] ptlrpcd_01_04 R running task [ 8089.124802] 0 16839 2 0x00000088 [ 8089.124834] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.124836] [] ? kthread+0xd1/0xe0 [ 8089.124837] Call Trace: [ 8089.124839] [] ? kthread+0xd1/0xe0 [ 8089.124841] [] ? insert_kthread_work+0x40/0x40 [ 8089.124843] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.124874] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124876] [] ? insert_kthread_work+0x40/0x40 [ 8089.124878] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124878] Call Trace: [ 8089.124908] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.124910] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.124912] [] ? insert_kthread_work+0x40/0x40 [ 8089.124915] [] ? del_timer_sync+0x52/0x60 [ 8089.124947] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.124949] [] ? insert_kthread_work+0x40/0x40 [ 8089.124949] Task dump for CPU 67: [ 8089.124981] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.124983] [] ? wake_up_state+0x20/0x20 [ 8089.124984] Task dump for CPU 55: [ 8089.125017] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125018] ptlrpcd_01_33 R [ 8089.125049] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125050] ptlrpcd_01_26 R [ 8089.125084] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125084] running task [ 8089.125087] [] ? kthread+0xd1/0xe0 [ 8089.125088] 0 16868 2 0x00000088 [ 8089.125090] [] ? wake_up_state+0x20/0x20 [ 8089.125091] running task 0 16861 2 0x00000088 [ 8089.125093] [] ? insert_kthread_work+0x40/0x40 [ 8089.125094] Call Trace: [ 8089.125127] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125128] Call Trace: [ 8089.125129] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125160] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125163] [] ? kthread+0xd1/0xe0 [ 8089.125193] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125195] [] ? insert_kthread_work+0x40/0x40 [ 8089.125225] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125227] [] ? insert_kthread_work+0x40/0x40 [ 8089.125257] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125258] Task dump for CPU 35: [ 8089.125290] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125292] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125324] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125327] [] ? wake_up_state+0x20/0x20 [ 8089.125328] [] ? insert_kthread_work+0x40/0x40 [ 8089.125331] [] ? wake_up_state+0x20/0x20 [ 8089.125363] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125364] Task dump for CPU 20: [ 8089.125365] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.125396] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125398] [] ? kthread+0xd1/0xe0 [ 8089.125399] Call Trace: [ 8089.125402] [] ? kthread+0xd1/0xe0 [ 8089.125404] [] ? insert_kthread_work+0x40/0x40 [ 8089.125435] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125437] [] ? insert_kthread_work+0x40/0x40 [ 8089.125439] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125440] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.125470] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125472] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125474] [] ? insert_kthread_work+0x40/0x40 [ 8089.125475] Call Trace: [ 8089.125507] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125509] [] ? insert_kthread_work+0x40/0x40 [ 8089.125540] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125542] [] ? wake_up_state+0x20/0x20 [ 8089.125543] Task dump for CPU 56: [ 8089.125575] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125607] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125640] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125642] [] ? kthread+0xd1/0xe0 [ 8089.125645] [] ? wake_up_state+0x20/0x20 [ 8089.125646] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.125648] [] ? insert_kthread_work+0x40/0x40 [ 8089.125680] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125681] Call Trace: [ 8089.125683] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125685] [] ? kthread+0xd1/0xe0 [ 8089.125715] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125717] [] ? insert_kthread_work+0x40/0x40 [ 8089.125719] [] ? insert_kthread_work+0x40/0x40 [ 8089.125749] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125749] Task dump for CPU 32: [ 8089.125751] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125783] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125786] [] ? insert_kthread_work+0x40/0x40 [ 8089.125789] [] ? wake_up_state+0x20/0x20 [ 8089.125790] Task dump for CPU 24: [ 8089.125791] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.125822] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.125824] Call Trace: [ 8089.125826] [] ? kthread+0xd1/0xe0 [ 8089.125857] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125859] [] ? insert_kthread_work+0x40/0x40 [ 8089.125860] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.125890] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.125892] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.125892] Call Trace: [ 8089.125925] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.125927] [] ? insert_kthread_work+0x40/0x40 [ 8089.125959] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.125962] [] ? wake_up_state+0x20/0x20 [ 8089.125962] Task dump for CPU 57: [ 8089.125993] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126025] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126058] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126060] [] ? kthread+0xd1/0xe0 [ 8089.126061] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.126063] [] ? wake_up_state+0x20/0x20 [ 8089.126065] [] ? insert_kthread_work+0x40/0x40 [ 8089.126066] Call Trace: [ 8089.126098] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126100] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126103] [] ? kthread+0xd1/0xe0 [ 8089.126105] [] ? insert_kthread_work+0x40/0x40 [ 8089.126107] [] sched_show_task+0xbf/0x120 [ 8089.126109] [] ? insert_kthread_work+0x40/0x40 [ 8089.126110] Task dump for CPU 33: [ 8089.126112] [] dump_cpu_task+0x39/0x70 [ 8089.126114] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126117] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.126119] [] ? insert_kthread_work+0x40/0x40 [ 8089.126121] [] rcu_check_callbacks+0x482/0x770 [ 8089.126122] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.126123] Task dump for CPU 19: [ 8089.126125] [] update_process_times+0x46/0x80 [ 8089.126126] Call Trace: [ 8089.126129] [] tick_sched_handle+0x30/0x70 [ 8089.126160] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126162] [] tick_sched_timer+0x39/0x80 [ 8089.126163] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.126193] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126194] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.126195] Call Trace: [ 8089.126227] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126229] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.126231] [] ? del_timer_sync+0x52/0x60 [ 8089.126233] [] ? wake_up_state+0x20/0x20 [ 8089.126235] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.126266] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126297] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126299] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.126329] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126331] [] ? kthread+0xd1/0xe0 [ 8089.126334] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.126367] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126369] [] ? insert_kthread_work+0x40/0x40 [ 8089.126371] [] apic_timer_interrupt+0x16a/0x170 [ 8089.126373] [] ? wake_up_state+0x20/0x20 [ 8089.126375] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126408] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126410] [] ? insert_kthread_work+0x40/0x40 [ 8089.126412] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.126414] [] ? kthread+0xd1/0xe0 [ 8089.126415] Task dump for CPU 34: [ 8089.126417] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.126419] [] ? insert_kthread_work+0x40/0x40 [ 8089.126422] [] _raw_spin_lock+0x30/0x40 [ 8089.126424] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126432] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.126434] [] ? insert_kthread_work+0x40/0x40 [ 8089.126435] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.126445] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.126445] Task dump for CPU 20: [ 8089.126446] Call Trace: [ 8089.126476] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.126506] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126508] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.126540] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126570] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126570] Call Trace: [ 8089.126600] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126633] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126663] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126696] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126699] [] ? wake_up_state+0x20/0x20 [ 8089.126728] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126730] [] ? wake_up_state+0x20/0x20 [ 8089.126762] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126794] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.126826] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126829] [] ? kthread+0xd1/0xe0 [ 8089.126831] [] ? wake_up_state+0x20/0x20 [ 8089.126833] [] kthread+0xd1/0xe0 [ 8089.126835] [] ? insert_kthread_work+0x40/0x40 [ 8089.126867] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.126869] [] ? insert_kthread_work+0x40/0x40 [ 8089.126871] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126874] [] ? kthread+0xd1/0xe0 [ 8089.126876] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126878] [] ? insert_kthread_work+0x40/0x40 [ 8089.126879] [] ? insert_kthread_work+0x40/0x40 [ 8089.126881] [] ? insert_kthread_work+0x40/0x40 [ 8089.126882] Task dump for CPU 35: [ 8089.126884] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.126885] Task dump for CPU 64: [ 8089.126886] Task dump for CPU 54: [ 8089.126888] [] ? insert_kthread_work+0x40/0x40 [ 8089.126889] ptlrpcd_01_16 R [ 8089.126890] ptlrpcd_01_09 R [ 8089.126891] running task [ 8089.126891] Task dump for CPU 24: [ 8089.126892] ptlrpcd_01_18 R [ 8089.126893] 0 16851 2 0x00000088 [ 8089.126894] running task [ 8089.126894] running task [ 8089.126895] 0 16844 2 0x00000088 [ 8089.126896] Call Trace: [ 8089.126897] 0 16853 2 0x00000088 [ 8089.126898] Call Trace: [ 8089.126898] Call Trace: [ 8089.126928] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126929] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.126960] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.126990] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.126990] Call Trace: [ 8089.126993] [] sched_show_task+0xbf/0x120 [ 8089.127022] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127054] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127084] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127087] [] dump_cpu_task+0x39/0x70 [ 8089.127119] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127121] [] ? wake_up_state+0x20/0x20 [ 8089.127151] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127153] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.127155] [] ? wake_up_state+0x20/0x20 [ 8089.127186] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127219] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127221] [] rcu_check_callbacks+0x482/0x770 [ 8089.127252] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127255] [] ? kthread+0xd1/0xe0 [ 8089.127257] [] ? wake_up_state+0x20/0x20 [ 8089.127259] [] update_process_times+0x46/0x80 [ 8089.127262] [] ? kthread+0xd1/0xe0 [ 8089.127264] [] ? insert_kthread_work+0x40/0x40 [ 8089.127295] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127298] [] tick_sched_handle+0x30/0x70 [ 8089.127300] [] ? insert_kthread_work+0x40/0x40 [ 8089.127302] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127304] [] ? kthread+0xd1/0xe0 [ 8089.127306] [] tick_sched_timer+0x39/0x80 [ 8089.127308] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127310] [] ? insert_kthread_work+0x40/0x40 [ 8089.127312] [] ? insert_kthread_work+0x40/0x40 [ 8089.127314] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.127316] [] ? insert_kthread_work+0x40/0x40 [ 8089.127317] Task dump for CPU 32: [ 8089.127319] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127321] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.127321] Task dump for CPU 67: [ 8089.127324] [] ? insert_kthread_work+0x40/0x40 [ 8089.127326] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.127326] ptlrpcd_01_35 R [ 8089.127327] ptlrpcd_01_33 R [ 8089.127328] Task dump for CPU 19: [ 8089.127329] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.127330] running task [ 8089.127331] 0 16870 2 0x00000088 [ 8089.127332] running task [ 8089.127334] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.127335] 0 16868 2 0x00000088 [ 8089.127336] Call Trace: [ 8089.127338] [] apic_timer_interrupt+0x16a/0x170 [ 8089.127339] Call Trace: [ 8089.127340] ptlrpcd_01_31 R running task 0 16866 2 0x00000088 [ 8089.127371] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127402] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127403] Call Trace: [ 8089.127433] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127435] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.127464] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127467] [] ? del_timer_sync+0x52/0x60 [ 8089.127499] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127502] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.127534] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127568] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127571] [] ? wake_up_state+0x20/0x20 [ 8089.127573] [] _raw_spin_lock+0x30/0x40 [ 8089.127575] [] ? wake_up_state+0x20/0x20 [ 8089.127605] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127637] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127645] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.127677] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127711] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127714] [] ? kthread+0xd1/0xe0 [ 8089.127723] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.127725] [] ? kthread+0xd1/0xe0 [ 8089.127727] [] ? wake_up_state+0x20/0x20 [ 8089.127729] [] ? insert_kthread_work+0x40/0x40 [ 8089.127759] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.127761] [] ? insert_kthread_work+0x40/0x40 [ 8089.127794] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127796] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127829] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127831] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127834] [] ? kthread+0xd1/0xe0 [ 8089.127836] [] ? insert_kthread_work+0x40/0x40 [ 8089.127866] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.127868] [] ? insert_kthread_work+0x40/0x40 [ 8089.127870] [] ? insert_kthread_work+0x40/0x40 [ 8089.127871] Task dump for CPU 33: [ 8089.127904] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.127907] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127910] [] ? wake_up_state+0x20/0x20 [ 8089.127912] [] ? insert_kthread_work+0x40/0x40 [ 8089.127945] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.127946] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.127947] Task dump for CPU 20: [ 8089.127949] [] kthread+0xd1/0xe0 [ 8089.127950] Call Trace: [ 8089.127954] [] ? insert_kthread_work+0x40/0x40 [ 8089.127985] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.127987] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.127988] ptlrpcd_01_19 R running task 0 16854 2 0x00000088 [ 8089.128018] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128020] [] ? insert_kthread_work+0x40/0x40 [ 8089.128020] Call Trace: [ 8089.128053] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128054] Task dump for CPU 55: [ 8089.128088] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128090] [] ? wake_up_state+0x20/0x20 [ 8089.128121] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128154] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128155] ptlrpcd_01_26 R running task 0 16861 2 0x00000088 [ 8089.128188] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128190] [] ? kthread+0xd1/0xe0 [ 8089.128191] Call Trace: [ 8089.128193] [] ? wake_up_state+0x20/0x20 [ 8089.128195] [] ? insert_kthread_work+0x40/0x40 [ 8089.128225] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128258] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128260] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128291] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128293] [] ? kthread+0xd1/0xe0 [ 8089.128295] [] ? insert_kthread_work+0x40/0x40 [ 8089.128328] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128330] [] ? insert_kthread_work+0x40/0x40 [ 8089.128330] Task dump for CPU 34: [ 8089.128333] [] ? wake_up_state+0x20/0x20 [ 8089.128335] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128367] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128369] [] ? insert_kthread_work+0x40/0x40 [ 8089.128372] [] ? kthread+0xd1/0xe0 [ 8089.128373] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.128374] Task dump for CPU 24: [ 8089.128376] [] ? insert_kthread_work+0x40/0x40 [ 8089.128376] Call Trace: [ 8089.128379] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128410] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128412] [] ? insert_kthread_work+0x40/0x40 [ 8089.128414] ptlrpcd_01_17 R running task 0 16852 2 0x00000088 [ 8089.128443] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128444] Task dump for CPU 56: [ 8089.128445] Call Trace: [ 8089.128478] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128510] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128512] [] ? wake_up_state+0x20/0x20 [ 8089.128545] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128546] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.128578] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128610] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128611] Call Trace: [ 8089.128613] [] ? kthread+0xd1/0xe0 [ 8089.128616] [] ? wake_up_state+0x20/0x20 [ 8089.128646] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128648] [] ? insert_kthread_work+0x40/0x40 [ 8089.128680] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128710] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128712] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128715] [] ? kthread+0xd1/0xe0 [ 8089.128747] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128749] [] ? insert_kthread_work+0x40/0x40 [ 8089.128751] [] ? insert_kthread_work+0x40/0x40 [ 8089.128753] [] ? wake_up_state+0x20/0x20 [ 8089.128754] Task dump for CPU 35: [ 8089.128756] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128788] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128790] [] ? insert_kthread_work+0x40/0x40 [ 8089.128793] [] ? kthread+0xd1/0xe0 [ 8089.128794] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.128797] [] ? insert_kthread_work+0x40/0x40 [ 8089.128797] Call Trace: [ 8089.128800] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.128831] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128833] [] ? insert_kthread_work+0x40/0x40 [ 8089.128863] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128863] Task dump for CPU 57: [ 8089.128896] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.128896] 19 20 24 [ 8089.128899] [] ? wake_up_state+0x20/0x20 [ 8089.128900] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.128932] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.128932] Call Trace: [ 8089.128935] [] ? kthread+0xd1/0xe0 [ 8089.128965] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.128967] [] ? insert_kthread_work+0x40/0x40 [ 8089.128996] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.128998] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129030] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129032] [] ? insert_kthread_work+0x40/0x40 [ 8089.129035] [] ? wake_up_state+0x20/0x20 [ 8089.129035] Task dump for CPU 32: [ 8089.129083] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.129086] [] ? kthread+0xd1/0xe0 [ 8089.129088] [] ? insert_kthread_work+0x40/0x40 [ 8089.129089] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.129091] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129092] Call Trace: [ 8089.129094] [] ? insert_kthread_work+0x40/0x40 [ 8089.129126] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129127] Task dump for CPU 64: [ 8089.129127] Task dump for CPU 54: [ 8089.129157] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129159] ptlrpcd_01_09 R [ 8089.129192] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129192] ptlrpcd_01_18 R [ 8089.129193] running task [ 8089.129195] [] ? wake_up_state+0x20/0x20 [ 8089.129197] 0 16844 2 0x00000088 [ 8089.129198] running task 0 16853 2 0x00000088 [ 8089.129230] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.129230] Call Trace: [ 8089.129231] Call Trace: [ 8089.129233] [] ? kthread+0xd1/0xe0 [ 8089.129281] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129312] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129314] [] ? insert_kthread_work+0x40/0x40 [ 8089.129361] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129391] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129393] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129443] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129476] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129478] [] ? insert_kthread_work+0x40/0x40 [ 8089.129480] [] ? wake_up_state+0x20/0x20 [ 8089.129482] [] ? wake_up_state+0x20/0x20 [ 8089.129483] Task dump for CPU 33: [ 8089.129533] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.129566] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.129568] [] ? kthread+0xd1/0xe0 [ 8089.129571] [] ? kthread+0xd1/0xe0 [ 8089.129573] [] ? insert_kthread_work+0x40/0x40 [ 8089.129574] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.129576] [] ? insert_kthread_work+0x40/0x40 [ 8089.129578] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129578] Call Trace: [ 8089.129580] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129582] [] ? insert_kthread_work+0x40/0x40 [ 8089.129615] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129617] [] ? insert_kthread_work+0x40/0x40 [ 8089.129617] Task dump for CPU 67: [ 8089.129647] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129648] Task dump for CPU 55: [ 8089.129680] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129681] ptlrpcd_01_33 R [ 8089.129682] ptlrpcd_01_26 R [ 8089.129684] [] ? wake_up_state+0x20/0x20 [ 8089.129685] running task [ 8089.129686] 0 16868 2 0x00000088 [ 8089.129687] running task 0 16861 2 0x00000088 [ 8089.129719] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.129719] Call Trace: [ 8089.129720] Call Trace: [ 8089.129722] [] ? kthread+0xd1/0xe0 [ 8089.129769] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129799] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.129801] [] ? insert_kthread_work+0x40/0x40 [ 8089.129849] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129879] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.129881] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.129931] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129963] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.129965] [] ? insert_kthread_work+0x40/0x40 [ 8089.129967] [] ? wake_up_state+0x20/0x20 [ 8089.129970] [] ? wake_up_state+0x20/0x20 [ 8089.129970] Task dump for CPU 34: [ 8089.130020] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130052] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130055] [] ? kthread+0xd1/0xe0 [ 8089.130057] [] ? kthread+0xd1/0xe0 [ 8089.130058] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.130060] [] ? insert_kthread_work+0x40/0x40 [ 8089.130062] [] ? insert_kthread_work+0x40/0x40 [ 8089.130063] Call Trace: [ 8089.130065] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130067] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130099] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130101] [] ? insert_kthread_work+0x40/0x40 [ 8089.130103] [] ? insert_kthread_work+0x40/0x40 [ 8089.130133] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130133] Task dump for CPU 56: [ 8089.130166] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130169] [] ? wake_up_state+0x20/0x20 [ 8089.130170] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.130202] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130203] Call Trace: [ 8089.130205] [] ? kthread+0xd1/0xe0 [ 8089.130235] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130237] [] ? insert_kthread_work+0x40/0x40 [ 8089.130266] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130268] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130300] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130302] [] ? insert_kthread_work+0x40/0x40 [ 8089.130304] [] ? wake_up_state+0x20/0x20 [ 8089.130305] Task dump for CPU 35: [ 8089.130337] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130339] [] ? kthread+0xd1/0xe0 [ 8089.130340] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.130342] [] ? insert_kthread_work+0x40/0x40 [ 8089.130343] Call Trace: [ 8089.130345] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130377] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130379] [] ? insert_kthread_work+0x40/0x40 [ 8089.130409] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130410] Task dump for CPU 57: [ 8089.130442] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130445] [] ? wake_up_state+0x20/0x20 [ 8089.130446] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.130478] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130478] Call Trace: [ 8089.130481] [] ? kthread+0xd1/0xe0 [ 8089.130511] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130513] [] ? insert_kthread_work+0x40/0x40 [ 8089.130542] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130544] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130575] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130578] [] ? insert_kthread_work+0x40/0x40 [ 8089.130580] [] ? wake_up_state+0x20/0x20 [ 8089.130580] Task dump for CPU 32: [ 8089.130612] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130615] [] ? kthread+0xd1/0xe0 [ 8089.130616] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.130618] [] ? insert_kthread_work+0x40/0x40 [ 8089.130618] Call Trace: [ 8089.130620] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130650] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130652] [] ? insert_kthread_work+0x40/0x40 [ 8089.130681] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130682] Task dump for CPU 64: [ 8089.130683] Task dump for CPU 54: [ 8089.130715] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130716] ptlrpcd_01_09 R [ 8089.130718] [] ? wake_up_state+0x20/0x20 [ 8089.130719] ptlrpcd_01_18 R [ 8089.130720] running task [ 8089.130721] 0 16844 2 0x00000088 [ 8089.130752] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.130753] running task 0 16853 2 0x00000088 [ 8089.130754] Call Trace: [ 8089.130756] [] ? kthread+0xd1/0xe0 [ 8089.130756] Call Trace: [ 8089.130786] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130788] [] ? insert_kthread_work+0x40/0x40 [ 8089.130819] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.130848] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130850] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.130881] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.130913] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130915] [] ? insert_kthread_work+0x40/0x40 [ 8089.130948] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.130950] [] ? wake_up_state+0x20/0x20 [ 8089.130951] Task dump for CPU 33: [ 8089.130953] [] ? wake_up_state+0x20/0x20 [ 8089.130984] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131017] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131019] [] ? kthread+0xd1/0xe0 [ 8089.131022] [] ? kthread+0xd1/0xe0 [ 8089.131024] [] ? insert_kthread_work+0x40/0x40 [ 8089.131025] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.131027] [] ? insert_kthread_work+0x40/0x40 [ 8089.131029] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131029] Call Trace: [ 8089.131031] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131033] [] ? insert_kthread_work+0x40/0x40 [ 8089.131064] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131066] [] ? insert_kthread_work+0x40/0x40 [ 8089.131066] Task dump for CPU 67: [ 8089.131096] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131097] Task dump for CPU 55: [ 8089.131129] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131130] ptlrpcd_01_33 R [ 8089.131130] ptlrpcd_01_26 R [ 8089.131133] [] ? wake_up_state+0x20/0x20 [ 8089.131133] running task [ 8089.131134] 0 16868 2 0x00000088 [ 8089.131135] running task 0 16861 2 0x00000088 [ 8089.131167] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131167] Call Trace: [ 8089.131168] Call Trace: [ 8089.131170] [] ? kthread+0xd1/0xe0 [ 8089.131200] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131231] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131233] [] ? insert_kthread_work+0x40/0x40 [ 8089.131262] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131294] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131296] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131328] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131361] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131362] [] ? insert_kthread_work+0x40/0x40 [ 8089.131365] [] ? wake_up_state+0x20/0x20 [ 8089.131367] [] ? wake_up_state+0x20/0x20 [ 8089.131368] Task dump for CPU 34: [ 8089.131399] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131433] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131435] [] ? kthread+0xd1/0xe0 [ 8089.131438] [] ? kthread+0xd1/0xe0 [ 8089.131439] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.131441] [] ? insert_kthread_work+0x40/0x40 [ 8089.131443] [] ? insert_kthread_work+0x40/0x40 [ 8089.131443] Call Trace: [ 8089.131445] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131447] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131478] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131480] [] ? insert_kthread_work+0x40/0x40 [ 8089.131482] [] ? insert_kthread_work+0x40/0x40 [ 8089.131511] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131511] Task dump for CPU 56: [ 8089.131543] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131546] [] ? wake_up_state+0x20/0x20 [ 8089.131547] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.131578] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131578] Call Trace: [ 8089.131581] [] ? kthread+0xd1/0xe0 [ 8089.131611] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131613] [] ? insert_kthread_work+0x40/0x40 [ 8089.131642] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131645] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131677] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131679] [] ? insert_kthread_work+0x40/0x40 [ 8089.131681] [] ? wake_up_state+0x20/0x20 [ 8089.131681] Task dump for CPU 35: [ 8089.131714] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131716] [] ? kthread+0xd1/0xe0 [ 8089.131717] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.131719] [] ? insert_kthread_work+0x40/0x40 [ 8089.131719] Call Trace: [ 8089.131721] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131751] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131753] [] ? insert_kthread_work+0x40/0x40 [ 8089.131783] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131783] Task dump for CPU 57: [ 8089.131814] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131817] [] ? wake_up_state+0x20/0x20 [ 8089.131818] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.131850] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.131850] Call Trace: [ 8089.131852] [] ? kthread+0xd1/0xe0 [ 8089.131883] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.131885] [] ? insert_kthread_work+0x40/0x40 [ 8089.131916] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.131917] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.131968] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.131970] [] ? insert_kthread_work+0x40/0x40 [ 8089.131972] [] ? wake_up_state+0x20/0x20 [ 8089.131973] Task dump for CPU 32: [ 8089.132023] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132027] [] ? kthread+0xd1/0xe0 [ 8089.132029] [] ? insert_kthread_work+0x40/0x40 [ 8089.132030] ptlrpcd_01_35 R running task 0 16870 2 0x00000088 [ 8089.132032] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.132032] Call Trace: [ 8089.132034] [] ? insert_kthread_work+0x40/0x40 [ 8089.132065] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132066] Task dump for CPU 64: [ 8089.132067] Task dump for CPU 54: [ 8089.132097] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132098] ptlrpcd_01_09 R [ 8089.132099] ptlrpcd_01_18 R [ 8089.132132] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132132] running task [ 8089.132133] 0 16844 2 0x00000088 [ 8089.132136] [] ? wake_up_state+0x20/0x20 [ 8089.132137] running task 0 16853 2 0x00000088 [ 8089.132137] Call Trace: [ 8089.132170] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132170] Call Trace: [ 8089.132218] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132220] [] ? kthread+0xd1/0xe0 [ 8089.132250] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132298] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132300] [] ? insert_kthread_work+0x40/0x40 [ 8089.132330] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132380] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132382] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.132414] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132417] [] ? wake_up_state+0x20/0x20 [ 8089.132419] [] ? insert_kthread_work+0x40/0x40 [ 8089.132421] [] ? wake_up_state+0x20/0x20 [ 8089.132471] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132472] Task dump for CPU 33: [ 8089.132504] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132506] [] ? kthread+0xd1/0xe0 [ 8089.132509] [] ? kthread+0xd1/0xe0 [ 8089.132511] [] ? insert_kthread_work+0x40/0x40 [ 8089.132513] [] ? insert_kthread_work+0x40/0x40 [ 8089.132515] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.132516] ptlrpcd_01_05 R running task 0 16840 2 0x00000088 [ 8089.132518] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.132520] [] ? insert_kthread_work+0x40/0x40 [ 8089.132521] Call Trace: [ 8089.132523] [] ? insert_kthread_work+0x40/0x40 [ 8089.132523] Task dump for CPU 67: [ 8089.132554] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132555] Task dump for CPU 55: [ 8089.132585] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132586] ptlrpcd_01_33 R running task [ 8089.132587] 0 16868 2 0x00000088 [ 8089.132620] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132621] Call Trace: [ 8089.132623] [] ? wake_up_state+0x20/0x20 [ 8089.132624] ptlrpcd_01_26 R running task 0 16861 2 0x00000088 [ 8089.132671] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132704] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132704] Call Trace: [ 8089.132752] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132754] [] ? kthread+0xd1/0xe0 [ 8089.132787] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.132837] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132839] [] ? insert_kthread_work+0x40/0x40 [ 8089.132871] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.132874] [] ? wake_up_state+0x20/0x20 [ 8089.132876] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.132908] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.132958] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132960] [] ? insert_kthread_work+0x40/0x40 [ 8089.132962] [] ? wake_up_state+0x20/0x20 [ 8089.132965] [] ? kthread+0xd1/0xe0 [ 8089.132965] Task dump for CPU 34: [ 8089.132997] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.132999] [] ? insert_kthread_work+0x40/0x40 [ 8089.133002] [] ? kthread+0xd1/0xe0 [ 8089.133004] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133007] [] ? insert_kthread_work+0x40/0x40 [ 8089.133009] [] ? insert_kthread_work+0x40/0x40 [ 8089.133010] ptlrpcd_01_04 R running task 0 16839 2 0x00000088 [ 8089.133013] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133013] Call Trace: [ 8089.133015] [] ? insert_kthread_work+0x40/0x40 [ 8089.133046] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133046] Task dump for CPU 56: [ 8089.133076] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133109] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133110] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.133113] [] ? wake_up_state+0x20/0x20 [ 8089.133113] Call Trace: [ 8089.133146] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133178] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133180] [] ? kthread+0xd1/0xe0 [ 8089.133212] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133213] [] ? insert_kthread_work+0x40/0x40 [ 8089.133245] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133247] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133250] [] ? wake_up_state+0x20/0x20 [ 8089.133251] [] ? insert_kthread_work+0x40/0x40 [ 8089.133283] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133284] Task dump for CPU 35: [ 8089.133286] [] ? kthread+0xd1/0xe0 [ 8089.133288] [] ? insert_kthread_work+0x40/0x40 [ 8089.133291] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133292] ptlrpcd_01_16 R running task 0 16851 2 0x00000088 [ 8089.133293] [] ? insert_kthread_work+0x40/0x40 [ 8089.133294] Call Trace: [ 8089.133295] Task dump for CPU 57: [ 8089.133296] [ 8089.133298] [] sched_show_task+0xbf/0x120 [ 8089.133301] [] dump_cpu_task+0x39/0x70 [ 8089.133302] ptlrpcd_01_10 R running task 0 16845 2 0x00000088 [ 8089.133304] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.133305] Call Trace: [ 8089.133307] [] rcu_check_callbacks+0x482/0x770 [ 8089.133337] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133340] [] update_process_times+0x46/0x80 [ 8089.133371] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133374] [] tick_sched_handle+0x30/0x70 [ 8089.133405] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133408] [] tick_sched_timer+0x39/0x80 [ 8089.133410] [] ? wake_up_state+0x20/0x20 [ 8089.133412] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.133443] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133445] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.133447] [] ? kthread+0xd1/0xe0 [ 8089.133449] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.133451] [] ? insert_kthread_work+0x40/0x40 [ 8089.133452] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.133454] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133457] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.133458] [] ? insert_kthread_work+0x40/0x40 [ 8089.133460] [] apic_timer_interrupt+0x16a/0x170 [ 8089.133461] Task dump for CPU 64: [ 8089.133462] Task dump for CPU 54: [ 8089.133463] [ 8089.133464] ptlrpcd_01_09 R [ 8089.133466] [] ? native_queued_spin_lock_slowpath+0x126/0x200 [ 8089.133466] ptlrpcd_01_18 R [ 8089.133467] running task [ 8089.133469] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.133470] 0 16844 2 0x00000088 [ 8089.133471] running task 0 16853 2 0x00000088 [ 8089.133473] [] _raw_spin_lock+0x30/0x40 [ 8089.133474] Call Trace: [ 8089.133474] Call Trace: [ 8089.133482] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.133512] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133543] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133553] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.133582] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133613] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133643] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.133675] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133707] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133738] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133740] [] ? wake_up_state+0x20/0x20 [ 8089.133743] [] ? wake_up_state+0x20/0x20 [ 8089.133773] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.133804] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133836] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133870] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.133873] [] ? kthread+0xd1/0xe0 [ 8089.133875] [] ? kthread+0xd1/0xe0 [ 8089.133877] [] ? wake_up_state+0x20/0x20 [ 8089.133879] [] ? insert_kthread_work+0x40/0x40 [ 8089.133881] [] ? insert_kthread_work+0x40/0x40 [ 8089.133914] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.133916] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133918] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133921] [] kthread+0xd1/0xe0 [ 8089.133923] [] ? insert_kthread_work+0x40/0x40 [ 8089.133925] [] ? insert_kthread_work+0x40/0x40 [ 8089.133927] [] ? insert_kthread_work+0x40/0x40 [ 8089.133928] Task dump for CPU 67: [ 8089.133928] Task dump for CPU 55: [ 8089.133930] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.133932] ptlrpcd_01_33 R [ 8089.133934] [] ? insert_kthread_work+0x40/0x40 [ 8089.133934] ptlrpcd_01_26 R [ 8089.133935] running task [ 8089.133936] 0 16868 2 0x00000088 [ 8089.133936] running task [ 8089.133937] 0 16861 2 0x00000088 [ 8089.133938] Call Trace: [ 8089.133939] Call Trace: [ 8089.133940] 32 33 34 [ 8089.133940] [ 8089.133943] [] sched_show_task+0xbf/0x120 [ 8089.133973] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.133976] [] dump_cpu_task+0x39/0x70 [ 8089.134005] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.134007] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8089.134039] [] ? ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.134041] [] rcu_check_callbacks+0x482/0x770 [ 8089.134043] [] ? wake_up_state+0x20/0x20 [ 8089.134045] [] update_process_times+0x46/0x80 [ 8089.134076] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.134079] [] tick_sched_handle+0x30/0x70 [ 8089.134081] [] ? kthread+0xd1/0xe0 [ 8089.134083] [] tick_sched_timer+0x39/0x80 [ 8089.134085] [] ? insert_kthread_work+0x40/0x40 [ 8089.134087] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8089.134089] [] ? ret_from_fork_nospec_begin+0x7/0x21 [ 8089.134091] [] ? tick_sched_do_timer+0x50/0x50 [ 8089.134093] [] ? insert_kthread_work+0x40/0x40 [ 8089.134094] [] hrtimer_interrupt+0xb9/0x1f0 [ 8089.134095] Task dump for CPU 56: [ 8089.134097] [] local_apic_timer_interrupt+0x3b/0x60 [ 8089.134097] 35 [ 8089.134100] [] smp_apic_timer_interrupt+0x43/0x60 [ 8089.134102] [] apic_timer_interrupt+0x16a/0x170 [ 8089.134103] ptlrpcd_01_25 R running task 0 16860 2 0x00000088 [ 8089.134104] Call Trace: [ 8089.134106] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8089.134136] [] ? ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.134138] [] queued_spin_lock_slowpath+0xb/0xf [ 8089.134170] [] ? ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.134172] [] _raw_spin_lock+0x30/0x40 [ 8089.134181] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8089.134191] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8089.134222] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8089.134256] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8089.134259] [] ? insert_kthread_work+0x40/0x40 [ 8089.134289] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8089.134291] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.134324] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8089.134326] [] ? insert_kthread_work+0x40/0x40 [ 8089.134329] [] ? wake_up_state+0x20/0x20 [ 8089.134362] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8089.134364] [] kthread+0xd1/0xe0 [ 8089.134367] [] ? insert_kthread_work+0x40/0x40 [ 8089.134369] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8089.134372] [] ? insert_kthread_work+0x40/0x40 [ 8089.134390] Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 [ 8092.749748] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ptlrpcd_00_34:16832] [ 8092.749778] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8092.749801] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8092.749803] CPU: 1 PID: 16832 Comm: ptlrpcd_00_34 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8092.749804] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8092.749805] task: ffff8f484f838000 ti: ffff8f484f834000 task.ti: ffff8f484f834000 [ 8092.749808] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8092.749809] RSP: 0018:ffff8f484f837b58 EFLAGS: 00000246 [ 8092.749810] RAX: 0000000000000000 RBX: ffff8f4759d50480 RCX: 0000000000090000 [ 8092.749811] RDX: ffff8f487f8db8c0 RSI: 0000000001290001 RDI: ffff8f686e2b6b40 [ 8092.749812] RBP: ffff8f484f837b58 R08: ffff8f487f45b8c0 R09: 0000000000000000 [ 8092.749812] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8092.749813] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000005172c6d [ 8092.749814] FS: 0000000000000000(0000) GS:ffff8f487f440000(0000) knlGS:0000000000000000 [ 8092.749815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8092.749816] CR2: 00007ffff7ff8000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8092.749817] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8092.749818] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8092.749818] Call Trace: [ 8092.749821] [] queued_spin_lock_slowpath+0xb/0xf [ 8092.749823] [] _raw_spin_lock+0x30/0x40 [ 8092.749831] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8092.749841] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8092.749873] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8092.749905] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8092.749935] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8092.749970] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8092.749973] [] ? wake_up_state+0x20/0x20 [ 8092.750006] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8092.750009] [] kthread+0xd1/0xe0 [ 8092.750011] [] ? insert_kthread_work+0x40/0x40 [ 8092.750013] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8092.750015] [] ? insert_kthread_work+0x40/0x40 [ 8092.750035] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8093.058741] NMI watchdog: BUG: soft lockup - CPU#43 stuck for 23s! [ptlrpcd_00_14:16812] [ 8093.058770] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8093.058792] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8093.058794] CPU: 43 PID: 16812 Comm: ptlrpcd_00_14 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8093.058795] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8093.058796] task: ffff8f484c621080 ti: ffff8f484c62c000 task.ti: ffff8f484c62c000 [ 8093.058799] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8093.058800] RSP: 0018:ffff8f484c62fb58 EFLAGS: 00000246 [ 8093.058801] RAX: 0000000000000000 RBX: ffff8f4801fd9680 RCX: 0000000001590000 [ 8093.058802] RDX: ffff8f687ef5b8c0 RSI: 0000000000f90001 RDI: ffff8f686e2b6b40 [ 8093.058803] RBP: ffff8f484c62fb58 R08: ffff8f487fa5b8c0 R09: 0000000000000000 [ 8093.058804] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8093.058804] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e057be9c [ 8093.058806] FS: 0000000000000000(0000) GS:ffff8f487fa40000(0000) knlGS:0000000000000000 [ 8093.058807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8093.058807] CR2: 00002aaab4006338 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8093.058808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8093.058809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8093.058810] Call Trace: [ 8093.058812] [] queued_spin_lock_slowpath+0xb/0xf [ 8093.058814] [] _raw_spin_lock+0x30/0x40 [ 8093.058822] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8093.058832] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8093.058864] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8093.058896] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8093.058926] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8093.058960] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8093.058963] [] ? wake_up_state+0x20/0x20 [ 8093.058996] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8093.058999] [] kthread+0xd1/0xe0 [ 8093.059001] [] ? insert_kthread_work+0x40/0x40 [ 8093.059003] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8093.059005] [] ? insert_kthread_work+0x40/0x40 [ 8093.059025] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8096.769651] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ptlrpcd_00_31:16829] [ 8096.769681] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8096.769703] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8096.769706] CPU: 4 PID: 16829 Comm: ptlrpcd_00_31 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8096.769707] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8096.769708] task: ffff8f484f80c200 ti: ffff8f484f81c000 task.ti: ffff8f484f81c000 [ 8096.769712] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8096.769713] RSP: 0018:ffff8f484f81fb58 EFLAGS: 00000246 [ 8096.769714] RAX: 0000000000000000 RBX: ffff8f4845698d80 RCX: 0000000000210000 [ 8096.769715] RDX: ffff8f487f89b8c0 RSI: 0000000001210001 RDI: ffff8f686e2b6b40 [ 8096.769716] RBP: ffff8f484f81fb58 R08: ffff8f487f51b8c0 R09: 0000000000000000 [ 8096.769716] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8096.769717] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000e1529bd0 [ 8096.769719] FS: 0000000000000000(0000) GS:ffff8f487f500000(0000) knlGS:0000000000000000 [ 8096.769719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.769720] CR2: 00002aaaabaa0aa0 CR3: 0000001f06eda000 CR4: 00000000003607e0 [ 8096.769721] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8096.769722] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8096.769722] Call Trace: [ 8096.769726] [] queued_spin_lock_slowpath+0xb/0xf [ 8096.769728] [] _raw_spin_lock+0x30/0x40 [ 8096.769736] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8096.769746] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8096.769778] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8096.769812] [] ? ptlrpc_unregister_reply+0x120/0x880 [ptlrpc] [ 8096.769843] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8096.769872] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8096.769907] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8096.769909] [] ? wake_up_state+0x20/0x20 [ 8096.769942] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8096.769945] [] kthread+0xd1/0xe0 [ 8096.769947] [] ? insert_kthread_work+0x40/0x40 [ 8096.769949] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8096.769951] [] ? insert_kthread_work+0x40/0x40 [ 8096.769971] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8096.775650] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ptlrpcd_00_26:16824] [ 8096.775679] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8096.775701] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8096.775703] CPU: 5 PID: 16824 Comm: ptlrpcd_00_26 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8096.775704] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8096.775705] task: ffff8f484d3de300 ti: ffff8f484f800000 task.ti: ffff8f484f800000 [ 8096.775709] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8096.775710] RSP: 0018:ffff8f484f803b58 EFLAGS: 00000246 [ 8096.775710] RAX: 0000000000000000 RBX: ffff8f4847d53180 RCX: 0000000000290000 [ 8096.775711] RDX: ffff8f487fb1b8c0 RSI: 0000000001710001 RDI: ffff8f686e2b6b40 [ 8096.775712] RBP: ffff8f484f803b58 R08: ffff8f487f55b8c0 R09: 0000000000000000 [ 8096.775713] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8096.775714] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000fc20c75e [ 8096.775715] FS: 0000000000000000(0000) GS:ffff8f487f540000(0000) knlGS:0000000000000000 [ 8096.775716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.775717] CR2: 00002aaaabaa0aa0 CR3: 0000001dff1b8000 CR4: 00000000003607e0 [ 8096.775718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8096.775718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8096.775719] Call Trace: [ 8096.775722] [] queued_spin_lock_slowpath+0xb/0xf [ 8096.775724] [] _raw_spin_lock+0x30/0x40 [ 8096.775732] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8096.775742] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8096.775774] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8096.775778] [] ? del_timer_sync+0x52/0x60 [ 8096.775810] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8096.775840] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8096.775875] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8096.775878] [] ? wake_up_state+0x20/0x20 [ 8096.775912] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8096.775914] [] kthread+0xd1/0xe0 [ 8096.775916] [] ? insert_kthread_work+0x40/0x40 [ 8096.775918] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8096.775920] [] ? insert_kthread_work+0x40/0x40 [ 8096.775941] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8096.787650] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ptlrpcd_00_17:16815] [ 8096.787679] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8096.787701] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8096.787703] CPU: 7 PID: 16815 Comm: ptlrpcd_00_17 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8096.787703] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8096.787704] task: ffff8f484c624200 ti: ffff8f484d3c4000 task.ti: ffff8f484d3c4000 [ 8096.787707] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8096.787708] RSP: 0018:ffff8f484d3c7b58 EFLAGS: 00000246 [ 8096.787709] RAX: 0000000000000000 RBX: ffff8f4855cd6780 RCX: 0000000000390000 [ 8096.787710] RDX: ffff8f687eedb8c0 RSI: 0000000000e90001 RDI: ffff8f686e2b6b40 [ 8096.787711] RBP: ffff8f484d3c7b58 R08: ffff8f487f5db8c0 R09: 0000000000000000 [ 8096.787711] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8096.787712] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000099c3dd79 [ 8096.787713] FS: 0000000000000000(0000) GS:ffff8f487f5c0000(0000) knlGS:0000000000000000 [ 8096.787714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.787715] CR2: 00002aaaabaa0aa0 CR3: 0000001fbf0b8000 CR4: 00000000003607e0 [ 8096.787716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8096.787716] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8096.787717] Call Trace: [ 8096.787719] [] queued_spin_lock_slowpath+0xb/0xf [ 8096.787721] [] _raw_spin_lock+0x30/0x40 [ 8096.787729] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8096.787739] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8096.787770] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8096.787802] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8096.787834] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8096.787868] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8096.787871] [] ? wake_up_state+0x20/0x20 [ 8096.787903] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8096.787906] [] kthread+0xd1/0xe0 [ 8096.787908] [] ? insert_kthread_work+0x40/0x40 [ 8096.787910] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8096.787912] [] ? insert_kthread_work+0x40/0x40 [ 8096.787932] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8096.842649] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [ptlrpcd_00_04:16802] [ 8096.842678] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8096.842701] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8096.842703] CPU: 16 PID: 16802 Comm: ptlrpcd_00_04 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8096.842704] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8096.842705] task: ffff8f484fb7d280 ti: ffff8f484c648000 task.ti: ffff8f484c648000 [ 8096.842708] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8096.842709] RSP: 0018:ffff8f484c64bb58 EFLAGS: 00000246 [ 8096.842710] RAX: 0000000000000000 RBX: ffff8f4818acad00 RCX: 0000000000810000 [ 8096.842711] RDX: ffff8f687f39b8c0 RSI: 0000000002110001 RDI: ffff8f686e2b6b40 [ 8096.842711] RBP: ffff8f484c64bb58 R08: ffff8f487f81b8c0 R09: 0000000000000000 [ 8096.842712] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8096.842713] R13: 0000000000000003 R14: 0000000000000013 R15: 000000005b63345b [ 8096.842714] FS: 0000000000000000(0000) GS:ffff8f487f800000(0000) knlGS:0000000000000000 [ 8096.842715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8096.842716] CR2: 00002aaaabaa0aa0 CR3: 0000003ff8218000 CR4: 00000000003607e0 [ 8096.842717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8096.842718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8096.842718] Call Trace: [ 8096.842721] [] queued_spin_lock_slowpath+0xb/0xf [ 8096.842723] [] _raw_spin_lock+0x30/0x40 [ 8096.842731] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8096.842740] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8096.842773] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8096.842806] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8096.842838] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8096.842874] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8096.842877] [] ? wake_up_state+0x20/0x20 [ 8096.842911] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8096.842913] [] kthread+0xd1/0xe0 [ 8096.842915] [] ? insert_kthread_work+0x40/0x40 [ 8096.842917] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8096.842919] [] ? insert_kthread_work+0x40/0x40 [ 8096.842941] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8097.034644] NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [ptlrpcd_00_03:16801] [ 8097.034674] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.034697] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.034699] CPU: 36 PID: 16801 Comm: ptlrpcd_00_03 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.034700] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.034701] task: ffff8f484fb7c200 ti: ffff8f484c668000 task.ti: ffff8f484c668000 [ 8097.034704] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 [ 8097.034705] RSP: 0018:ffff8f484c66bb58 EFLAGS: 00000246 [ 8097.034706] RAX: 0000000000000000 RBX: ffff8f476dda3a80 RCX: 0000000001210000 [ 8097.034707] RDX: ffff8f687ef9b8c0 RSI: 0000000001010001 RDI: ffff8f686e2b6b40 [ 8097.034707] RBP: ffff8f484c66bb58 R08: ffff8f487f89b8c0 R09: 0000000000000000 [ 8097.034708] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.034709] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000a2b8b603 [ 8097.034710] FS: 0000000000000000(0000) GS:ffff8f487f880000(0000) knlGS:0000000000000000 [ 8097.034711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.034712] CR2: 00002aaaab1139e5 CR3: 0000003ff84f6000 CR4: 00000000003607e0 [ 8097.034713] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.034714] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.034714] Call Trace: [ 8097.034717] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.034719] [] _raw_spin_lock+0x30/0x40 [ 8097.034727] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.034737] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.034769] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.034772] [] ? del_timer_sync+0x52/0x60 [ 8097.034804] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.034834] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.034869] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.034872] [] ? wake_up_state+0x20/0x20 [ 8097.034905] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.034908] [] kthread+0xd1/0xe0 [ 8097.034910] [] ? insert_kthread_work+0x40/0x40 [ 8097.034912] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.034914] [] ? insert_kthread_work+0x40/0x40 [ 8097.034934] Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 [ 8097.049644] NMI watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [ptlrpcd_00_29:16827] [ 8097.049672] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.049694] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.049697] CPU: 40 PID: 16827 Comm: ptlrpcd_00_29 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.049697] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.049698] task: ffff8f484f80a100 ti: ffff8f484f814000 task.ti: ffff8f484f814000 [ 8097.049701] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.049702] RSP: 0018:ffff8f484f817b58 EFLAGS: 00000246 [ 8097.049703] RAX: 0000000000000000 RBX: ffff8f66ea5d0900 RCX: 0000000001410000 [ 8097.049704] RDX: ffff8f487fadb8c0 RSI: 0000000001690001 RDI: ffff8f686e2b6b40 [ 8097.049705] RBP: ffff8f484f817b58 R08: ffff8f487f99b8c0 R09: 0000000000000000 [ 8097.049706] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.049706] R13: 0000000000000003 R14: 0000000000000013 R15: 000000006c28f74f [ 8097.049708] FS: 0000000000000000(0000) GS:ffff8f487f980000(0000) knlGS:0000000000000000 [ 8097.049709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.049709] CR2: 00002aaaab0fc0a0 CR3: 0000001f09af4000 CR4: 00000000003607e0 [ 8097.049710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.049711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.049711] Call Trace: [ 8097.049714] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.049716] [] _raw_spin_lock+0x30/0x40 [ 8097.049724] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.049734] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.049765] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.049797] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.049827] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.049861] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.049864] [] ? wake_up_state+0x20/0x20 [ 8097.049897] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.049899] [] kthread+0xd1/0xe0 [ 8097.049902] [] ? insert_kthread_work+0x40/0x40 [ 8097.049903] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.049905] [] ? insert_kthread_work+0x40/0x40 [ 8097.049926] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.061644] NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [ptlrpcd_00_28:16826] [ 8097.061672] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.061695] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.061697] CPU: 44 PID: 16826 Comm: ptlrpcd_00_28 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.061698] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.061699] task: ffff8f484f809080 ti: ffff8f484f810000 task.ti: ffff8f484f810000 [ 8097.061702] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.061703] RSP: 0018:ffff8f484f813b58 EFLAGS: 00000246 [ 8097.061704] RAX: 0000000000000000 RBX: ffff8f477890cc80 RCX: 0000000001610000 [ 8097.061704] RDX: ffff8f487fc1b8c0 RSI: 0000000001910001 RDI: ffff8f686e2b6b40 [ 8097.061705] RBP: ffff8f484f813b58 R08: ffff8f487fa9b8c0 R09: 0000000000000000 [ 8097.061706] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.061707] R13: 0000000000000003 R14: 0000000000000013 R15: 000000008d8972f1 [ 8097.061708] FS: 0000000000000000(0000) GS:ffff8f487fa80000(0000) knlGS:0000000000000000 [ 8097.061709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.061710] CR2: 00007ffff7ff8000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8097.061711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.061711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.061712] Call Trace: [ 8097.061714] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.061716] [] _raw_spin_lock+0x30/0x40 [ 8097.061725] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.061734] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.061766] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.061798] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.061828] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.061863] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.061865] [] ? wake_up_state+0x20/0x20 [ 8097.061898] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.061901] [] kthread+0xd1/0xe0 [ 8097.061903] [] ? insert_kthread_work+0x40/0x40 [ 8097.061905] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.061907] [] ? insert_kthread_work+0x40/0x40 [ 8097.061927] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.067643] NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [ptlrpcd_00_07:16805] [ 8097.067672] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.067694] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.067696] CPU: 46 PID: 16805 Comm: ptlrpcd_00_07 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.067697] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.067698] task: ffff8f484c679080 ti: ffff8f484c654000 task.ti: ffff8f484c654000 [ 8097.067701] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.067702] RSP: 0018:ffff8f484c657b58 EFLAGS: 00000246 [ 8097.067703] RAX: 0000000000000000 RBX: ffff8f4725823f00 RCX: 0000000001710000 [ 8097.067704] RDX: ffff8f687f1db8c0 RSI: 0000000001d90001 RDI: ffff8f686e2b6b40 [ 8097.067705] RBP: ffff8f484c657b58 R08: ffff8f487fb1b8c0 R09: 0000000000000000 [ 8097.067705] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.067706] R13: 0000000000000003 R14: 0000000000000013 R15: 00000000af1fa668 [ 8097.067707] FS: 0000000000000000(0000) GS:ffff8f487fb00000(0000) knlGS:0000000000000000 [ 8097.067708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.067709] CR2: 00002aaaab1139e5 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8097.067710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.067711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.067711] Call Trace: [ 8097.067714] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.067716] [] _raw_spin_lock+0x30/0x40 [ 8097.067724] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.067734] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.067765] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.067798] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.067828] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.067862] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.067865] [] ? wake_up_state+0x20/0x20 [ 8097.067898] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.067901] [] kthread+0xd1/0xe0 [ 8097.067903] [] ? insert_kthread_work+0x40/0x40 [ 8097.067905] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.067907] [] ? insert_kthread_work+0x40/0x40 [ 8097.067927] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.079643] NMI watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [ptlrpcd_00_20:16818] [ 8097.079672] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.079695] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.079697] CPU: 50 PID: 16818 Comm: ptlrpcd_00_20 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.079698] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.079699] task: ffff8f484d3d8000 ti: ffff8f484d3e0000 task.ti: ffff8f484d3e0000 [ 8097.079702] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.079703] RSP: 0018:ffff8f484d3e3b58 EFLAGS: 00000246 [ 8097.079703] RAX: 0000000000000000 RBX: ffff8f4818757980 RCX: 0000000001910000 [ 8097.079704] RDX: ffff8f687ec1b8c0 RSI: 0000000000910001 RDI: ffff8f686e2b6b40 [ 8097.079705] RBP: ffff8f484d3e3b58 R08: ffff8f487fc1b8c0 R09: 0000000000000000 [ 8097.079706] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.079707] R13: 0000000000000003 R14: 0000000000000013 R15: 000000000dcee365 [ 8097.079708] FS: 0000000000000000(0000) GS:ffff8f487fc00000(0000) knlGS:0000000000000000 [ 8097.079709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.079709] CR2: 00007fe199cff000 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8097.079710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.079711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.079711] Call Trace: [ 8097.079714] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.079716] [] _raw_spin_lock+0x30/0x40 [ 8097.079724] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.079734] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.079766] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.079798] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.079827] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.079861] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.079864] [] ? wake_up_state+0x20/0x20 [ 8097.079896] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.079899] [] kthread+0xd1/0xe0 [ 8097.079901] [] ? insert_kthread_work+0x40/0x40 [ 8097.079903] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.079905] [] ? insert_kthread_work+0x40/0x40 [ 8097.079925] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.082643] NMI watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [ptlrpcd_00_16:16814] [ 8097.082672] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.082694] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.082696] CPU: 51 PID: 16814 Comm: ptlrpcd_00_16 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.082697] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.082698] task: ffff8f484c623180 ti: ffff8f484c634000 task.ti: ffff8f484c634000 [ 8097.082701] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.082702] RSP: 0018:ffff8f484c637b58 EFLAGS: 00000246 [ 8097.082703] RAX: 0000000000000000 RBX: ffff8f4818be0900 RCX: 0000000001990000 [ 8097.082703] RDX: ffff8f487fa9b8c0 RSI: 0000000001610001 RDI: ffff8f686e2b6b40 [ 8097.082704] RBP: ffff8f484c637b58 R08: ffff8f487fc5b8c0 R09: 0000000000000000 [ 8097.082705] R10: 0000000000000000 R11: 000000000000000f R12: ffffffffc0e0860b [ 8097.082706] R13: 0000000000000003 R14: 0000000000000013 R15: 0000000001fe4b42 [ 8097.082707] FS: 0000000000000000(0000) GS:ffff8f487fc40000(0000) knlGS:0000000000000000 [ 8097.082708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.082709] CR2: 00002aaaaad94d70 CR3: 0000001a35610000 CR4: 00000000003607e0 [ 8097.082709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.082710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.082711] Call Trace: [ 8097.082713] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.082715] [] _raw_spin_lock+0x30/0x40 [ 8097.082723] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.082733] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.082765] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.082797] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.082826] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.082860] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.082863] [] ? wake_up_state+0x20/0x20 [ 8097.082896] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.082898] [] kthread+0xd1/0xe0 [ 8097.082900] [] ? insert_kthread_work+0x40/0x40 [ 8097.082902] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.082904] [] ? insert_kthread_work+0x40/0x40 [ 8097.082925] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.597882] NMI watchdog: Watchdog detected hard LOCKUP on cpu 28 [ 8097.597907] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) gdrdrv(POE) iTCO_wdt iTCO_vendor_support rpcrdma nvidia_drm(POE) ib_iser joydev sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass nvidia_modeset(POE) sg pcspkr i2c_i801 lpc_ich nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xt_conntrack nf_conntrack libcrc32c iptable_filter ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ib_ipoib rdma_ucm ib_umad iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 sch_fq_codel binfmt_misc msr_safe(OE) ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic [ 8097.597926] nvidia_uvm(OE) mlx5_ib ib_uverbs be2iscsi ib_core bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs 8021q garp mrp stp llc nvidia(POE) ast drm_kms_helper crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt ghash_clmulni_intel mlx5_core fb_sys_fops igb ttm aesni_intel mlxfw lrw devlink gf128mul dca glue_helper ablk_helper drm dm_multipath ptp cryptd i2c_algo_bit pps_core drm_panel_orientation_quirks wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse [ 8097.597928] CPU: 28 PID: 16850 Comm: ptlrpcd_01_15 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.597929] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.597930] task: ffff8f484fb9b180 ti: ffff8f484fbb0000 task.ti: ffff8f484fbb0000 [ 8097.597933] RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.597934] RSP: 0018:ffff8f687ee83de8 EFLAGS: 00000046 [ 8097.597935] RAX: 0000000000000000 RBX: 0000000000000087 RCX: 0000000000e30000 [ 8097.597936] RDX: ffff8f687f05b8d0 RSI: 00000000011b0101 RDI: ffffffff9107a7c0 [ 8097.597936] RBP: ffff8f687ee83de8 R08: ffff8f687ee9b8d0 R09: 0000000000000000 [ 8097.597937] R10: 0000000000042a40 R11: 0000000000100000 R12: ffffffff9107a3c0 [ 8097.597938] R13: 000000000000001c R14: ffffffff9107a5c0 R15: ffff8f484fb9b180 [ 8097.597939] FS: 0000000000000000(0000) GS:ffff8f687ee80000(0000) knlGS:0000000000000000 [ 8097.597939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8097.597940] CR2: 00002aaaabaa0aa0 CR3: 0000003f1892c000 CR4: 00000000003607e0 [ 8097.597941] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8097.597941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 8097.597942] Call Trace: [ 8097.597945] [ 8097.597945] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.597947] [] _raw_spin_lock_irqsave+0x47/0x50 [ 8097.597949] [] rcu_check_callbacks+0x589/0x770 [ 8097.597952] [] update_process_times+0x46/0x80 [ 8097.597954] [] tick_sched_handle+0x30/0x70 [ 8097.597956] [] tick_sched_timer+0x39/0x80 [ 8097.597958] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8097.597960] [] ? tick_sched_do_timer+0x50/0x50 [ 8097.597961] [] hrtimer_interrupt+0xb9/0x1f0 [ 8097.597963] [] local_apic_timer_interrupt+0x3b/0x60 [ 8097.597965] [] smp_apic_timer_interrupt+0x43/0x60 [ 8097.597966] [] apic_timer_interrupt+0x16a/0x170 [ 8097.597969] [ 8097.597969] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.597971] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.597973] [] _raw_spin_lock+0x30/0x40 [ 8097.597980] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.597989] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.598018] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.598021] [] ? del_timer_sync+0x52/0x60 [ 8097.598050] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.598077] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.598109] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.598112] [] ? wake_up_state+0x20/0x20 [ 8097.598142] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.598144] [] kthread+0xd1/0xe0 [ 8097.598147] [] ? insert_kthread_work+0x40/0x40 [ 8097.598148] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.598150] [] ? insert_kthread_work+0x40/0x40 [ 8097.598167] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 17 15 91 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b [ 8097.598169] Kernel panic - not syncing: Hard LOCKUP [ 8097.598170] CPU: 28 PID: 16850 Comm: ptlrpcd_01_15 Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1160.53.1.1chaos.ch6.x86_64 #1 [ 8097.598171] Hardware name: Penguin Computing Relion X1904GT/MG20-OP0-ZB, BIOS R04 07/31/2017 [ 8097.598171] Call Trace: [ 8097.598175] [] dump_stack+0x19/0x1b [ 8097.598177] [] panic+0xe8/0x21f [ 8097.598182] [] ? show_regs+0x58/0x290 [ 8097.598187] [] nmi_panic+0x3f/0x40 [ 8097.598192] [] watchdog_overflow_callback+0x119/0x140 [ 8097.598197] [] __perf_event_overflow+0x57/0x100 [ 8097.598199] [] perf_event_overflow+0x14/0x20 [ 8097.598203] [] handle_pmi_common+0x1a0/0x250 [ 8097.598208] [] ? ioremap_page_range+0x2e8/0x480 [ 8097.598213] [] ? vunmap_page_range+0x234/0x470 [ 8097.598215] [] ? unmap_kernel_range_noflush+0x11/0x20 [ 8097.598221] [] ? ghes_copy_tofrom_phys+0x120/0x230 [ 8097.598223] [] intel_pmu_handle_irq+0xcf/0x1d0 [ 8097.598225] [] perf_event_nmi_handler+0x31/0x50 [ 8097.598227] [] nmi_handle.isra.0+0x9c/0x170 [ 8097.598229] [] do_nmi+0x228/0x450 [ 8097.598231] [] end_repeat_nmi+0x1e/0x81 [ 8097.598233] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.598235] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.598237] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.598239] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.598241] [] _raw_spin_lock_irqsave+0x47/0x50 [ 8097.598243] [] rcu_check_callbacks+0x589/0x770 [ 8097.598245] [] update_process_times+0x46/0x80 [ 8097.598247] [] tick_sched_handle+0x30/0x70 [ 8097.598249] [] tick_sched_timer+0x39/0x80 [ 8097.598251] [] __hrtimer_run_queues+0x13e/0x2f0 [ 8097.598253] [] ? tick_sched_do_timer+0x50/0x50 [ 8097.598254] [] hrtimer_interrupt+0xb9/0x1f0 [ 8097.598256] [] local_apic_timer_interrupt+0x3b/0x60 [ 8097.598258] [] smp_apic_timer_interrupt+0x43/0x60 [ 8097.598259] [] apic_timer_interrupt+0x16a/0x170 [ 8097.598261] [] ? native_queued_spin_lock_slowpath+0x122/0x200 [ 8097.598263] [] queued_spin_lock_slowpath+0xb/0xf [ 8097.598265] [] _raw_spin_lock+0x30/0x40 [ 8097.598272] [] cfs_percpt_lock+0x58/0x110 [libcfs] [ 8097.598281] [] LNetMDUnlink+0x78/0x180 [lnet] [ 8097.598309] [] ptlrpc_unregister_reply+0x156/0x880 [ptlrpc] [ 8097.598313] [] ? del_timer_sync+0x52/0x60 [ 8097.598358] [] ptlrpc_expire_one_request+0xfe/0x550 [ptlrpc] [ 8097.598405] [] ptlrpc_expired_set+0xaf/0x1a0 [ptlrpc] [ 8097.598454] [] ptlrpcd+0x29c/0x570 [ptlrpc] [ 8097.598457] [] ? wake_up_state+0x20/0x20 [ 8097.598506] [] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [ 8097.598508] [] kthread+0xd1/0xe0 [ 8097.598511] [] ? insert_kthread_work+0x40/0x40 [ 8097.598512] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8097.598514] [] ? insert_kthread_work+0x40/0x40