-- Logs begin at Tue 2019-12-10 05:57:26 PST, end at Thu 2019-12-12 22:56:01 PST. -- Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys cpuset Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys cpu Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys cpuacct Dec 10 05:57:26 fir-md1-s1 kernel: Linux version 3.10.0-957.27.2.el7_lustre.pl2.x86_64 (sthiell@oak-rbh01) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 10 05:57:26 fir-md1-s1 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=abdfca31-9e32-4c60-981c-98bd3cab6b0a ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Dec 10 05:57:26 fir-md1-s1 kernel: e820: BIOS-provided physical RAM map: Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000100000-0x000000004f882fff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000004f883000-0x000000005788bfff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000005788c000-0x000000006cacefff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006cacf000-0x000000006efcefff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000107f37ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000001080000000-0x000000207ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000002080000000-0x000000307ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000003080000000-0x000000407ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: NX (Execute Disable) protection: active Dec 10 05:57:26 fir-md1-s1 kernel: e820: update [mem 0x3705b020-0x3708cc5f] usable ==> usable Dec 10 05:57:26 fir-md1-s1 kernel: e820: update [mem 0x37029020-0x3705ac5f] usable ==> usable Dec 10 05:57:26 fir-md1-s1 kernel: e820: update [mem 0x37020020-0x3702805f] usable ==> usable Dec 10 05:57:26 fir-md1-s1 kernel: e820: update [mem 0x37007020-0x3701f65f] usable ==> usable Dec 10 05:57:26 fir-md1-s1 kernel: extended physical RAM map: Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000000000-0x000000000008efff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000090000-0x000000000009ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000100000-0x000000003700701f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037007020-0x000000003701f65f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003701f660-0x000000003702001f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037020020-0x000000003702805f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037028060-0x000000003702901f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037029020-0x000000003705ac5f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003705ac60-0x000000003705b01f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003705b020-0x000000003708cc5f] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003708cc60-0x000000004f882fff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000004f883000-0x000000005788bfff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000005788c000-0x000000006cacefff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006cacf000-0x000000006efcefff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000100000000-0x000000107f37ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000107f380000-0x000000107fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000001080000000-0x000000207ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000207ff80000-0x000000207fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000002080000000-0x000000307ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000307ff80000-0x000000307fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000003080000000-0x000000407ff7ffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000407ff80000-0x000000407fffffff] reserved Dec 10 05:57:26 fir-md1-s1 kernel: efi: EFI v2.50 by Dell Inc. Dec 10 05:57:26 fir-md1-s1 kernel: efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x6eab5000 SMBIOS 3.0=0x6eab3000 Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem01: type=2, attr=0xf, range=[0x0000000000001000-0x0000000000002000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem02: type=7, attr=0xf, range=[0x0000000000002000-0x0000000000010000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem03: type=3, attr=0xf, range=[0x0000000000010000-0x0000000000014000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem04: type=7, attr=0xf, range=[0x0000000000014000-0x0000000000063000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem05: type=3, attr=0xf, range=[0x0000000000063000-0x000000000008f000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem06: type=10, attr=0xf, range=[0x000000000008f000-0x0000000000090000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem07: type=3, attr=0xf, range=[0x0000000000090000-0x00000000000a0000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem08: type=4, attr=0xf, range=[0x0000000000100000-0x0000000000120000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem09: type=7, attr=0xf, range=[0x0000000000120000-0x0000000000c00000) (10MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem10: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem11: type=2, attr=0xf, range=[0x0000000001000000-0x000000000267b000) (22MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem12: type=7, attr=0xf, range=[0x000000000267b000-0x0000000004000000) (25MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem13: type=4, attr=0xf, range=[0x0000000004000000-0x000000000403b000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem14: type=7, attr=0xf, range=[0x000000000403b000-0x0000000037007000) (815MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem15: type=2, attr=0xf, range=[0x0000000037007000-0x000000004eee6000) (382MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem16: type=7, attr=0xf, range=[0x000000004eee6000-0x000000004eeea000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem17: type=2, attr=0xf, range=[0x000000004eeea000-0x000000004eeec000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem18: type=1, attr=0xf, range=[0x000000004eeec000-0x000000004f009000) (1MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem19: type=2, attr=0xf, range=[0x000000004f009000-0x000000004f128000) (1MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem20: type=1, attr=0xf, range=[0x000000004f128000-0x000000004f237000) (1MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem21: type=3, attr=0xf, range=[0x000000004f237000-0x000000004f883000) (6MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem22: type=0, attr=0xf, range=[0x000000004f883000-0x000000005788c000) (128MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem23: type=3, attr=0xf, range=[0x000000005788c000-0x000000005796e000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem24: type=4, attr=0xf, range=[0x000000005796e000-0x000000005b4cf000) (59MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem25: type=3, attr=0xf, range=[0x000000005b4cf000-0x000000005b8cf000) (4MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem26: type=7, attr=0xf, range=[0x000000005b8cf000-0x0000000067b63000) (194MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem27: type=4, attr=0xf, range=[0x0000000067b63000-0x0000000067b70000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem28: type=7, attr=0xf, range=[0x0000000067b70000-0x0000000067b74000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem29: type=4, attr=0xf, range=[0x0000000067b74000-0x00000000681aa000) (6MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem30: type=7, attr=0xf, range=[0x00000000681aa000-0x00000000681ab000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem31: type=4, attr=0xf, range=[0x00000000681ab000-0x00000000681b5000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem32: type=7, attr=0xf, range=[0x00000000681b5000-0x00000000681b6000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem33: type=4, attr=0xf, range=[0x00000000681b6000-0x00000000681ba000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem34: type=7, attr=0xf, range=[0x00000000681ba000-0x00000000681bb000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem35: type=4, attr=0xf, range=[0x00000000681bb000-0x00000000681cc000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem36: type=7, attr=0xf, range=[0x00000000681cc000-0x00000000681cd000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem37: type=4, attr=0xf, range=[0x00000000681cd000-0x00000000681d2000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem38: type=7, attr=0xf, range=[0x00000000681d2000-0x00000000681d3000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem39: type=4, attr=0xf, range=[0x00000000681d3000-0x00000000681db000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem40: type=7, attr=0xf, range=[0x00000000681db000-0x00000000681dc000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem41: type=4, attr=0xf, range=[0x00000000681dc000-0x00000000681de000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem42: type=7, attr=0xf, range=[0x00000000681de000-0x00000000681df000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem43: type=4, attr=0xf, range=[0x00000000681df000-0x00000000681f0000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem44: type=7, attr=0xf, range=[0x00000000681f0000-0x00000000681f1000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem45: type=4, attr=0xf, range=[0x00000000681f1000-0x00000000681f4000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem46: type=7, attr=0xf, range=[0x00000000681f4000-0x00000000681f6000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem47: type=4, attr=0xf, range=[0x00000000681f6000-0x00000000681ff000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem48: type=7, attr=0xf, range=[0x00000000681ff000-0x0000000068200000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem49: type=4, attr=0xf, range=[0x0000000068200000-0x0000000068202000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem50: type=7, attr=0xf, range=[0x0000000068202000-0x0000000068203000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem51: type=4, attr=0xf, range=[0x0000000068203000-0x0000000068207000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem52: type=7, attr=0xf, range=[0x0000000068207000-0x0000000068208000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem53: type=4, attr=0xf, range=[0x0000000068208000-0x000000006853d000) (3MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem54: type=7, attr=0xf, range=[0x000000006853d000-0x000000006853e000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem55: type=4, attr=0xf, range=[0x000000006853e000-0x0000000068552000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem56: type=7, attr=0xf, range=[0x0000000068552000-0x0000000068554000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem57: type=4, attr=0xf, range=[0x0000000068554000-0x0000000068564000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem58: type=7, attr=0xf, range=[0x0000000068564000-0x0000000068565000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem59: type=4, attr=0xf, range=[0x0000000068565000-0x000000006857a000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem60: type=7, attr=0xf, range=[0x000000006857a000-0x000000006857b000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem61: type=4, attr=0xf, range=[0x000000006857b000-0x000000006858b000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem62: type=7, attr=0xf, range=[0x000000006858b000-0x000000006858c000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem63: type=4, attr=0xf, range=[0x000000006858c000-0x00000000685b4000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem64: type=7, attr=0xf, range=[0x00000000685b4000-0x00000000685b5000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem65: type=4, attr=0xf, range=[0x00000000685b5000-0x00000000685cf000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem66: type=7, attr=0xf, range=[0x00000000685cf000-0x00000000685d0000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem67: type=4, attr=0xf, range=[0x00000000685d0000-0x00000000685eb000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem68: type=7, attr=0xf, range=[0x00000000685eb000-0x00000000685ec000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem69: type=4, attr=0xf, range=[0x00000000685ec000-0x000000006862f000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem70: type=7, attr=0xf, range=[0x000000006862f000-0x0000000068630000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem71: type=4, attr=0xf, range=[0x0000000068630000-0x0000000068641000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem72: type=7, attr=0xf, range=[0x0000000068641000-0x0000000068643000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem73: type=4, attr=0xf, range=[0x0000000068643000-0x0000000068648000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem74: type=7, attr=0xf, range=[0x0000000068648000-0x0000000068649000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem75: type=4, attr=0xf, range=[0x0000000068649000-0x0000000068658000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem76: type=7, attr=0xf, range=[0x0000000068658000-0x0000000068659000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem77: type=4, attr=0xf, range=[0x0000000068659000-0x000000006867a000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem78: type=7, attr=0xf, range=[0x000000006867a000-0x000000006867b000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem79: type=4, attr=0xf, range=[0x000000006867b000-0x00000000686da000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem80: type=7, attr=0xf, range=[0x00000000686da000-0x00000000686db000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem81: type=4, attr=0xf, range=[0x00000000686db000-0x00000000686de000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem82: type=7, attr=0xf, range=[0x00000000686de000-0x00000000686df000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem83: type=4, attr=0xf, range=[0x00000000686df000-0x00000000686e5000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem84: type=7, attr=0xf, range=[0x00000000686e5000-0x00000000686e6000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem85: type=4, attr=0xf, range=[0x00000000686e6000-0x00000000686e8000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem86: type=7, attr=0xf, range=[0x00000000686e8000-0x00000000686e9000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem87: type=4, attr=0xf, range=[0x00000000686e9000-0x00000000686ed000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem88: type=7, attr=0xf, range=[0x00000000686ed000-0x00000000686ee000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem89: type=4, attr=0xf, range=[0x00000000686ee000-0x00000000686f6000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem90: type=7, attr=0xf, range=[0x00000000686f6000-0x00000000686f7000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem91: type=4, attr=0xf, range=[0x00000000686f7000-0x0000000068701000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem92: type=7, attr=0xf, range=[0x0000000068701000-0x0000000068702000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem93: type=4, attr=0xf, range=[0x0000000068702000-0x0000000068704000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem94: type=7, attr=0xf, range=[0x0000000068704000-0x0000000068705000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem95: type=4, attr=0xf, range=[0x0000000068705000-0x0000000068722000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem96: type=7, attr=0xf, range=[0x0000000068722000-0x0000000068723000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem97: type=4, attr=0xf, range=[0x0000000068723000-0x000000006b8cf000) (49MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem98: type=7, attr=0xf, range=[0x000000006b8cf000-0x000000006b8d0000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem99: type=3, attr=0xf, range=[0x000000006b8d0000-0x000000006cacf000) (17MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem100: type=6, attr=0x800000000000000f, range=[0x000000006cacf000-0x000000006cbcf000) (1MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem101: type=5, attr=0x800000000000000f, range=[0x000000006cbcf000-0x000000006cdcf000) (2MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem102: type=0, attr=0xf, range=[0x000000006cdcf000-0x000000006efcf000) (34MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem103: type=10, attr=0xf, range=[0x000000006efcf000-0x000000006fdff000) (14MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem104: type=9, attr=0xf, range=[0x000000006fdff000-0x000000006ffff000) (2MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem105: type=4, attr=0xf, range=[0x000000006ffff000-0x0000000070000000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem106: type=7, attr=0xf, range=[0x0000000100000000-0x000000107f380000) (63475MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem107: type=7, attr=0xf, range=[0x0000001080000000-0x000000207ff80000) (65535MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem108: type=7, attr=0xf, range=[0x0000002080000000-0x000000307ff80000) (65535MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem109: type=7, attr=0xf, range=[0x0000003080000000-0x000000407ff80000) (65535MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem110: type=0, attr=0x9, range=[0x0000000070000000-0x0000000080000000) (256MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem111: type=11, attr=0x800000000000000f, range=[0x0000000080000000-0x0000000090000000) (256MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem112: type=11, attr=0x800000000000000f, range=[0x00000000fec10000-0x00000000fec11000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem113: type=11, attr=0x800000000000000f, range=[0x00000000fed80000-0x00000000fed81000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem114: type=0, attr=0x0, range=[0x000000107f380000-0x0000001080000000) (12MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem115: type=0, attr=0x0, range=[0x000000207ff80000-0x0000002080000000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem116: type=0, attr=0x0, range=[0x000000307ff80000-0x0000003080000000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: efi: mem117: type=0, attr=0x0, range=[0x000000407ff80000-0x0000004080000000) (0MB) Dec 10 05:57:26 fir-md1-s1 kernel: SMBIOS 3.2.0 present. Dec 10 05:57:26 fir-md1-s1 kernel: DMI: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.10.6 08/15/2019 Dec 10 05:57:26 fir-md1-s1 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Dec 10 05:57:26 fir-md1-s1 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Dec 10 05:57:26 fir-md1-s1 kernel: e820: last_pfn = 0x407ff80 max_arch_pfn = 0x400000000 Dec 10 05:57:26 fir-md1-s1 kernel: MTRR default type: uncachable Dec 10 05:57:26 fir-md1-s1 kernel: MTRR fixed ranges enabled: Dec 10 05:57:26 fir-md1-s1 kernel: 00000-9FFFF write-back Dec 10 05:57:26 fir-md1-s1 kernel: A0000-FFFFF uncachable Dec 10 05:57:26 fir-md1-s1 kernel: MTRR variable ranges enabled: Dec 10 05:57:26 fir-md1-s1 kernel: 0 base 0000FF000000 mask FFFFFF000000 write-protect Dec 10 05:57:26 fir-md1-s1 kernel: 1 base 000000000000 mask FFFF80000000 write-back Dec 10 05:57:26 fir-md1-s1 kernel: 2 base 000070000000 mask FFFFF0000000 uncachable Dec 10 05:57:26 fir-md1-s1 kernel: 3 disabled Dec 10 05:57:26 fir-md1-s1 kernel: 4 disabled Dec 10 05:57:26 fir-md1-s1 kernel: 5 disabled Dec 10 05:57:26 fir-md1-s1 kernel: 6 disabled Dec 10 05:57:26 fir-md1-s1 kernel: 7 disabled Dec 10 05:57:26 fir-md1-s1 kernel: TOM2: 0000004080000000 aka 264192M Dec 10 05:57:26 fir-md1-s1 kernel: PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC Dec 10 05:57:26 fir-md1-s1 kernel: e820: last_pfn = 0x70000 max_arch_pfn = 0x400000000 Dec 10 05:57:26 fir-md1-s1 kernel: Base memory trampoline at [ffff884bc0099000] 99000 size 24576 Dec 10 05:57:26 fir-md1-s1 kernel: Using GB pages for direct mapping Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc53000, 0x318cc53fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc54000, 0x318cc54fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc55000, 0x318cc55fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc56000, 0x318cc56fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc57000, 0x318cc57fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc58000, 0x318cc58fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc59000, 0x318cc59fff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc5a000, 0x318cc5afff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc5b000, 0x318cc5bfff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc5c000, 0x318cc5cfff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc5d000, 0x318cc5dfff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: BRK [0x318cc5e000, 0x318cc5efff] PGTABLE Dec 10 05:57:26 fir-md1-s1 kernel: RAMDISK: [mem 0x3708d000-0x383d1fff] Dec 10 05:57:26 fir-md1-s1 kernel: Early table checksum verification disabled Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: RSDP 000000006fffe014 00024 (v02 DELL ) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: XSDT 000000006fffd0e8 000AC (v01 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: FACP 000000006fff0000 00114 (v06 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: DSDT 000000006ffdc000 1038C (v02 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: FACS 000000006fdd3000 00040 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SSDT 000000006fffc000 000D2 (v02 DELL PE_SC3 00000002 MSFT 04000000) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: BERT 000000006fffb000 00030 (v01 DELL BERT 00000001 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: HEST 000000006fffa000 006DC (v01 DELL HEST 00000001 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SSDT 000000006fff9000 00294 (v01 DELL PE_SC3 00000001 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SRAT 000000006fff8000 00420 (v03 DELL PE_SC3 00000001 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: MSCT 000000006fff7000 0004E (v01 DELL PE_SC3 00000000 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SLIT 000000006fff6000 0003C (v01 DELL PE_SC3 00000001 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: CRAT 000000006fff3000 02DC0 (v01 DELL PE_SC3 00000001 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: EINJ 000000006fff2000 00150 (v01 DELL PE_SC3 00000001 AMD 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SLIC 000000006fff1000 00024 (v01 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: HPET 000000006ffef000 00038 (v01 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: APIC 000000006ffee000 004B2 (v03 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: MCFG 000000006ffed000 0003C (v01 DELL PE_SC3 00000002 DELL 00000001) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SSDT 000000006ffdb000 00629 (v02 DELL xhc_port 00000001 INTL 20170119) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IVRS 000000006ffda000 00210 (v02 DELL PE_SC3 00000001 AMD 00000000) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: SSDT 000000006ffd8000 01658 (v01 AMD CPMCMN 00000001 INTL 20170119) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x01 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x02 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x03 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x04 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x05 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x08 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x09 -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0a -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0b -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0c -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0d -> Node 0 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x10 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x11 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x12 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x13 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x14 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x15 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x18 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x19 -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1a -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1b -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1c -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1d -> Node 1 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x20 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x21 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x22 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x23 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x24 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x25 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x28 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x29 -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2a -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2b -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2c -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2d -> Node 2 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x30 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x31 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x32 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x33 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x34 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x35 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x38 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x39 -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3a -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3b -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3c -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3d -> Node 3 Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00100000-0x7fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 2 PXM 2 [mem 0x2080000000-0x307fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: SRAT: Node 3 PXM 3 [mem 0x3080000000-0x407fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: NUMA: Initialized distance table, cnt=4 Dec 10 05:57:26 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: NODE_DATA(0) allocated [mem 0x107f359000-0x107f37ffff] Dec 10 05:57:26 fir-md1-s1 kernel: NODE_DATA(1) allocated [mem 0x207ff59000-0x207ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: NODE_DATA(2) allocated [mem 0x307ff59000-0x307ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: NODE_DATA(3) allocated [mem 0x407ff58000-0x407ff7efff] Dec 10 05:57:26 fir-md1-s1 kernel: Reserving 176MB of memory at 704MB for crashkernel (System RAM: 261692MB) Dec 10 05:57:26 fir-md1-s1 kernel: Zone ranges: Dec 10 05:57:26 fir-md1-s1 kernel: DMA [mem 0x00001000-0x00ffffff] Dec 10 05:57:26 fir-md1-s1 kernel: DMA32 [mem 0x01000000-0xffffffff] Dec 10 05:57:26 fir-md1-s1 kernel: Normal [mem 0x100000000-0x407ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: Movable zone start for each node Dec 10 05:57:26 fir-md1-s1 kernel: Early memory node ranges Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x00001000-0x0008efff] Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x00090000-0x0009ffff] Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x00100000-0x4f882fff] Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x5788c000-0x6cacefff] Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x6ffff000-0x6fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: node 0: [mem 0x100000000-0x107f37ffff] Dec 10 05:57:26 fir-md1-s1 kernel: node 1: [mem 0x1080000000-0x207ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: node 2: [mem 0x2080000000-0x307ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: node 3: [mem 0x3080000000-0x407ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: Initmem setup node 0 [mem 0x00001000-0x107f37ffff] Dec 10 05:57:26 fir-md1-s1 kernel: On node 0 totalpages: 16661989 Dec 10 05:57:26 fir-md1-s1 kernel: DMA zone: 64 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: DMA zone: 1126 pages reserved Dec 10 05:57:26 fir-md1-s1 kernel: DMA zone: 3998 pages, LIFO batch:0 Dec 10 05:57:26 fir-md1-s1 kernel: DMA32 zone: 6380 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: DMA32 zone: 408263 pages, LIFO batch:31 Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 253902 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 16249728 pages, LIFO batch:31 Dec 10 05:57:26 fir-md1-s1 kernel: Initmem setup node 1 [mem 0x1080000000-0x207ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: On node 1 totalpages: 16777088 Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Dec 10 05:57:26 fir-md1-s1 kernel: Initmem setup node 2 [mem 0x2080000000-0x307ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: On node 2 totalpages: 16777088 Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Dec 10 05:57:26 fir-md1-s1 kernel: Initmem setup node 3 [mem 0x3080000000-0x407ff7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: On node 3 totalpages: 16777088 Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Dec 10 05:57:26 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PM-Timer IO Port: 0x408 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x30] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x18] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x28] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x38] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x02] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x12] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x22] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0a] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2a] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3a] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x04] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x14] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x24] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x34] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x0c] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x1c] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2c] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x3c] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x01] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x11] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x21] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x31] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x09] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x19] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x29] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x39] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0x03] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0x13] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0x23] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0x33] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0x0b] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0x1b] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0x2b] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0x3b] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0x05] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0x15] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x25] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x35] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x0d] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x1d] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2d] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x3d] enabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x40] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x41] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x42] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x43] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x44] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x45] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x46] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x47] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x48] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x49] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4a] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4b] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4c] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4d] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4e] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4f] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x50] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x51] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x52] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x53] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x54] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x55] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x56] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x57] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x58] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x59] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5a] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5b] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5c] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5d] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5e] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5f] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x60] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x61] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x62] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x63] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x64] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x65] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x66] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x67] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x68] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x69] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6a] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6b] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6c] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6d] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6e] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6f] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x70] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x71] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x72] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x73] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x74] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x75] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x76] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x77] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x78] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x79] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7a] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7b] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7c] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7d] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7e] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7f] lapic_id[0x00] disabled) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x80] address[0xfec00000] gsi_base[0]) Dec 10 05:57:26 fir-md1-s1 kernel: IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x81] address[0xfd880000] gsi_base[24]) Dec 10 05:57:26 fir-md1-s1 kernel: IOAPIC[1]: apic_id 129, version 33, address 0xfd880000, GSI 24-55 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x82] address[0xe0900000] gsi_base[56]) Dec 10 05:57:26 fir-md1-s1 kernel: IOAPIC[2]: apic_id 130, version 33, address 0xe0900000, GSI 56-87 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x83] address[0xc5900000] gsi_base[88]) Dec 10 05:57:26 fir-md1-s1 kernel: IOAPIC[3]: apic_id 131, version 33, address 0xc5900000, GSI 88-119 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x84] address[0xaa900000] gsi_base[120]) Dec 10 05:57:26 fir-md1-s1 kernel: IOAPIC[4]: apic_id 132, version 33, address 0xaa900000, GSI 120-151 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IRQ0 used by override. Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: IRQ9 used by override. Dec 10 05:57:26 fir-md1-s1 kernel: Using ACPI (MADT) for SMP configuration information Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: HPET id: 0x10228201 base: 0xfed00000 Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Allowing 128 CPUs, 80 hotplug CPUs Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x0008f000-0x0008ffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37007000-0x37007fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3701f000-0x3701ffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37020000-0x37020fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37028000-0x37028fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37029000-0x37029fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3705a000-0x3705afff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3705b000-0x3705bfff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3708c000-0x3708cfff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x4f883000-0x5788bfff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6cacf000-0x6efcefff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6efcf000-0x6fdfefff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6fdff000-0x6fffefff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x70000000-0x8fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x90000000-0xfec0ffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec10000-0xfec10fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec11000-0xfed7ffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed80000-0xfed80fff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed81000-0xffffffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x107f380000-0x107fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x207ff80000-0x207fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x307ff80000-0x307fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: [mem 0x90000000-0xfec0ffff] available for PCI devices Dec 10 05:57:26 fir-md1-s1 kernel: Booting paravirtualized kernel on bare hardware Dec 10 05:57:26 fir-md1-s1 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:4 Dec 10 05:57:26 fir-md1-s1 kernel: PERCPU: Embedded 38 pages/cpu @ffff885bfee00000 s118784 r8192 d28672 u262144 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [0] 000 004 008 012 016 020 024 028 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [0] 032 036 040 044 048 052 056 060 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [0] 064 068 072 076 080 084 088 092 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [0] 096 100 104 108 112 116 120 124 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [1] 001 005 009 013 017 021 025 029 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [1] 033 037 041 045 049 053 057 061 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [1] 065 069 073 077 081 085 089 093 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [1] 097 101 105 109 113 117 121 125 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [2] 002 006 010 014 018 022 026 030 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [2] 034 038 042 046 050 054 058 062 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [2] 066 070 074 078 082 086 090 094 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [2] 098 102 106 110 114 118 122 126 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [3] 003 007 011 015 019 023 027 031 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [3] 035 039 043 047 051 055 059 063 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [3] 067 071 075 079 083 087 091 095 Dec 10 05:57:26 fir-md1-s1 kernel: pcpu-alloc: [3] 099 103 107 111 115 119 123 127 Dec 10 05:57:26 fir-md1-s1 kernel: Built 4 zonelists in Zone order, mobility grouping on. Total pages: 65945355 Dec 10 05:57:26 fir-md1-s1 kernel: Policy zone: Normal Dec 10 05:57:26 fir-md1-s1 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=abdfca31-9e32-4c60-981c-98bd3cab6b0a ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Dec 10 05:57:26 fir-md1-s1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 Dec 10 05:57:26 fir-md1-s1 kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form Dec 10 05:57:26 fir-md1-s1 kernel: Memory: 9613428k/270532096k available (7676k kernel code, 2559084k absent, 4654532k reserved, 6045k data, 1876k init) Dec 10 05:57:26 fir-md1-s1 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=4 Dec 10 05:57:26 fir-md1-s1 kernel: Hierarchical RCU implementation. Dec 10 05:57:26 fir-md1-s1 kernel: RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. Dec 10 05:57:26 fir-md1-s1 kernel: NR_IRQS:327936 nr_irqs:3624 0 Dec 10 05:57:26 fir-md1-s1 kernel: Console: colour dummy device 80x25 Dec 10 05:57:26 fir-md1-s1 kernel: console [ttyS0] enabled Dec 10 05:57:26 fir-md1-s1 kernel: allocated 1072693248 bytes of page_cgroup Dec 10 05:57:26 fir-md1-s1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Dec 10 05:57:26 fir-md1-s1 kernel: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl Dec 10 05:57:26 fir-md1-s1 kernel: hpet clockevent registered Dec 10 05:57:26 fir-md1-s1 kernel: tsc: Fast TSC calibration using PIT Dec 10 05:57:26 fir-md1-s1 kernel: tsc: Detected 1996.233 MHz processor Dec 10 05:57:26 fir-md1-s1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3992.46 BogoMIPS (lpj=1996233) Dec 10 05:57:26 fir-md1-s1 kernel: pid_max: default: 131072 minimum: 1024 Dec 10 05:57:26 fir-md1-s1 kernel: Security Framework initialized Dec 10 05:57:26 fir-md1-s1 kernel: SELinux: Initializing. Dec 10 05:57:26 fir-md1-s1 kernel: SELinux: Starting in permissive mode Dec 10 05:57:26 fir-md1-s1 kernel: Yama: becoming mindful. Dec 10 05:57:26 fir-md1-s1 kernel: Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys memory Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys devices Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys freezer Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys net_cls Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys blkio Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys perf_event Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys hugetlb Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys pids Dec 10 05:57:26 fir-md1-s1 kernel: Initializing cgroup subsys net_prio Dec 10 05:57:26 fir-md1-s1 kernel: tseg: 0070000000 Dec 10 05:57:26 fir-md1-s1 kernel: LVT offset 2 assigned for vector 0xf4 Dec 10 05:57:26 fir-md1-s1 kernel: Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 Dec 10 05:57:26 fir-md1-s1 kernel: Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768 Dec 10 05:57:26 fir-md1-s1 kernel: tlb_flushall_shift: 6 Dec 10 05:57:26 fir-md1-s1 kernel: Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp Dec 10 05:57:26 fir-md1-s1 kernel: FEATURE SPEC_CTRL Not Present Dec 10 05:57:26 fir-md1-s1 kernel: FEATURE IBPB_SUPPORT Present Dec 10 05:57:26 fir-md1-s1 kernel: Spectre V2 : Enabling Indirect Branch Prediction Barrier Dec 10 05:57:26 fir-md1-s1 kernel: Spectre V2 : Mitigation: Full retpoline Dec 10 05:57:26 fir-md1-s1 kernel: Freeing SMP alternatives: 28k freed Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Core revision 20130517 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: All ACPI Tables successfully acquired Dec 10 05:57:26 fir-md1-s1 kernel: ftrace: allocating 29216 entries in 115 pages Dec 10 05:57:26 fir-md1-s1 kernel: Switched APIC routing to physical flat. Dec 10 05:57:26 fir-md1-s1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: CPU0: AMD EPYC 7401P 24-Core Processor (fam: 17, model: 01, stepping: 02) Dec 10 05:57:26 fir-md1-s1 kernel: random: fast init done Dec 10 05:57:26 fir-md1-s1 kernel: APIC calibration not consistent with PM-Timer: 101ms instead of 100ms Dec 10 05:57:26 fir-md1-s1 kernel: APIC delta adjusted to PM-Timer: 623827 (636297) Dec 10 05:57:26 fir-md1-s1 kernel: Performance Events: Fam17h core perfctr, AMD PMU driver. Dec 10 05:57:26 fir-md1-s1 kernel: ... version: 0 Dec 10 05:57:26 fir-md1-s1 kernel: ... bit width: 48 Dec 10 05:57:26 fir-md1-s1 kernel: ... generic registers: 6 Dec 10 05:57:26 fir-md1-s1 kernel: ... value mask: 0000ffffffffffff Dec 10 05:57:26 fir-md1-s1 kernel: ... max period: 00007fffffffffff Dec 10 05:57:26 fir-md1-s1 kernel: ... fixed-purpose events: 0 Dec 10 05:57:26 fir-md1-s1 kernel: ... event mask: 000000000000003f Dec 10 05:57:26 fir-md1-s1 kernel: NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #1 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #2 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #3 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #4 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #5 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #6 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #7 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #8 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #9 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #10 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #11 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #12 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #13 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #14 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #15 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #16 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #17 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #18 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #19 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #20 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #21 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #22 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #23 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #24 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #25 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #26 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #27 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #28 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #29 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #30 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #31 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #32 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #33 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #34 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #35 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #36 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #37 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #38 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #39 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #40 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #41 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #42 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #43 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #44 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #45 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #46 OK Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #47 Dec 10 05:57:26 fir-md1-s1 kernel: Brought up 48 CPUs Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Max logical packages: 3 Dec 10 05:57:26 fir-md1-s1 kernel: smpboot: Total of 48 processors activated (191638.36 BogoMIPS) Dec 10 05:57:26 fir-md1-s1 kernel: node 0 initialised, 15462980 pages in 274ms Dec 10 05:57:26 fir-md1-s1 kernel: node 1 initialised, 15989367 pages in 278ms Dec 10 05:57:26 fir-md1-s1 kernel: node 2 initialised, 15989367 pages in 279ms Dec 10 05:57:26 fir-md1-s1 kernel: node 3 initialised, 15984547 pages in 287ms Dec 10 05:57:26 fir-md1-s1 kernel: devtmpfs: initialized Dec 10 05:57:26 fir-md1-s1 kernel: EVM: security.selinux Dec 10 05:57:26 fir-md1-s1 kernel: EVM: security.ima Dec 10 05:57:26 fir-md1-s1 kernel: EVM: security.capability Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x6efcf000-0x6fdfefff] (14876672 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: atomic64 test passed for x86-64 platform with CX8 and with SSE Dec 10 05:57:26 fir-md1-s1 kernel: pinctrl core: initialized pinctrl subsystem Dec 10 05:57:26 fir-md1-s1 kernel: RTC time: 13:57:21, date: 12/10/19 Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 16 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: bus type PCI registered Dec 10 05:57:26 fir-md1-s1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Dec 10 05:57:26 fir-md1-s1 kernel: PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) Dec 10 05:57:26 fir-md1-s1 kernel: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 Dec 10 05:57:26 fir-md1-s1 kernel: PCI: Using configuration type 1 for base access Dec 10 05:57:26 fir-md1-s1 kernel: PCI: Dell System detected, enabling pci=bfsort. Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Added _OSI(Module Device) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Device) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Added _OSI(3.0 _SCP Extensions) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Aggregator Device) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Added _OSI(Linux-Dell-Video) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: EC: Look up EC in DSDT Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Executed 2 blocks of module-level executable AML code Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Interpreter enabled Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: (supports S0 S5) Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Using IOAPIC for interrupt routing Dec 10 05:57:26 fir-md1-s1 kernel: HEST: Table parsing has been initialized. Dec 10 05:57:26 fir-md1-s1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Enabled 1 GPEs in block 00 to 1F Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-3f]) Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:00: PCIe AER handled by firmware Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration Dec 10 05:57:26 fir-md1-s1 kernel: PCI host bridge to bus 0000:00 Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0d00-0x3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0xe1000000-0xfebfffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x10000000000-0x2bf3fffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [bus 00-3f] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:00.0: [1022:1450] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:00.2: [1022:1451] type 00 class 0x080600 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:01.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:02.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: [1022:1453] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:04.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.0: [1022:1460] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.1: [1022:1461] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.2: [1022:1462] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.3: [1022:1463] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.4: [1022:1464] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.5: [1022:1465] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.6: [1022:1466] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:18.7: [1022:1467] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.0: [1022:1460] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.1: [1022:1461] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.2: [1022:1462] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.3: [1022:1463] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.4: [1022:1464] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.5: [1022:1465] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.6: [1022:1466] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:19.7: [1022:1467] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.0: [1022:1460] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.1: [1022:1461] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.2: [1022:1462] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.3: [1022:1463] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.4: [1022:1464] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.5: [1022:1465] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.6: [1022:1466] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1a.7: [1022:1467] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.0: [1022:1460] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.1: [1022:1461] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.2: [1022:1462] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.3: [1022:1463] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.4: [1022:1464] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.5: [1022:1465] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.6: [1022:1466] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:1b.7: [1022:1467] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: [15b3:101b] type 00 class 0x020700 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x10: [mem 0xe2000000-0xe3ffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: PME# supported from D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.0: [1022:145a] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.2: [1022:1456] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x18: [mem 0xf7300000-0xf73fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x24: [mem 0xf7400000-0xf7401fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.3: [1022:145f] type 00 class 0x0c0330 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.3: reg 0x10: [mem 0xf7200000-0xf72fffff 64bit] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.0: [1022:1455] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.1: [1022:1468] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x18: [mem 0xf7000000-0xf70fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x24: [mem 0xf7100000-0xf7101fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: on NUMA node 0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 40-7f]) Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:01: PCIe AER handled by firmware Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration Dec 10 05:57:26 fir-md1-s1 kernel: PCI host bridge to bus 0000:40 Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0xc6000000-0xe0ffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0x2bf40000000-0x47e7fffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [bus 40-7f] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:00.0: [1022:1450] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:00.2: [1022:1451] type 00 class 0x080600 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:01.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:02.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:03.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:04.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.0: [1022:145a] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.2: [1022:1456] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x18: [mem 0xdb300000-0xdb3fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x24: [mem 0xdb400000-0xdb401fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.3: [1022:145f] type 00 class 0x0c0330 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.3: reg 0x10: [mem 0xdb200000-0xdb2fffff 64bit] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.3: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.0: [1022:1455] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.1: [1022:1468] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x18: [mem 0xdb000000-0xdb0fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x24: [mem 0xdb100000-0xdb101fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: on NUMA node 1 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 80-bf]) Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:02: PCIe AER handled by firmware Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration Dec 10 05:57:26 fir-md1-s1 kernel: PCI host bridge to bus 0000:80 Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0xab000000-0xc5ffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x47e80000000-0x63dbfffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [bus 80-bf] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:00.0: [1022:1450] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:00.2: [1022:1451] type 00 class 0x080600 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: [1022:1453] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: [1022:1453] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:02.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: [1022:1453] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:04.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: [14e4:165f] type 00 class 0x020000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x10: [mem 0xac230000-0xac23ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x18: [mem 0xac240000-0xac24ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x20: [mem 0xac250000-0xac25ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: [14e4:165f] type 00 class 0x020000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x10: [mem 0xac200000-0xac20ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x18: [mem 0xac210000-0xac21ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x20: [mem 0xac220000-0xac22ffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: [1556:be00] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: [102b:0536] type 00 class 0x030000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x10: [mem 0xab000000-0xabffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x14: [mem 0xc0808000-0xc080bfff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x18: [mem 0xc0000000-0xc07fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: [1000:00d1] type 00 class 0x010700 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x10: [mem 0xac000000-0xac0fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x18: [mem 0xac100000-0xac1fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x20: [mem 0xc0d00000-0xc0dfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x24: [io 0x8000-0x80ff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: supports D1 D2 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.0: [1022:145a] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.2: [1022:1456] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x18: [mem 0xc0b00000-0xc0bfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x24: [mem 0xc0c00000-0xc0c01fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.0: [1022:1455] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.1: [1022:1468] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x18: [mem 0xc0900000-0xc09fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x24: [mem 0xc0a00000-0xc0a01fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.2: [1022:7901] type 00 class 0x010601 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.2: reg 0x24: [mem 0xc0a02000-0xc0a02fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.2: PME# supported from D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: on NUMA node 2 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC03] (domain 0000 [bus c0-ff]) Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: PCIe AER handled by firmware Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration Dec 10 05:57:26 fir-md1-s1 kernel: acpi PNP0A08:03: host bridge window [mem 0x63dc0000000-0xffffffffffff window] ([0x80000000000-0xffffffffffff] ignored, not CPU addressable) Dec 10 05:57:26 fir-md1-s1 kernel: PCI host bridge to bus 0000:c0 Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x90000000-0xaaffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x63dc0000000-0x7ffffffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [bus c0-ff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:00.0: [1022:1450] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:00.2: [1022:1451] type 00 class 0x080600 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: [1022:1453] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:02.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:03.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:04.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.0: [1022:1452] type 00 class 0x060000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: [1022:1454] type 01 class 0x060400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: [1000:005f] type 00 class 0x010400 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x10: [io 0xc000-0xc0ff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x14: [mem 0xa5500000-0xa550ffff 64bit] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x1c: [mem 0xa5400000-0xa54fffff 64bit] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: supports D1 D2 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.0: [1022:145a] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.2: [1022:1456] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x18: [mem 0xa5200000-0xa52fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x24: [mem 0xa5300000-0xa5301fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.0: [1022:1455] type 00 class 0x130000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.1: [1022:1468] type 00 class 0x108000 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x18: [mem 0xa5000000-0xa50fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x24: [mem 0xa5100000-0xa5101fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: on NUMA node 3 Dec 10 05:57:26 fir-md1-s1 kernel: vgaarb: device added: PCI:0000:83:00.0,decodes=io+mem,owns=io+mem,locks=none Dec 10 05:57:26 fir-md1-s1 kernel: vgaarb: loaded Dec 10 05:57:26 fir-md1-s1 kernel: vgaarb: bridge control possible 0000:83:00.0 Dec 10 05:57:26 fir-md1-s1 kernel: SCSI subsystem initialized Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: bus type USB registered Dec 10 05:57:26 fir-md1-s1 kernel: usbcore: registered new interface driver usbfs Dec 10 05:57:26 fir-md1-s1 kernel: usbcore: registered new interface driver hub Dec 10 05:57:26 fir-md1-s1 kernel: usbcore: registered new device driver usb Dec 10 05:57:26 fir-md1-s1 kernel: EDAC MC: Ver: 3.0.0 Dec 10 05:57:26 fir-md1-s1 kernel: PCI: Using ACPI for IRQ routing Dec 10 05:57:26 fir-md1-s1 kernel: PCI: pci_cache_line_size set to 64 bytes Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x37007020-0x37ffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x37020020-0x37ffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x37029020-0x37ffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x3705b020-0x37ffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x4f883000-0x4fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x6cacf000-0x6fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x107f380000-0x107fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x207ff80000-0x207fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x307ff80000-0x307fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x407ff80000-0x407fffffff] Dec 10 05:57:26 fir-md1-s1 kernel: NetLabel: Initializing Dec 10 05:57:26 fir-md1-s1 kernel: NetLabel: domain hash size = 128 Dec 10 05:57:26 fir-md1-s1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Dec 10 05:57:26 fir-md1-s1 kernel: NetLabel: unlabeled traffic allowed by default Dec 10 05:57:26 fir-md1-s1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Dec 10 05:57:26 fir-md1-s1 kernel: hpet0: 3 comparators, 32-bit 14.318180 MHz counter Dec 10 05:57:26 fir-md1-s1 kernel: Switched to clocksource hpet Dec 10 05:57:26 fir-md1-s1 kernel: pnp: PnP ACPI init Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: bus type PNP registered Dec 10 05:57:26 fir-md1-s1 kernel: system 00:00: [mem 0x80000000-0x8fffffff] has been reserved Dec 10 05:57:26 fir-md1-s1 kernel: system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) Dec 10 05:57:26 fir-md1-s1 kernel: pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) Dec 10 05:57:26 fir-md1-s1 kernel: pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) Dec 10 05:57:26 fir-md1-s1 kernel: pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) Dec 10 05:57:26 fir-md1-s1 kernel: pnp: PnP ACPI: found 4 devices Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: bus type PNP unregistered Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: BAR 14: assigned [mem 0xe1000000-0xe10fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: BAR 6: assigned [mem 0xe1000000-0xe10fffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe10fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 10 [mem 0x000d0000-0x000d3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 11 [mem 0x000d4000-0x000d7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 12 [mem 0x000d8000-0x000dbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 13 [mem 0x000dc000-0x000dffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 14 [mem 0x000e0000-0x000e3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 18 [mem 0x000f0000-0x000fffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 19 [io 0x0d00-0x3fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 20 [mem 0xe1000000-0xfebfffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:00: resource 21 [mem 0x10000000000-0x2bf3fffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:01: resource 1 [mem 0xe1000000-0xe10fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:01: resource 2 [mem 0xe2000000-0xe3ffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:02: resource 1 [mem 0xf7200000-0xf74fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:03: resource 1 [mem 0xf7000000-0xf71fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: resource 4 [io 0x4000-0x7fff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: resource 5 [mem 0xc6000000-0xe0ffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:40: resource 6 [mem 0x2bf40000000-0x47e7fffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:41: resource 1 [mem 0xdb200000-0xdb4fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:42: resource 1 [mem 0xdb000000-0xdb1fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: BAR 14: assigned [mem 0xac300000-0xac3fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: BAR 6: assigned [mem 0xac300000-0xac33ffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: BAR 6: assigned [mem 0xac340000-0xac37ffff pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac300000-0xac3fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: BAR 6: no space for [mem size 0x00040000 pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: BAR 6: failed to assign [mem size 0x00040000 pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: resource 5 [mem 0x000a0000-0x000bffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: resource 6 [io 0x8000-0xbfff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: resource 7 [mem 0xab000000-0xc5ffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:80: resource 8 [mem 0x47e80000000-0x63dbfffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:81: resource 1 [mem 0xac300000-0xac3fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:81: resource 2 [mem 0xac200000-0xac2fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:82: resource 1 [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:82: resource 2 [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:83: resource 1 [mem 0xc0000000-0xc08fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:83: resource 2 [mem 0xab000000-0xabffffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:84: resource 0 [io 0x8000-0x8fff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:84: resource 1 [mem 0xc0d00000-0xc0dfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:84: resource 2 [mem 0xac000000-0xac1fffff 64bit pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:85: resource 1 [mem 0xc0b00000-0xc0cfffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:86: resource 1 [mem 0xc0900000-0xc0afffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: no space for [mem size 0x00100000 pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: resource 5 [mem 0x90000000-0xaaffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c0: resource 6 [mem 0x63dc0000000-0x7ffffffffff window] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c1: resource 0 [io 0xc000-0xcfff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c1: resource 1 [mem 0xa5400000-0xa55fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c2: resource 1 [mem 0xa5200000-0xa53fffff] Dec 10 05:57:26 fir-md1-s1 kernel: pci_bus 0000:c3: resource 1 [mem 0xa5000000-0xa51fffff] Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 2 Dec 10 05:57:26 fir-md1-s1 kernel: TCP established hash table entries: 524288 (order: 10, 4194304 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: TCP: Hash tables configured (established 524288 bind 65536) Dec 10 05:57:26 fir-md1-s1 kernel: TCP: reno registered Dec 10 05:57:26 fir-md1-s1 kernel: UDP hash table entries: 65536 (order: 9, 2097152 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 1 Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: Boot video device Dec 10 05:57:26 fir-md1-s1 kernel: PCI: CLS 64 bytes, default 64 Dec 10 05:57:26 fir-md1-s1 kernel: Unpacking initramfs... Dec 10 05:57:26 fir-md1-s1 kernel: Freeing initrd memory: 19732k freed Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:01.0 to group 0 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:02.0 to group 1 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.0 to group 2 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.1 to group 3 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:04.0 to group 4 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.0 to group 5 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.1 to group 6 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.0 to group 7 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.1 to group 8 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.0 to group 9 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.3 to group 9 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.0 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.1 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.2 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.3 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.4 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.5 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.6 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.7 to group 10 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.0 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.1 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.2 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.3 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.4 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.5 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.6 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.7 to group 11 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.0 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.1 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.2 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.3 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.4 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.5 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.6 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.7 to group 12 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.0 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.1 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.2 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.3 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.4 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.5 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.6 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.7 to group 13 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:01:00.0 to group 14 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.0 to group 15 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.2 to group 16 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.3 to group 17 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.0 to group 18 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.1 to group 19 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:01.0 to group 20 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:02.0 to group 21 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:03.0 to group 22 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:04.0 to group 23 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.0 to group 24 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.1 to group 25 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.0 to group 26 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.1 to group 27 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.0 to group 28 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.2 to group 29 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.3 to group 30 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.0 to group 31 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.1 to group 32 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.0 to group 33 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.1 to group 34 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.2 to group 35 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:02.0 to group 36 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.0 to group 37 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.1 to group 38 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:04.0 to group 39 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.0 to group 40 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.1 to group 41 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.0 to group 42 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.1 to group 43 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.0 to group 44 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.1 to group 44 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:82:00.0 to group 45 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:83:00.0 to group 45 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:84:00.0 to group 46 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.0 to group 47 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.2 to group 48 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.0 to group 49 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.1 to group 50 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.2 to group 51 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.0 to group 52 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.1 to group 53 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:02.0 to group 54 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:03.0 to group 55 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:04.0 to group 56 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.0 to group 57 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.1 to group 58 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.0 to group 59 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.1 to group 60 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c1:00.0 to group 61 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.0 to group 62 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.2 to group 63 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.0 to group 64 Dec 10 05:57:26 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.1 to group 65 Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Dec 10 05:57:26 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Dec 10 05:57:26 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:80:00.2 cap 0x40 Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Dec 10 05:57:26 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:c0:00.2 cap 0x40 Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Dec 10 05:57:26 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Interrupt remapping enabled Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: virtual APIC enabled Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:00:00.2: irq 26 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:40:00.2: irq 27 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:80:00.2: irq 28 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c0:00.2: irq 29 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: AMD-Vi: Lazy IO/TLB flushing enabled Dec 10 05:57:26 fir-md1-s1 kernel: perf: AMD NB counters detected Dec 10 05:57:26 fir-md1-s1 kernel: perf: AMD LLC counters detected Dec 10 05:57:26 fir-md1-s1 kernel: sha1_ssse3: Using SHA-NI optimized SHA-1 implementation Dec 10 05:57:26 fir-md1-s1 kernel: sha256_ssse3: Using SHA-256-NI optimized SHA-256 implementation Dec 10 05:57:26 fir-md1-s1 kernel: futex hash table entries: 32768 (order: 9, 2097152 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: Initialise system trusted keyring Dec 10 05:57:26 fir-md1-s1 kernel: audit: initializing netlink socket (disabled) Dec 10 05:57:26 fir-md1-s1 kernel: type=2000 audit(1575986239.206:1): initialized Dec 10 05:57:26 fir-md1-s1 kernel: HugeTLB registered 1 GB page size, pre-allocated 0 pages Dec 10 05:57:26 fir-md1-s1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Dec 10 05:57:26 fir-md1-s1 kernel: zpool: loaded Dec 10 05:57:26 fir-md1-s1 kernel: zbud: loaded Dec 10 05:57:26 fir-md1-s1 kernel: VFS: Disk quotas dquot_6.6.0 Dec 10 05:57:26 fir-md1-s1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Dec 10 05:57:26 fir-md1-s1 kernel: msgmni has been set to 32768 Dec 10 05:57:26 fir-md1-s1 kernel: Key type big_key registered Dec 10 05:57:26 fir-md1-s1 kernel: SELinux: Registering netfilter hooks Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 38 Dec 10 05:57:26 fir-md1-s1 kernel: Key type asymmetric registered Dec 10 05:57:26 fir-md1-s1 kernel: Asymmetric key parser 'x509' registered Dec 10 05:57:26 fir-md1-s1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) Dec 10 05:57:26 fir-md1-s1 kernel: io scheduler noop registered Dec 10 05:57:26 fir-md1-s1 kernel: io scheduler deadline registered (default) Dec 10 05:57:26 fir-md1-s1 kernel: io scheduler cfq registered Dec 10 05:57:26 fir-md1-s1 kernel: io scheduler mq-deadline registered Dec 10 05:57:26 fir-md1-s1 kernel: io scheduler kyber registered Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:03.1: irq 30 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:07.1: irq 31 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:08.1: irq 33 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:40:07.1: irq 34 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:40:08.1: irq 36 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:01.1: irq 37 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:01.2: irq 38 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:03.1: irq 39 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:07.1: irq 41 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:08.1: irq 43 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:01.1: irq 44 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:07.1: irq 46 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:08.1: irq 48 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:01:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:07.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:02:00.3: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:00:07.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:00:08.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:03:00.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:00:08.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:40:07.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:41:00.3: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:40:07.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:40:08.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:42:00.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:40:08.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:01.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:81:00.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:80:01.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:01.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:82:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:83:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:80:01.2:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:03.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:84:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:80:03.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:07.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:85:00.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:80:07.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:80:08.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:86:00.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:80:08.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:01.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c1:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:c0:01.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:07.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c2:00.2: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:c0:07.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pcieport 0000:c0:08.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.0: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pci 0000:c3:00.1: Signaling PME through PCIe PME interrupt Dec 10 05:57:26 fir-md1-s1 kernel: pcie_pme 0000:c0:08.1:pcie001: service driver pcie_pme loaded Dec 10 05:57:26 fir-md1-s1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Dec 10 05:57:26 fir-md1-s1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Dec 10 05:57:26 fir-md1-s1 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Dec 10 05:57:26 fir-md1-s1 kernel: efifb: probing for efifb Dec 10 05:57:26 fir-md1-s1 kernel: efifb: framebuffer at 0xab000000, mapped to 0xffff9e8759800000, using 3072k, total 3072k Dec 10 05:57:26 fir-md1-s1 kernel: efifb: mode is 1024x768x32, linelength=4096, pages=1 Dec 10 05:57:26 fir-md1-s1 kernel: efifb: scrolling: redraw Dec 10 05:57:26 fir-md1-s1 kernel: efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 Dec 10 05:57:26 fir-md1-s1 kernel: Console: switching to colour frame buffer device 128x48 Dec 10 05:57:26 fir-md1-s1 kernel: fb0: EFI VGA frame buffer device Dec 10 05:57:26 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Power Button [PWRB] Dec 10 05:57:26 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 Dec 10 05:57:26 fir-md1-s1 kernel: ACPI: Power Button [PWRF] Dec 10 05:57:26 fir-md1-s1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. Dec 10 05:57:26 fir-md1-s1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Dec 10 05:57:26 fir-md1-s1 kernel: 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Dec 10 05:57:26 fir-md1-s1 kernel: 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Dec 10 05:57:26 fir-md1-s1 kernel: Non-volatile memory driver v1.3 Dec 10 05:57:26 fir-md1-s1 kernel: Linux agpgart interface v0.103 Dec 10 05:57:26 fir-md1-s1 kernel: crash memory driver: version 1.1 Dec 10 05:57:26 fir-md1-s1 kernel: rdac: device handler registered Dec 10 05:57:26 fir-md1-s1 kernel: hp_sw: device handler registered Dec 10 05:57:26 fir-md1-s1 kernel: emc: device handler registered Dec 10 05:57:26 fir-md1-s1 kernel: alua: device handler registered Dec 10 05:57:26 fir-md1-s1 kernel: libphy: Fixed MDIO Bus: probed Dec 10 05:57:26 fir-md1-s1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Dec 10 05:57:26 fir-md1-s1 kernel: ehci-pci: EHCI PCI platform driver Dec 10 05:57:26 fir-md1-s1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Dec 10 05:57:26 fir-md1-s1 kernel: ohci-pci: OHCI PCI platform driver Dec 10 05:57:26 fir-md1-s1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 1 Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 50 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 51 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 52 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 53 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 54 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 55 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 56 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 57 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb1: Product: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: usb usb1: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb usb1: SerialNumber: 0000:02:00.3 Dec 10 05:57:26 fir-md1-s1 kernel: hub 1-0:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 1-0:1.0: 2 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 2 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: Product: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb usb2: SerialNumber: 0000:02:00.3 Dec 10 05:57:26 fir-md1-s1 kernel: hub 2-0:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 2-0:1.0: 2 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 3 Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 59 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 60 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 61 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 62 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 63 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 64 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 65 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 66 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb3: Product: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: usb usb3: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb usb3: SerialNumber: 0000:41:00.3 Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-0:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-0:1.0: 2 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 4 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: Product: xHCI Host Controller Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb usb4: SerialNumber: 0000:41:00.3 Dec 10 05:57:26 fir-md1-s1 kernel: hub 4-0:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 4-0:1.0: 2 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: usbcore: registered new interface driver usbserial_generic Dec 10 05:57:26 fir-md1-s1 kernel: usbserial: USB Serial support registered for generic Dec 10 05:57:26 fir-md1-s1 kernel: i8042: PNP: No PS/2 controller found. Probing ports directly. Dec 10 05:57:26 fir-md1-s1 kernel: usb 1-1: new high-speed USB device number 2 using xhci_hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1: new high-speed USB device number 2 using xhci_hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb 1-1: New USB device found, idVendor=0424, idProduct=2744 Dec 10 05:57:26 fir-md1-s1 kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Dec 10 05:57:26 fir-md1-s1 kernel: usb 1-1: Product: USB2734 Dec 10 05:57:26 fir-md1-s1 kernel: usb 1-1: Manufacturer: Microchip Tech Dec 10 05:57:26 fir-md1-s1 kernel: hub 1-1:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 1-1:1.0: 4 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1: New USB device found, idVendor=1604, idProduct=10c0 Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Dec 10 05:57:26 fir-md1-s1 kernel: usb 2-1: New USB device found, idVendor=0424, idProduct=5744 Dec 10 05:57:26 fir-md1-s1 kernel: usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0 Dec 10 05:57:26 fir-md1-s1 kernel: usb 2-1: Product: USB5734 Dec 10 05:57:26 fir-md1-s1 kernel: usb 2-1: Manufacturer: Microchip Tech Dec 10 05:57:26 fir-md1-s1 kernel: hub 2-1:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 2-1:1.0: 4 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: usb: port power management may be unreliable Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1:1.0: 4 ports detected Dec 10 05:57:26 fir-md1-s1 kernel: i8042: No controller found Dec 10 05:57:26 fir-md1-s1 kernel: tsc: Refined TSC clocksource calibration: 1996.249 MHz Dec 10 05:57:26 fir-md1-s1 kernel: mousedev: PS/2 mouse device common for all mice Dec 10 05:57:26 fir-md1-s1 kernel: rtc_cmos 00:01: RTC can wake from S4 Dec 10 05:57:26 fir-md1-s1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Dec 10 05:57:26 fir-md1-s1 kernel: rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs Dec 10 05:57:26 fir-md1-s1 kernel: cpuidle: using governor menu Dec 10 05:57:26 fir-md1-s1 kernel: EFI Variables Facility v0.08 2004-May-17 Dec 10 05:57:26 fir-md1-s1 kernel: hidraw: raw HID events driver (C) Jiri Kosina Dec 10 05:57:26 fir-md1-s1 kernel: usbcore: registered new interface driver usbhid Dec 10 05:57:26 fir-md1-s1 kernel: usbhid: USB HID core driver Dec 10 05:57:26 fir-md1-s1 kernel: drop_monitor: Initializing network drop monitor service Dec 10 05:57:26 fir-md1-s1 kernel: TCP: cubic registered Dec 10 05:57:26 fir-md1-s1 kernel: Initializing XFRM netlink socket Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 10 Dec 10 05:57:26 fir-md1-s1 kernel: NET: Registered protocol family 17 Dec 10 05:57:26 fir-md1-s1 kernel: mpls_gso: MPLS GSO support Dec 10 05:57:26 fir-md1-s1 kernel: mce: Using 23 MCE banks Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU0: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU1: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU2: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU3: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU4: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU5: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU6: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU7: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU8: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU9: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU10: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU11: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU12: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU13: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU14: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU15: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU16: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU17: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU18: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU19: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU20: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU21: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU22: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU23: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU24: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU25: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU26: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU27: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU28: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU29: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU30: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU31: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU32: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU33: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU34: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU35: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU36: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU37: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU38: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU39: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU40: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU41: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU42: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU43: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU44: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU45: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU46: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: CPU47: patch_level=0x08001250 Dec 10 05:57:26 fir-md1-s1 kernel: microcode: Microcode Update Driver: v2.01 , Peter Oruba Dec 10 05:57:26 fir-md1-s1 kernel: PM: Hibernation image not present or could not be loaded. Dec 10 05:57:26 fir-md1-s1 kernel: Loading compiled-in X.509 certificates Dec 10 05:57:26 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' Dec 10 05:57:26 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' Dec 10 05:57:26 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux kernel signing key: 468656045a39b52ff2152c315f6198c3e658f24d' Dec 10 05:57:26 fir-md1-s1 kernel: registered taskstats version 1 Dec 10 05:57:26 fir-md1-s1 kernel: Key type trusted registered Dec 10 05:57:26 fir-md1-s1 kernel: Key type encrypted registered Dec 10 05:57:26 fir-md1-s1 kernel: IMA: No TPM chip found, activating TPM-bypass! (rc=-19) Dec 10 05:57:26 fir-md1-s1 kernel: Magic number: 15:608:988 Dec 10 05:57:26 fir-md1-s1 kernel: machinecheck machinecheck1: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: clockevents clockevent61: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: memory memory1632: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: memory memory1186: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: memory memory845: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: memory memory399: hash matches Dec 10 05:57:26 fir-md1-s1 kernel: rtc_cmos 00:01: setting system clock to 2019-12-10 13:57:25 UTC (1575986245) Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.1: new high-speed USB device number 3 using xhci_hcd Dec 10 05:57:26 fir-md1-s1 kernel: Switched to clocksource tsc Dec 10 05:57:26 fir-md1-s1 kernel: Freeing unused kernel memory: 1876k freed Dec 10 05:57:26 fir-md1-s1 kernel: Write protecting the kernel read-only data: 12288k Dec 10 05:57:26 fir-md1-s1 kernel: Freeing unused kernel memory: 504k freed Dec 10 05:57:26 fir-md1-s1 kernel: Freeing unused kernel memory: 596k freed Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.1: New USB device found, idVendor=1604, idProduct=10c0 Dec 10 05:57:26 fir-md1-s1 systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Dec 10 05:57:26 fir-md1-s1 systemd[1]: Detected architecture x86-64. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Running in initial RAM disk. Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1.1:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1.1:1.0: 4 ports detected Dec 10 05:57:26 fir-md1-s1 systemd[1]: Set hostname to . Dec 10 05:57:26 fir-md1-s1 systemd[1]: Reached target Timers. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Created slice Root Slice. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Created slice System Slice. Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.4: new high-speed USB device number 4 using xhci_hcd Dec 10 05:57:26 fir-md1-s1 systemd[1]: Reached target Slices. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Listening on Journal Socket. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Reached target Local File Systems. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Starting Journal Service... Dec 10 05:57:26 fir-md1-s1 systemd[1]: Starting Setup Virtual Console... Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.4: New USB device found, idVendor=1604, idProduct=10c0 Dec 10 05:57:26 fir-md1-s1 kernel: usb 3-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Dec 10 05:57:26 fir-md1-s1 systemd[1]: Starting Create list of required static device nodes for the current kernel... Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1.4:1.0: USB hub found Dec 10 05:57:26 fir-md1-s1 kernel: hub 3-1.4:1.0: 4 ports detected Dec 10 05:57:26 fir-md1-s1 systemd[1]: Listening on udev Control Socket. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Listening on udev Kernel Socket. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Reached target Sockets. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Starting dracut cmdline hook... Dec 10 05:57:26 fir-md1-s1 systemd[1]: Starting Apply Kernel Variables... Dec 10 05:57:26 fir-md1-s1 systemd[1]: Reached target Swap. Dec 10 05:57:26 fir-md1-s1 systemd[1]: Started Journal Service. Dec 10 05:57:26 fir-md1-s1 kernel: pps_core: LinuxPPS API ver. 1 registered Dec 10 05:57:26 fir-md1-s1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti Dec 10 05:57:26 fir-md1-s1 kernel: PTP clock support registered Dec 10 05:57:26 fir-md1-s1 kernel: megasas: 07.705.02.00-rh1 Dec 10 05:57:26 fir-md1-s1 kernel: mlx_compat: loading out-of-tree module taints kernel. Dec 10 05:57:26 fir-md1-s1 kernel: mlx_compat: module verification failed: signature and/or required key missing - tainting kernel Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW now in Ready state Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: 64 bit DMA mask and 32 bit consistent mask Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 68 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 69 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 70 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 71 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 72 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 73 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 74 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 75 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 76 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 77 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 78 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 79 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 80 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 81 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 82 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 83 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 84 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 85 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 86 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 87 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 88 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 89 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 90 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 91 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 92 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 93 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 94 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 95 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 96 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 97 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 98 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 99 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 100 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 101 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 102 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 103 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 104 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 105 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 106 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 107 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 108 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 109 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 110 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 111 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 112 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 113 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 114 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 115 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware supports msix : (96) Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: current msix/online cpus : (48/48) Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: RDPQ mode : (disabled) Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Configured max firmware commands: 927 Dec 10 05:57:26 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW supports sync cache : No Dec 10 05:57:26 fir-md1-s1 kernel: Compat-mlnx-ofed backport release: 1c4bf42 Dec 10 05:57:26 fir-md1-s1 kernel: Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 1c4bf42 Dec 10 05:57:26 fir-md1-s1 kernel: compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git Dec 10 05:57:26 fir-md1-s1 kernel: tg3.c:v3.137 (May 11, 2014) Dec 10 05:57:26 fir-md1-s1 kernel: libata version 3.00 loaded. Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas version 31.00.00.00 loaded Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (263565236 kB) Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7d Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7e Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Dec 10 05:57:26 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: version 3.0 Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 120 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 121 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 122 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 123 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 124 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 125 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 126 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 127 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 128 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 129 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 130 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 131 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 132 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 133 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 134 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 135 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode Dec 10 05:57:26 fir-md1-s1 kernel: ahci 0000:86:00.2: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part Dec 10 05:57:26 fir-md1-s1 kernel: scsi host2: ahci Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 136 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 137 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 138 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 139 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 140 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 141 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 142 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 143 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 144 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 145 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 146 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 147 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 148 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 149 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 150 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 151 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 152 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 153 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 154 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 155 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 156 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 157 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 158 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 159 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 160 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 161 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 162 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 163 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 164 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 165 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 166 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 167 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 168 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 169 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 170 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 171 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 172 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 173 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 174 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 175 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 176 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 177 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 178 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 179 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 180 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 181 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 182 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 183 for MSI/MSI-X Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 136 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 137 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 138 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 139 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 140 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 141 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 142 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 143 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 144 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 145 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 146 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 147 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 148 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 149 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 150 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 151 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 152 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 153 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 154 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 155 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 156 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 157 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 158 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 159 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 160 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 161 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 162 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 163 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 164 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 165 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 166 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 167 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 168 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 169 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 170 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 171 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 172 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 173 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 174 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 175 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 176 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 177 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 178 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 179 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 180 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 181 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 182 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 183 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: iomem(0x00000000ac000000), mapped(0xffff9e875a200000), size(1048576) Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: ioport(0x0000000000008000), size(256) Dec 10 05:57:26 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: firmware version: 20.26.1040 Dec 10 05:57:26 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16 GT/s x16 link) Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Dec 10 05:57:26 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Dec 10 05:57:26 fir-md1-s1 kernel: ata1: SATA max UDMA/133 abar m4096@0xc0a02000 port 0xc0a02100 irq 120 Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Init cmd return status SUCCESS for SCSI host 0 Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware type : Legacy(64 VD) firmware Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: controller type : iMR(0MB) Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Online Controller Reset(OCR) : Enabled Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Secure JBOD support : No Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: NVMe passthru support : No Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: INIT adapter done Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: unevenspan support : yes Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware crash dump : no Dec 10 05:57:27 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: jbod sync map : no Dec 10 05:57:27 fir-md1-s1 kernel: scsi host0: Avago SAS based MegaRAID driver Dec 10 05:57:27 fir-md1-s1 kernel: scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.30 PQ: 0 ANSI: 5 Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: Allocated physical memory: size(38831 kB) Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: Current Controller Queue Depth(7564), Max Controller Queue Depth(7680) Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128) Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 185 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 186 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 187 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 188 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 189 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 190 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 191 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 192 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 193 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 194 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 195 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 196 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 197 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 198 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 199 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 200 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 201 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 202 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 203 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 204 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 205 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 206 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 207 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 208 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 209 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 210 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 211 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 212 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 213 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 214 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 215 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 216 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 217 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 218 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 219 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 220 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 221 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 222 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 223 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 224 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 225 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 226 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 227 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 228 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 229 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 230 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 231 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 232 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 233 for MSI/MSI-X Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: mlx5_pcie_event:303:(pid 318): PCIe slot advertised sufficient power (27W). Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: mlx5_fw_tracer_start:776:(pid 299): FWTracer: Ownership granted and active Dec 10 05:57:27 fir-md1-s1 kernel: ata1: SATA link down (SStatus 0 SControl 300) Dec 10 05:57:27 fir-md1-s1 kernel: mlx5_ib: Mellanox Connect-IB Infiniband driver v4.7-1.0.0 Dec 10 05:57:27 fir-md1-s1 kernel: sd 0:2:0:0: [sda] 233308160 512-byte logical blocks: (119 GB/111 GiB) Dec 10 05:57:27 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Write Protect is off Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: FW Package Version(12.00.00.00) Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: SAS3616: FWVersion(12.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00) Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: Protocol=(Initiator,Target,NVMe), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ) Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: Enabled Extended Tags as Controller Supports Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 Dec 10 05:57:27 fir-md1-s1 kernel: scsi host1: Fusion MPT SAS Host Dec 10 05:57:27 fir-md1-s1 kernel: mpt3sas_cm0: sending port enable !! Dec 10 05:57:27 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 Dec 10 05:57:27 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA Dec 10 05:57:27 fir-md1-s1 kernel: sda: sda1 sda2 sda3 Dec 10 05:57:27 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk Dec 10 05:57:27 fir-md1-s1 kernel: random: crng init done Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: hba_port entry: ffff887bf0b86880, port: 255 is added to hba_port list Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00db90c00), phys(21) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0011), sas_address(0x300705b00db90c00), phy(16) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0011), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0011), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: Enclosure LSI VirtualSES 03 PQ: 0 ANSI: 7 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: set ignore_delay_remove for handle(0x0011) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: SES: handle(0x0011), sas_addr(0x300705b00db90c00), phy(16), device_name(0x300705b00db90c00) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: enclosure logical id(0x300605b00d110c00), slot(16) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: enclosure level(0x0000), connector name( C3 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: serial_number(300605B00D110C00) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:0:0: qdepth(1), tagged(0), simple(0), ordered(0), scsi_level(8), cmd_que(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: log_info(0x31200206): originator(PL), code(0x20), sub_code(0x0206) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(1) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: enclosure logical id(0x300605b00d110c00), slot(5) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: enclosure level(0x0000), connector name( C1 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: serial_number(021815000354 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: enclosure logical id(0x300605b00d110c00), slot(5) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: enclosure level(0x0000), connector name( C1 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: serial_number(021815000354 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:1: Mode parameters changed Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: enclosure logical id(0x300605b00d110c00), slot(5) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: enclosure level(0x0000), connector name( C1 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: serial_number(021815000354 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:2: Mode parameters changed Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: enclosure logical id(0x300605b00d110c00), slot(5) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: enclosure level(0x0000), connector name( C1 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: serial_number(021815000354 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(1) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Dec 10 05:57:29 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: enclosure logical id(0x300605b00d110c00), slot(13) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: enclosure level(0x0000), connector name( C3 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: serial_number(021825001369 ) Dec 10 05:57:29 fir-md1-s1 kernel: scsi 1:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: enclosure logical id(0x300605b00d110c00), slot(13) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: enclosure level(0x0000), connector name( C3 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: serial_number(021825001369 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:1: Mode parameters changed Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: enclosure logical id(0x300605b00d110c00), slot(13) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: enclosure level(0x0000), connector name( C3 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: serial_number(021825001369 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:2:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984da0f9b14), phy(12) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(1) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984da0f9b14), phy(12) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: SSP: handle(0x0019), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: enclosure logical id(0x300605b00d110c00), slot(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: enclosure level(0x0000), connector name( C0 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: serial_number(021812047179 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: SSP: handle(0x0019), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: enclosure logical id(0x300605b00d110c00), slot(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: enclosure level(0x0000), connector name( C0 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: serial_number(021812047179 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: SSP: handle(0x0019), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: enclosure logical id(0x300605b00d110c00), slot(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: enclosure level(0x0000), connector name( C0 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: serial_number(021812047179 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: SSP: handle(0x0019), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: enclosure logical id(0x300605b00d110c00), slot(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: enclosure level(0x0000), connector name( C0 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: serial_number(021812047179 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:3:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984dfa20c14), phy(4) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(1) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984dfa20c14), phy(4) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) Dec 10 05:57:30 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: SSP: handle(0x001a), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: enclosure logical id(0x300605b00d110c00), slot(9) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: enclosure level(0x0000), connector name( C2 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: serial_number(021825001558 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: SSP: handle(0x001a), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: enclosure logical id(0x300605b00d110c00), slot(9) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: enclosure level(0x0000), connector name( C2 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: serial_number(021825001558 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: SSP: handle(0x001a), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: enclosure logical id(0x300605b00d110c00), slot(9) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: enclosure level(0x0000), connector name( C2 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: serial_number(021825001558 ) Dec 10 05:57:30 fir-md1-s1 kernel: scsi 1:0:4:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Dec 10 05:57:35 fir-md1-s1 kernel: mpt3sas_cm0: port enable: SUCCESS Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (owned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] 926167040 512-byte logical blocks: (474 GB/441 GiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] 4096-byte physical blocks Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (owned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:2:0: rdac: LUN 0 (IOSHIP) (owned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:0: [sde] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:2:1: rdac: LUN 1 (IOSHIP) (unowned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:3:0: rdac: LUN 0 (IOSHIP) (unowned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] 926167040 512-byte logical blocks: (474 GB/441 GiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] 4096-byte physical blocks Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:3:1: rdac: LUN 1 (IOSHIP) (owned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:3:2: rdac: LUN 2 (IOSHIP) (unowned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:2: [sdi] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:4:0: rdac: LUN 0 (IOSHIP) (unowned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:0: [sdj] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:2: [sdi] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:2: [sdi] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:2: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: scsi 1:0:4:1: rdac: LUN 1 (IOSHIP) (owned) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:0: [sdj] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:0: [sdj] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:1: [sdk] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:1: [sdk] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:1: [sdk] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:1: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:3:2: [sdi] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:1: [sdk] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:4:0: [sdj] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Write Protect is off Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Mode Sense: 83 00 10 08 Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Dec 10 05:57:35 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Attached SCSI disk Dec 10 05:57:35 fir-md1-s1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) Dec 10 05:57:36 fir-md1-s1 systemd-journald[351]: Received SIGTERM from PID 1 (systemd). Dec 10 05:57:36 fir-md1-s1 kernel: SELinux: Disabled at runtime. Dec 10 05:57:36 fir-md1-s1 kernel: SELinux: Unregistering netfilter hooks Dec 10 05:57:36 fir-md1-s1 kernel: type=1404 audit(1575986256.170:2): selinux=0 auid=4294967295 ses=4294967295 Dec 10 05:57:36 fir-md1-s1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Dec 10 05:57:36 fir-md1-s1 systemd[1]: Inserted module 'ip_tables' Dec 10 05:57:36 fir-md1-s1 kernel: EXT4-fs (sda2): re-mounted. Opts: (null) Dec 10 05:57:36 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 Dec 10 05:57:36 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection Dec 10 05:57:36 fir-md1-s1 kernel: device-mapper: uevent: version 1.0.3 Dec 10 05:57:36 fir-md1-s1 kernel: device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com Dec 10 05:57:36 fir-md1-s1 kernel: ACPI Error: No handler for Region [SYSI] (ffff884d29af6a68) [IPMI] (20130517/evregion-162) Dec 10 05:57:36 fir-md1-s1 kernel: ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) Dec 10 05:57:36 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff884d29af35a0), AE_NOT_EXIST (20130517/psparse-536) Dec 10 05:57:37 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff884d29af3500), AE_NOT_EXIST (20130517/psparse-536) Dec 10 05:57:37 fir-md1-s1 kernel: ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: 3 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 235 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 236 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 3 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 4 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 0 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 1 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 gets LSB 6 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:02:00.2: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: 5 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: irq 238 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 gets LSB 1 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 gets LSB 2 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 gets LSB 3 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:03:00.1: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: 3 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 240 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 241 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 3 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 4 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 0 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 1 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 gets LSB 6 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:41:00.2: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: 5 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: irq 243 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 gets LSB 1 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 gets LSB 2 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 gets LSB 3 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:42:00.1: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: 3 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 245 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 246 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 3 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 4 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 0 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 1 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 gets LSB 6 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:85:00.2: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: 5 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: irq 248 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 gets LSB 1 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 gets LSB 2 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 gets LSB 3 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:86:00.1: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: 3 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 250 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 251 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 3 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 4 can access 4 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 0 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 1 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 gets LSB 6 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c2:00.2: enabled Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: 5 command queues available Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: irq 253 for MSI/MSI-X Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 can access 7 LSB regions Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 gets LSB 1 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 gets LSB 2 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 gets LSB 3 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 gets LSB 4 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 gets LSB 5 Dec 10 05:57:37 fir-md1-s1 kernel: ccp 0000:c3:00.1: enabled Dec 10 05:57:37 fir-md1-s1 kernel: cryptd: max_cpu_qlen set to 1000 Dec 10 05:57:37 fir-md1-s1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:0:0: Attached scsi generic sg1 type 13 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:0: Attached scsi generic sg2 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:1: Attached scsi generic sg3 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:2: Attached scsi generic sg4 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:1:31: Attached scsi generic sg5 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:2:0: Attached scsi generic sg6 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:2:1: Attached scsi generic sg7 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:2:31: Attached scsi generic sg8 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:0: Attached scsi generic sg9 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:1: Attached scsi generic sg10 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:2: Attached scsi generic sg11 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:3:31: Attached scsi generic sg12 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:4:0: Attached scsi generic sg13 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:4:1: Attached scsi generic sg14 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:4:31: Attached scsi generic sg15 type 0 Dec 10 05:57:37 fir-md1-s1 kernel: ipmi message handler version 39.2 Dec 10 05:57:37 fir-md1-s1 kernel: AVX2 version of gcm_enc/dec engaged. Dec 10 05:57:37 fir-md1-s1 kernel: AES CTR mode by8 optimization enabled Dec 10 05:57:37 fir-md1-s1 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input2 Dec 10 05:57:37 fir-md1-s1 kernel: ipmi device interface Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:0: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) Dec 10 05:57:37 fir-md1-s1 kernel: alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) Dec 10 05:57:37 fir-md1-s1 kernel: IPMI System Interface driver Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:1: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:1:2: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:1:31: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:2:0: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:2:1: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:2:31: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:0: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:1: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: [io 0x0ca8] regsize 1 spacing 4 irq 10 Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si: Adding ACPI-specified kcs state machine Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:3:2: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:3:31: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:4:0: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: sd 1:0:4:1: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: scsi 1:0:4:31: Embedded Enclosure Device Dec 10 05:57:37 fir-md1-s1 kernel: ses 1:0:0:0: Attached Enclosure device Dec 10 05:57:37 fir-md1-s1 kernel: dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: Using irq 10 Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) Dec 10 05:57:37 fir-md1-s1 kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized Dec 10 05:57:37 fir-md1-s1 kernel: kvm: Nested Paging enabled Dec 10 05:57:37 fir-md1-s1 kernel: MCE: In-kernel MCE decoding enabled. Dec 10 05:57:37 fir-md1-s1 kernel: AMD64 EDAC driver v3.4.0 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 0). Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC0: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:18.3 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 1). Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC1: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:19.3 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 2). Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC2: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1a.3 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 3). Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Dec 10 05:57:37 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC MC3: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1b.3 Dec 10 05:57:37 fir-md1-s1 kernel: EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.0' (POLLED) Dec 10 05:58:03 fir-md1-s1 kernel: device-mapper: multipath round-robin: version 1.2.0 loaded Dec 10 05:58:22 fir-md1-s1 kernel: Adding 4194300k swap on /dev/sda3. Priority:-2 extents:1 across:4194300k FS Dec 10 05:58:22 fir-md1-s1 kernel: type=1305 audit(1575986302.427:3): audit_pid=11921 old=0 auid=4294967295 ses=4294967295 res=1 Dec 10 05:58:22 fir-md1-s1 kernel: RPC: Registered named UNIX socket transport module. Dec 10 05:58:22 fir-md1-s1 kernel: RPC: Registered udp transport module. Dec 10 05:58:22 fir-md1-s1 kernel: RPC: Registered tcp transport module. Dec 10 05:58:22 fir-md1-s1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Dec 10 05:58:23 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: slow_pci_heuristic:5575:(pid 12222): Max link speed = 100000, PCI BW = 126016 Dec 10 05:58:23 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) Dec 10 05:58:23 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) Dec 10 05:58:23 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 254 for MSI/MSI-X Dec 10 05:58:23 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 255 for MSI/MSI-X Dec 10 05:58:23 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 256 for MSI/MSI-X Dec 10 05:58:23 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 257 for MSI/MSI-X Dec 10 05:58:23 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 258 for MSI/MSI-X Dec 10 05:58:23 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready Dec 10 05:58:27 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Link is up at 1000 Mbps, full duplex Dec 10 05:58:27 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Flow control is on for TX and on for RX Dec 10 05:58:27 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: EEE is enabled Dec 10 05:58:27 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready Dec 10 05:58:28 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready Dec 10 05:58:28 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Dec 10 05:58:32 fir-md1-s1 kernel: FS-Cache: Loaded Dec 10 05:58:32 fir-md1-s1 kernel: FS-Cache: Netfs 'nfs' registered for caching Dec 10 05:58:32 fir-md1-s1 kernel: Key type dns_resolver registered Dec 10 05:58:32 fir-md1-s1 kernel: NFS: Registering the id_resolver key type Dec 10 05:58:32 fir-md1-s1 kernel: Key type id_resolver registered Dec 10 05:58:32 fir-md1-s1 kernel: Key type id_legacy registered Dec 10 06:19:28 fir-md1-s1 kernel: LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 Dec 10 06:19:28 fir-md1-s1 kernel: alg: No test for adler32 (adler32-zlib) Dec 10 06:19:28 fir-md1-s1 kernel: Lustre: Lustre: Build Version: 2.12.3_4_g142b4d4 Dec 10 06:19:29 fir-md1-s1 kernel: LNet: 38618:0:(config.c:1627:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down Dec 10 06:19:29 fir-md1-s1 kernel: LNet: Using FastReg for registration Dec 10 06:19:29 fir-md1-s1 kernel: LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] Dec 10 06:21:06 fir-md1-s1 kernel: LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 Dec 10 06:21:06 fir-md1-s1 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Dec 10 06:21:07 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.110.62@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:21:24 fir-md1-s1 kernel: LustreError: 38844:0:(mgc_request.c:249:do_config_log_add()) MGC10.0.10.51@o2ib7: failed processing log, type 1: rc = -5 Dec 10 06:21:37 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:21:43 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:21:44 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.54@o2ib7 added to recovery queue. Health = 900 Dec 10 06:21:47 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:21:49 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:21:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:21:55 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:22:04 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:22:12 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:22:17 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:22:17 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 2 previous similar messages Dec 10 06:22:24 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:22:24 fir-md1-s1 kernel: Lustre: fir-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Dec 10 06:22:24 fir-md1-s1 kernel: Lustre: fir-MDD0001: changelog on Dec 10 06:22:24 fir-md1-s1 kernel: Lustre: fir-MDT0001: in recovery but waiting for the first client to connect Dec 10 06:22:24 fir-md1-s1 kernel: Lustre: fir-MDT0001: Will be in recovery for at least 2:30, or until 1290 clients reconnect Dec 10 06:22:24 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.53@o2ib7) Dec 10 06:22:25 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to ac744819-a0e9-dce1-af3e-f5ed5c20fc63 (at 10.9.104.14@o2ib4) Dec 10 06:22:25 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Dec 10 06:22:26 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 0f0b3b20-bdc9-8cb4-ded0-8300910ff5a5 (at 10.9.107.21@o2ib4) Dec 10 06:22:26 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Dec 10 06:22:28 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 882378af-0b41-73ee-5c10-5cc51464645c (at 10.9.108.22@o2ib4) Dec 10 06:22:28 fir-md1-s1 kernel: Lustre: Skipped 110 previous similar messages Dec 10 06:22:32 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 8660dc7a-172c-047f-9f20-55fab5f17314 (at 10.9.104.57@o2ib4) Dec 10 06:22:32 fir-md1-s1 kernel: Lustre: Skipped 712 previous similar messages Dec 10 06:22:34 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:22:34 fir-md1-s1 kernel: Lustre: 39206:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0001: extended recovery timer reaching hard limit: 900, extend: 1 Dec 10 06:22:37 fir-md1-s1 kernel: Lustre: 39206:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0001: extended recovery timer reaching hard limit: 900, extend: 1 Dec 10 06:22:37 fir-md1-s1 kernel: Lustre: fir-MDT0001: Recovery over after 0:13, of 1290 clients 1290 recovered and 0 were evicted. Dec 10 06:22:40 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to fir-MDT0001-lwp-OST0034_UUID (at 10.0.10.109@o2ib7) Dec 10 06:22:40 fir-md1-s1 kernel: Lustre: Skipped 328 previous similar messages Dec 10 06:22:49 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:23:04 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:23:19 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:23:34 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:23:34 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 1 previous similar message Dec 10 06:23:49 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:23:49 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message Dec 10 06:24:19 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:24:19 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 1 previous similar message Dec 10 06:24:32 fir-md1-s1 kernel: LustreError: 11-0: fir-OST0007-osc-MDT0001: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 Dec 10 06:24:32 fir-md1-s1 kernel: Lustre: fir-OST0003-osc-MDT0001: Connection to fir-OST0003 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:24:32 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 10 06:24:34 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:24:34 fir-md1-s1 kernel: LNetError: 318:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages Dec 10 06:25:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:25:49 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:25:49 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 1 previous similar message Dec 10 06:25:49 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:25:49 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 4 previous similar messages Dec 10 06:25:59 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to (at 10.0.10.102@o2ib7) Dec 10 06:25:59 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Dec 10 06:26:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:27:00 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 10.0.10.102@o2ib7 (at 10.0.10.102@o2ib7) Dec 10 06:27:00 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 06:27:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:27:59 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:27:59 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 3 previous similar messages Dec 10 06:27:59 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:27:59 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 8 previous similar messages Dec 10 06:28:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:28:17 fir-md1-s1 kernel: Lustre: fir-OST000b-osc-MDT0001: Connection restored to 10.0.10.102@o2ib7 (at 10.0.10.102@o2ib7) Dec 10 06:28:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 10 06:29:06 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:30:10 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:31:10 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 10.0.10.102@o2ib7 (at 10.0.10.102@o2ib7) Dec 10 06:31:10 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 10 06:32:08 fir-md1-s1 kernel: LustreError: 11-0: fir-OST0013-osc-MDT0001: operation ost_statfs to node 10.0.10.103@o2ib7 failed: rc = -107 Dec 10 06:32:08 fir-md1-s1 kernel: Lustre: fir-OST0011-osc-MDT0001: Connection to fir-OST0011 (at 10.0.10.103@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:32:08 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:32:08 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Dec 10 06:32:29 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:32:29 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 8 previous similar messages Dec 10 06:32:29 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:32:29 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 18 previous similar messages Dec 10 06:32:50 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.104@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:35:27 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 10.0.10.104@o2ib7 (at 10.0.10.104@o2ib7) Dec 10 06:35:27 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:37:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.104@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:37:39 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Dec 10 06:39:59 fir-md1-s1 kernel: LustreError: 11-0: fir-OST0021-osc-MDT0001: operation ost_statfs to node 10.0.10.105@o2ib7 failed: rc = -107 Dec 10 06:39:59 fir-md1-s1 kernel: Lustre: fir-OST001d-osc-MDT0001: Connection to fir-OST001d (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:39:59 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:39:59 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Dec 10 06:40:09 fir-md1-s1 kernel: LustreError: 11-0: fir-OST002f-osc-MDT0001: operation ost_statfs to node 10.0.10.107@o2ib7 failed: rc = -107 Dec 10 06:40:09 fir-md1-s1 kernel: Lustre: fir-OST0027-osc-MDT0001: Connection to fir-OST0027 (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:40:09 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:40:09 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 10 06:40:24 fir-md1-s1 kernel: LustreError: 11-0: fir-OST0035-osc-MDT0001: operation ost_statfs to node 10.0.10.109@o2ib7 failed: rc = -107 Dec 10 06:40:24 fir-md1-s1 kernel: Lustre: fir-OST0033-osc-MDT0001: Connection to fir-OST0033 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:40:24 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:40:24 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 10 06:40:34 fir-md1-s1 kernel: LustreError: 11-0: fir-OST003d-osc-MDT0001: operation ost_statfs to node 10.0.10.111@o2ib7 failed: rc = -107 Dec 10 06:40:34 fir-md1-s1 kernel: Lustre: fir-OST0047-osc-MDT0001: Connection to fir-OST0047 (at 10.0.10.111@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:40:34 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 06:40:34 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 10 06:40:54 fir-md1-s1 kernel: LustreError: 11-0: fir-OST0051-osc-MDT0001: operation ost_statfs to node 10.0.10.113@o2ib7 failed: rc = -107 Dec 10 06:40:54 fir-md1-s1 kernel: Lustre: fir-OST004b-osc-MDT0001: Connection to fir-OST004b (at 10.0.10.113@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 06:40:54 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 10 06:40:54 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 10 06:41:04 fir-md1-s1 kernel: LNetError: 39686:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:41:04 fir-md1-s1 kernel: LNetError: 39686:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 34 previous similar messages Dec 10 06:42:49 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:42:49 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 5 previous similar messages Dec 10 06:44:02 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to (at 10.0.10.114@o2ib7) Dec 10 06:44:02 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Dec 10 06:46:24 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.0.10.114@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 06:46:24 fir-md1-s1 kernel: LustreError: Skipped 29 previous similar messages Dec 10 06:51:20 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 06:51:20 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 41 previous similar messages Dec 10 06:52:50 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 06:52:50 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 32 previous similar messages Dec 10 07:01:03 fir-md1-s1 kernel: LustreError: 11-0: fir-MDT0000-osp-MDT0001: operation ldlm_enqueue to node 10.0.10.52@o2ib7 failed: rc = -107 Dec 10 07:01:03 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Dec 10 07:01:03 fir-md1-s1 kernel: Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 07:01:03 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 10 07:01:25 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 07:01:25 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 41 previous similar messages Dec 10 07:01:25 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.101.20@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 07:01:25 fir-md1-s1 kernel: LustreError: Skipped 6 previous similar messages Dec 10 07:02:38 fir-md1-s1 kernel: LNetError: 38679:0:(lib-move.c:2963:lnet_resend_pending_msgs_locked()) Error sending GET to 12345-10.0.10.54@o2ib7: -125 Dec 10 07:02:51 fir-md1-s1 kernel: LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 Dec 10 07:02:51 fir-md1-s1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Dec 10 07:02:59 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 1 seconds Dec 10 07:02:59 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 45 previous similar messages Dec 10 07:03:21 fir-md1-s1 kernel: LNetError: 38679:0:(lib-move.c:2963:lnet_resend_pending_msgs_locked()) Error sending GET to 12345-10.0.10.54@o2ib7: -125 Dec 10 07:03:35 fir-md1-s1 kernel: LNetError: 38679:0:(lib-move.c:2963:lnet_resend_pending_msgs_locked()) Error sending GET to 12345-10.0.10.54@o2ib7: -125 Dec 10 07:03:35 fir-md1-s1 kernel: Lustre: fir-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 0@lo) Dec 10 07:03:35 fir-md1-s1 kernel: Lustre: Skipped 52 previous similar messages Dec 10 07:03:53 fir-md1-s1 kernel: LNetError: 38679:0:(lib-move.c:2963:lnet_resend_pending_msgs_locked()) Error sending GET to 12345-10.0.10.54@o2ib7: -125 Dec 10 07:03:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Dec 10 07:03:53 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Dec 10 07:03:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Dec 10 07:03:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1290 clients reconnect Dec 10 07:03:55 fir-md1-s1 kernel: LustreError: 39388:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff886bf346f080 x1652453607966240/t0(0) o601->fir-MDT0000-lwp-OST002c_UUID@10.0.10.107@o2ib7:221/0 lens 336/0 e 0 to 0 dl 1575990241 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:03:55 fir-md1-s1 kernel: LustreError: 39388:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 1 previous similar message Dec 10 07:03:56 fir-md1-s1 kernel: LustreError: 38901:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff886bf525f080 x1652453608045088/t0(0) o601->fir-MDT0000-lwp-OST0026_UUID@10.0.10.107@o2ib7:222/0 lens 336/0 e 0 to 0 dl 1575990242 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:03:56 fir-md1-s1 kernel: LustreError: 38901:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 15 previous similar messages Dec 10 07:03:57 fir-md1-s1 kernel: LustreError: 38905:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff888becad0900 x1652453160710704/t0(0) o601->fir-MDT0000-lwp-OST0012_UUID@10.0.10.103@o2ib7:449/0 lens 336/0 e 0 to 0 dl 1575990469 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:03:57 fir-md1-s1 kernel: LustreError: 38905:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 16 previous similar messages Dec 10 07:03:59 fir-md1-s1 kernel: LustreError: 39445:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff885bd1ec9680 x1652542437489184/t0(0) o601->fir-MDT0000-lwp-OST003b_UUID@10.0.10.110@o2ib7:419/0 lens 336/0 e 0 to 0 dl 1575990439 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:03:59 fir-md1-s1 kernel: LustreError: 39445:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 49 previous similar messages Dec 10 07:04:00 fir-md1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-MDT0001: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Dec 10 07:04:03 fir-md1-s1 kernel: LustreError: 38899:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff885be7b91b00 x1652453261776256/t0(0) o601->fir-MDT0000-lwp-OST001e_UUID@10.0.10.105@o2ib7:423/0 lens 336/0 e 0 to 0 dl 1575990443 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:04:03 fir-md1-s1 kernel: LustreError: 38899:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 81 previous similar messages Dec 10 07:04:11 fir-md1-s1 kernel: LustreError: 40968:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff888bd9aa8000 x1652452417930336/t0(0) o601->fir-MDT0000-lwp-OST0046_UUID@10.0.10.111@o2ib7:493/0 lens 336/0 e 0 to 0 dl 1575990513 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 10 07:04:11 fir-md1-s1 kernel: LustreError: 40968:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 333 previous similar messages Dec 10 07:04:14 fir-md1-s1 kernel: Lustre: 40919:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Dec 10 07:04:14 fir-md1-s1 kernel: Lustre: 40919:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 1 previous similar message Dec 10 07:04:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 0:21, of 1290 clients 1290 recovered and 0 were evicted. Dec 10 07:04:57 fir-md1-s1 kernel: Lustre: 38718:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1575990290/real 0] req@ffff887be3563f00 x1652542748116640/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.52@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1575990297 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Dec 10 07:04:57 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will fail Dec 10 07:11:30 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 07:11:30 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 77 previous similar messages Dec 10 07:13:00 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds Dec 10 07:13:00 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 81 previous similar messages Dec 10 07:21:33 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 07:21:33 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 72 previous similar messages Dec 10 07:23:02 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Dec 10 07:23:02 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 70 previous similar messages Dec 10 07:31:46 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 10 07:31:46 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 65 previous similar messages Dec 10 07:32:46 fir-md1-s1 kernel: LustreError: 11-0: fir-MDT0003-osp-MDT0000: operation out_update to node 10.0.10.53@o2ib7 failed: rc = -107 Dec 10 07:32:46 fir-md1-s1 kernel: Lustre: fir-MDT0003-osp-MDT0000: Connection to fir-MDT0003 (at 10.0.10.53@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 07:32:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 07:32:49 fir-md1-s1 kernel: LustreError: 11-0: fir-MDT0003-osp-MDT0001: operation mds_statfs to node 10.0.10.53@o2ib7 failed: rc = -107 Dec 10 07:32:49 fir-md1-s1 kernel: Lustre: fir-MDT0003-osp-MDT0001: Connection to fir-MDT0003 (at 10.0.10.53@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 10 07:33:59 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.18.21@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 07:33:59 fir-md1-s1 kernel: LustreError: Skipped 1384 previous similar messages Dec 10 07:34:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) Dec 10 07:34:25 fir-md1-s1 kernel: Lustre: Skipped 1389 previous similar messages Dec 10 07:37:42 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0001 Dec 10 07:37:42 fir-md1-s1 kernel: LustreError: 39319:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.40@o2ib4 arrived at 1575992262 with bad export cookie 14105850204137837618 Dec 10 07:37:42 fir-md1-s1 kernel: Lustre: fir-MDT0001: Not available for connect from 10.8.21.2@o2ib6 (stopping) Dec 10 07:37:42 fir-md1-s1 kernel: LustreError: 39319:0:(ldlm_lock.c:2710:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0001_UUID lock: ffff887bbe6a33c0/0xc3c20c0650922e5c lrc: 3/0,0 mode: CR/CR res: [0x240039b5a:0x34fc:0x0].0x0 bits 0x8/0x0 rrc: 25 type: IBT flags: 0x40000000000000 nid: 10.9.101.40@o2ib4 remote: 0x6b5a3d080a587381 expref: 33 pid: 39419 timeout: 0 lvb_type: 3 Dec 10 07:37:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.24.1@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 10 07:37:43 fir-md1-s1 kernel: LustreError: Skipped 1386 previous similar messages Dec 10 07:37:43 fir-md1-s1 kernel: LustreError: 11-0: fir-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 Dec 10 07:37:43 fir-md1-s1 kernel: Lustre: fir-MDT0001-osp-MDT0000: Connection to fir-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Dec 10 07:37:43 fir-md1-s1 kernel: Lustre: server umount fir-MDT0001 complete Dec 10 07:37:44 fir-md1-s1 kernel: LustreError: 41013:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.36@o2ib4 arrived at 1575992264 with bad export cookie 14105850204137842210 Dec 10 07:39:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Dec 10 07:39:02 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 10 07:40:28 fir-md1-s1 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Dec 10 07:40:28 fir-md1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0xbba64b52f5111279 to 0xc3c20c065249ff79 Dec 10 07:41:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 620d1500-6c55-6b8e-5d6d-f72ab08ba0d5 (at 10.9.102.46@o2ib4) Dec 10 07:41:36 fir-md1-s1 kernel: Lustre: Skipped 500 previous similar messages Dec 10 07:53:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 192d92f7-72ee-e6af-660a-7b659d74bb6a (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bdde66400, cur 1575993213 expire 1575993063 last 1575992986 Dec 10 07:53:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 82b9ac9e-bd42-fb9c-cb3e-f327857b510c (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff884c23446000, cur 1575993221 expire 1575993071 last 1575992994 Dec 10 08:06:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4437bcee-37db-e5e0-e8d5-9055e0a77d74 (at 10.9.106.45@o2ib4) Dec 10 08:06:23 fir-md1-s1 kernel: Lustre: Skipped 806 previous similar messages Dec 10 08:34:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.0.62@o2ib4) Dec 10 08:34:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client cec884d3-ca4b-8127-2f6b-7762665aa5f8 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9191800, cur 1575995671 expire 1575995521 last 1575995444 Dec 10 08:52:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fbefd9c2-b03e-16ab-7b85-ec9f835d33da (at 10.9.105.22@o2ib4) Dec 10 08:52:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 09:10:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.0.64@o2ib4) Dec 10 09:18:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client fb9a2d5e-e9b3-4fb9-b988-9954fcfb0920 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9461800, cur 1575998292 expire 1575998142 last 1575998065 Dec 10 09:18:12 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 09:52:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fb9a2d5e-e9b3-4fb9-b988-9954fcfb0920 (at 10.8.0.66@o2ib6) Dec 10 09:52:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:08:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.110.39@o2ib4) Dec 10 10:08:45 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:17:14 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client c4027f9f-ee2a-72d3-b779-f352c130a817 (at 10.8.21.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887d4b8a7400, cur 1576001834 expire 1576001684 last 1576001607 Dec 10 10:17:14 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:46:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5d310239-acc9-4 (at 10.9.108.39@o2ib4) Dec 10 10:51:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.21.14@o2ib6) Dec 10 10:51:49 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:52:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 98c710cf-a183-35fe-d60d-8494e153f1c3 (at 10.8.21.13@o2ib6) Dec 10 10:52:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:52:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 172ec88c-3454-1411-8e15-a9b5202e9e30 (at 10.8.21.8@o2ib6) Dec 10 10:52:06 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 10 10:52:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 40a204f8-61bd-7bf5-8e8b-66a640362528 (at 10.8.21.28@o2ib6) Dec 10 10:52:16 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 10 10:52:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 13210e44-89fd-e522-ded9-67ae564de904 (at 10.8.21.20@o2ib6) Dec 10 10:52:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 10:52:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.20.18@o2ib6) Dec 10 10:52:43 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 10 10:53:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.20.15@o2ib6) Dec 10 10:53:17 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Dec 10 10:54:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5f11dd29-1211-44a2-2612-f8309cf085b3 (at 10.8.21.18@o2ib6) Dec 10 10:54:46 fir-md1-s1 kernel: Lustre: Skipped 31 previous similar messages Dec 10 11:08:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.12@o2ib6) Dec 10 11:08:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 11:09:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 92c08489-d99f-9692-0d8e-5d862ef77698 (at 10.8.22.5@o2ib6) Dec 10 11:09:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 11:10:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.20.8@o2ib6) Dec 10 11:10:35 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 11:43:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.4@o2ib6) Dec 10 11:43:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 13:07:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.20.27@o2ib6) Dec 10 13:07:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 13:44:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 8fbd1a16-d09d-1ef7-e10d-4e68dc0a9f97 (at 10.8.23.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9775c00, cur 1576014288 expire 1576014138 last 1576014061 Dec 10 13:44:48 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Dec 10 14:21:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 8fbd1a16-d09d-1ef7-e10d-4e68dc0a9f97 (at 10.8.23.32@o2ib6) Dec 10 14:21:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 14:47:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 0bbd53e2-6989-83e6-f126-86a473496205 (at 10.8.21.36@o2ib6) Dec 10 14:47:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 15:28:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client ee4590b6-1057-e690-5db0-89b0af3963cd (at 10.8.22.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba94db000, cur 1576020533 expire 1576020383 last 1576020306 Dec 10 15:28:53 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 15:35:08 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c20915b7-72a8-8f0f-a961-7c81095a2283 (at 10.8.23.29@o2ib6) Dec 10 15:35:08 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 16:03:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ee4590b6-1057-e690-5db0-89b0af3963cd (at 10.8.22.30@o2ib6) Dec 10 16:03:37 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 17:01:21 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.20@o2ib6) Dec 10 17:01:21 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:18:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 43d748a2-b8c5-e7f9-8b00-d16d4390ff4d (at 10.8.22.6@o2ib6) Dec 10 18:18:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:30:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client b6bab463-5f5c-8f5c-f09a-8f0ce0f6e1cd (at 10.8.21.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9393800, cur 1576031439 expire 1576031289 last 1576031212 Dec 10 18:30:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:31:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 7515dbe4-f1c8-844a-9186-76f9c6288c34 (at 10.9.104.2@o2ib4) in 222 seconds. I think it's dead, and I am evicting it. exp ffff887ba96ff000, cur 1576031515 expire 1576031365 last 1576031293 Dec 10 18:31:55 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Dec 10 18:51:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.114.14@o2ib4) Dec 10 18:51:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:51:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.19.6@o2ib6) Dec 10 18:51:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:55:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.110.71@o2ib4) Dec 10 18:55:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:55:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.107.9@o2ib4) Dec 10 18:55:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:57:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e8872901-9e69-2d9a-e57a-55077a64186b (at 10.9.109.25@o2ib4) Dec 10 18:57:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:59:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 2ad8ff13-d978-9373-7245-882c6479cc4c (at 10.9.110.63@o2ib4) Dec 10 18:59:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 18:59:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.110.62@o2ib4) Dec 10 18:59:34 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 19:03:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b5acf087-1850-f5e1-236a-4cc1bab1a9f0 (at 10.9.104.34@o2ib4) Dec 10 19:03:18 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 10 19:05:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b6bab463-5f5c-8f5c-f09a-8f0ce0f6e1cd (at 10.8.21.31@o2ib6) Dec 10 19:05:06 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Dec 10 19:08:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7eb73248-6ba3-525f-5dd7-7492b5394353 (at 10.8.28.9@o2ib6) Dec 10 19:08:43 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Dec 10 19:42:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client aadbd140-afe6-3cc5-5efa-1bf64465f6e7 (at 10.8.20.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba935d800, cur 1576035757 expire 1576035607 last 1576035530 Dec 10 19:42:37 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Dec 10 20:17:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to aadbd140-afe6-3cc5-5efa-1bf64465f6e7 (at 10.8.20.34@o2ib6) Dec 10 20:17:47 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Dec 10 21:36:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 55ff50e7-08a4-be07-5499-ccc18f03f2c9 (at 10.8.23.17@o2ib6) Dec 10 21:36:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 23:32:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 77f07ca8-e3bd-72f6-4ac1-3da8889522b3 (at 10.8.22.19@o2ib6) Dec 10 23:32:57 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 10 23:41:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.20.5@o2ib6) Dec 10 23:41:19 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 01:15:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 09a03217-f2a1-2632-097f-38339f6cbc7c (at 10.8.22.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba97e9000, cur 1576055715 expire 1576055565 last 1576055488 Dec 11 01:15:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 01:16:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 37c7e464-6686-fdc0-1c81-eae75026a910 (at 10.8.22.2@o2ib6) Dec 11 01:16:05 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 01:47:40 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1b1ace85-4b01-f903-bb83-ddb9142a20b0 (at 10.8.23.25@o2ib6) Dec 11 01:47:40 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 01:50:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.1@o2ib6) Dec 11 01:50:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 02:10:25 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d48dfcab-ce8f-b93c-3409-a3e76df7c945 (at 10.8.23.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9243400, cur 1576059025 expire 1576058875 last 1576058798 Dec 11 02:10:25 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 02:46:46 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.22@o2ib6) Dec 11 02:46:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 07:09:32 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.7@o2ib6) Dec 11 07:09:32 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:29:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 5a6b489d-8a0c-1dc7-c222-8c5330c92213 (at 10.8.8.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be35ea000, cur 1576081769 expire 1576081619 last 1576081542 Dec 11 08:29:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:32:30 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 672e75f9-4fe3-4 (at 10.9.109.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886bf5ad0400, cur 1576081950 expire 1576081800 last 1576081723 Dec 11 08:32:30 fir-md1-s1 kernel: Lustre: Skipped 17 previous similar messages Dec 11 08:32:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 714da8dd-1047-4 (at 10.9.107.20@o2ib4) Dec 11 08:32:37 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:36:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.110.71@o2ib4) Dec 11 08:36:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:36:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e8872901-9e69-2d9a-e57a-55077a64186b (at 10.9.109.25@o2ib4) Dec 11 08:36:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:53:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.117.46@o2ib4) Dec 11 08:53:22 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:53:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4c497e0b-ea41-4 (at 10.8.9.1@o2ib6) Dec 11 08:53:55 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:57:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 8a77a7b3-28b8-5200-390a-7fe51bf1be0a (at 10.8.7.5@o2ib6) Dec 11 08:57:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:59:02 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.101.60@o2ib4) Dec 11 08:59:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:59:19 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 54fd6f2e-cb6c-4 (at 10.9.101.57@o2ib4) Dec 11 08:59:19 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 08:59:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c658bf97-675e-4 (at 10.9.101.59@o2ib4) Dec 11 08:59:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 09:00:58 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5a6b489d-8a0c-1dc7-c222-8c5330c92213 (at 10.8.8.20@o2ib6) Dec 11 09:00:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 09:04:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fc841094-f1fd-2756-1968-f74105b220e6 (at 10.8.8.30@o2ib6) Dec 11 09:04:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 09:08:34 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.102.48@o2ib4) Dec 11 09:08:34 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 11 09:16:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 6676e5f3-c59e-c628-05b4-c9153b23c3f7 (at 10.8.21.16@o2ib6) Dec 11 09:16:12 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 11 11:28:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5ce2e68e-76b2-bbc3-75c5-66a5c2b02651 (at 10.8.23.15@o2ib6) Dec 11 11:28:59 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 11 11:52:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 45ffa07c-203c-dad9-8f0d-e714fc6465b8 (at 10.8.22.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886bf9936800, cur 1576093950 expire 1576093800 last 1576093723 Dec 11 11:52:30 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 11 12:20:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 704e8622-7442-8eb3-b4e3-c86a69ef45af (at 10.8.20.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba958c800, cur 1576095627 expire 1576095477 last 1576095400 Dec 11 12:20:27 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:26:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.11@o2ib6) Dec 11 12:26:55 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:27:07 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4f86dcb5-8d8c-1599-bd44-005eb718eb65 (at 10.8.22.10@o2ib6) Dec 11 12:27:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:28:03 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.23@o2ib6) Dec 11 12:28:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:34:42 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096475/real 1576096475] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096482 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Dec 11 12:34:49 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096482/real 1576096482] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096489 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:34:56 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096489/real 1576096489] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096496 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:35:03 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096496/real 1576096496] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096503 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:35:10 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096503/real 1576096503] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096510 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:35:24 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096517/real 1576096517] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096524 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:35:24 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message Dec 11 12:35:45 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096538/real 1576096538] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096545 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:35:45 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Dec 11 12:36:20 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576096573/real 1576096573] req@ffff884c235b2400 x1652542886044416/t0(0) o104->fir-MDT0000@10.9.112.17@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576096580 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 11 12:36:20 fir-md1-s1 kernel: Lustre: 39434:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 11 12:37:09 fir-md1-s1 kernel: LustreError: 39434:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.112.17@o2ib4) failed to reply to blocking AST (req@ffff884c235b2400 x1652542886044416 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff888bf167de80/0xc3c20c06994047fb lrc: 4/0,0 mode: PR/PR res: [0x20003963a:0x2ae:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.112.17@o2ib4 remote: 0x66c20da50bf8090b expref: 420 pid: 39396 timeout: 110532 lvb_type: 0 Dec 11 12:37:09 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.112.17@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Dec 11 12:37:09 fir-md1-s1 kernel: LustreError: 38883:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.112.17@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffff888bf167de80/0xc3c20c06994047fb lrc: 3/0,0 mode: PR/PR res: [0x20003963a:0x2ae:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.112.17@o2ib4 remote: 0x66c20da50bf8090b expref: 421 pid: 39396 timeout: 0 lvb_type: 0 Dec 11 12:37:46 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b8b1dc75-1715-2d9e-e1ec-b7625b32320e (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc6a69400, cur 1576096666 expire 1576096516 last 1576096439 Dec 11 12:37:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:55:44 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 704e8622-7442-8eb3-b4e3-c86a69ef45af (at 10.8.20.21@o2ib6) Dec 11 12:55:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 12:56:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c3415e6e-dda3-8602-28df-a932f656881d (at 10.9.112.17@o2ib4) Dec 11 12:56:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:02:16 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4c497e0b-ea41-4 (at 10.8.9.1@o2ib6) Dec 11 13:02:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:02:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.13@o2ib6) Dec 11 13:02:31 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:04:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 37c7e464-6686-fdc0-1c81-eae75026a910 (at 10.8.22.2@o2ib6) Dec 11 13:04:05 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:05:43 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b34be8aa-32d9-4 (at 10.9.113.13@o2ib4) Dec 11 13:05:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:07:04 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.101.60@o2ib4) Dec 11 13:07:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:12:36 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.24.7@o2ib6) Dec 11 13:12:36 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 11 13:37:40 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client f5dfb63b-1da5-2f76-47e2-80171bbf932c (at 10.8.22.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf9705400, cur 1576100260 expire 1576100110 last 1576100033 Dec 11 13:37:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 000d6715-906a-fe00-99d9-1ba39760e7f7 (at 10.8.22.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba935c400, cur 1576100274 expire 1576100124 last 1576100047 Dec 11 13:37:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:45:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 85fbdf3d-35db-072c-03b7-e9977baaa2bf (at 10.8.23.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba935e800, cur 1576100753 expire 1576100603 last 1576100526 Dec 11 13:45:53 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 13:49:22 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.12@o2ib6) Dec 11 13:49:22 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:12:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.8@o2ib6) Dec 11 14:12:18 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:12:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.18@o2ib6) Dec 11 14:12:26 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:12:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.18@o2ib6) Dec 11 14:12:37 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:12:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 94396c8b-eccd-7da2-de85-f79420b2e641 (at 10.8.23.33@o2ib6) Dec 11 14:12:54 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 11 14:30:44 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client f9e2b822-92c5-4 (at 10.9.117.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8859cf6ce400, cur 1576103444 expire 1576103294 last 1576103217 Dec 11 14:30:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:33:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.117.46@o2ib4) Dec 11 14:33:18 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 11 14:37:17 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.21.2@o2ib6) Dec 11 14:37:17 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 14:37:29 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.32@o2ib6) Dec 11 14:37:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 15:05:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.23.27@o2ib6) Dec 11 15:05:14 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 15:09:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.27@o2ib6) Dec 11 15:09:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 17:52:45 fir-md1-s1 kernel: perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Dec 11 18:58:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 0aa269ad-def9-3be3-d596-fd7c0af955fb (at 10.8.20.26@o2ib6) Dec 11 18:58:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 19:34:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 207217ac-1163-df36-3120-8bf6c3ecbb93 (at 10.8.23.21@o2ib6) Dec 11 19:34:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 21:40:14 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e15078c5-8209-4 (at 10.8.25.17@o2ib6) Dec 11 21:40:14 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 21:40:58 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client e15078c5-8209-4 (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9499800, cur 1576129258 expire 1576129108 last 1576129031 Dec 11 21:40:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 21:41:08 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 99698eca-49dd-4 (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf43fc400, cur 1576129268 expire 1576129118 last 1576129041 Dec 11 21:47:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 208ccf09-d6ca-4 (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887926a88400, cur 1576129643 expire 1576129493 last 1576129416 Dec 11 22:06:23 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e15078c5-8209-4 (at 10.8.25.17@o2ib6) Dec 11 22:06:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 22:14:23 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d223256d-1b6e-4 (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88797e232000, cur 1576131263 expire 1576131113 last 1576131036 Dec 11 22:14:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 22:54:39 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.20@o2ib6) Dec 11 22:54:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 22:54:45 fir-md1-s1 kernel: Lustre: MGS: Connection restored to bd358c1a-07c6-3f9f-7c84-efdb04e29ef9 (at 10.8.21.1@o2ib6) Dec 11 22:54:45 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 22:55:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.26@o2ib6) Dec 11 22:55:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 11 23:40:53 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 62793aa5-4457-2f74-3453-81c7d0efe754 (at 10.8.22.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc761f000, cur 1576136453 expire 1576136303 last 1576136226 Dec 11 23:40:53 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:15:33 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 687b1eea-b865-b791-9de5-a67096eac725 (at 10.8.23.26@o2ib6) Dec 12 00:15:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:15:47 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ca09bd61-a4b3-111c-b997-9c7823236764 (at 10.8.22.17@o2ib6) Dec 12 00:15:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:15:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 00850750-7463-78da-94ee-623be2781c44 (at 10.8.22.22@o2ib6) Dec 12 00:15:49 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:16:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to a507eb44-8ff1-13e2-fab8-30d1823663f8 (at 10.8.22.24@o2ib6) Dec 12 00:16:06 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:19:56 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576138195/real 1576138195] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576138796 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Dec 12 00:19:56 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 12 00:19:56 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 00:19:56 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 00:19:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:20:11 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.14@o2ib6) Dec 12 00:29:57 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576138796/real 1576138796] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576139397 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 00:29:57 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 00:29:57 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 00:29:57 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 00:32:32 fir-md1-s1 kernel: Lustre: 40805:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576138796/real 1576138796] req@ffff887be1e81b00 x1652542919618352/t0(0) o5->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1576139552 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Dec 12 00:32:32 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 00:39:58 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576139397/real 1576139397] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576139998 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 00:39:58 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 00:39:58 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 00:45:09 fir-md1-s1 kernel: Lustre: 40805:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576139553/real 1576139553] req@ffff887bf70f2d00 x1652542920146672/t0(0) o5->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1576140309 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Dec 12 00:45:09 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 00:50:02 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576139998/real 1576139998] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576140599 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 00:50:02 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 00:50:02 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 00:52:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.25.17@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 00:52:49 fir-md1-s1 kernel: LustreError: Skipped 2725 previous similar messages Dec 12 00:54:30 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.25.17@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 00:57:46 fir-md1-s1 kernel: Lustre: 40805:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576140310/real 1576140310] req@ffff887805158d80 x1652542920402592/t0(0) o5->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1576141066 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Dec 12 00:57:46 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 01:00:03 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:00:03 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:00:03 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 01:01:31 fir-md1-s1 kernel: Lustre: fir-OST005e-osc-MDT0000: Connection to fir-OST005e (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:03:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 619199f2-141e-aa07-09cb-eb294e06c3f1 (at 10.9.116.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9397c00, cur 1576141381 expire 1576141231 last 1576141154 Dec 12 01:10:04 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576141203/real 1576141203] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576141804 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 01:10:04 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Dec 12 01:10:04 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:10:04 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:10:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 01:10:23 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 01:14:08 fir-md1-s1 kernel: Lustre: fir-OST005e-osc-MDT0000: Connection to fir-OST005e (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:16:49 fir-md1-s1 kernel: Lustre: fir-OST0058-osc-MDT0000: Connection to fir-OST0058 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:20:05 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576141804/real 1576141804] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576142405 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 01:20:05 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Dec 12 01:20:05 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:20:05 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:20:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 01:20:25 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 01:26:44 fir-md1-s1 kernel: Lustre: fir-OST005e-osc-MDT0000: Connection to fir-OST005e (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:30:06 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576142405/real 1576142405] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576143006 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 01:30:06 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Dec 12 01:30:06 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:30:06 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 01:33:02 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 01:39:20 fir-md1-s1 kernel: Lustre: fir-OST005e-osc-MDT0000: Connection to fir-OST005e (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:39:20 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 01:40:07 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576143006/real 1576143006] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576143607 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 01:40:07 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Dec 12 01:40:07 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:40:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 01:45:39 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576143607/real 1576143607] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576144208 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 01:50:08 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 01:58:16 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576144208/real 1576144208] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576144809 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:00:09 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576144809/real 1576144809] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576145410 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:10:10 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:10:53 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576145410/real 1576145410] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576146011 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:20:11 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:23:30 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576146011/real 1576146011] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576146612 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:30:12 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 02:36:07 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576146612/real 1576146612] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576147213 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:40:13 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 02:48:44 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576147213/real 1576147213] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576147814 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 02:50:14 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576147814/real 1576147814] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576148415 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:00:15 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:01:21 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 03:03:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 33fb836e-8923-4 (at 10.9.113.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888b8c72bc00, cur 1576148628 expire 1576148478 last 1576148401 Dec 12 03:03:48 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 03:08:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 112f7644-c2be-8370-fe29-78c9940c58ee (at 10.9.103.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc47de000, cur 1576148913 expire 1576148763 last 1576148686 Dec 12 03:08:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576148415/real 1576148415] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576149016 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:10:17 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 03:13:58 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576149017/real 1576149017] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576149618 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:20:18 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 03:26:35 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576149618/real 1576149618] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576150219 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:30:20 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:39:12 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576150220/real 1576150220] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576150821 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:40:21 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576150821/real 1576150821] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576151422 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 03:50:22 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 03:51:49 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576151422/real 1576151422] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576152023 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:00:23 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 04:03:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a83208a9-361d-4 (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bed3dfc00, cur 1576152202 expire 1576152052 last 1576151975 Dec 12 04:03:22 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 04:03:24 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 97bcf7cb-bf78-4 (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf3c82000, cur 1576152204 expire 1576152054 last 1576151977 Dec 12 04:03:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 04:04:26 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:08:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 46023962-0c0f-4f56-ba25-877d19751e9f (at 10.8.18.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba973d400, cur 1576152539 expire 1576152389 last 1576152312 Dec 12 04:08:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576152023/real 1576152023] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576152624 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:10:24 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Dec 12 04:17:03 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576152624/real 1576152624] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576153225 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:20:25 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 04:22:59 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client da9227f5-bd81-94e6-98e1-3a8a3bec89b0 (at 10.9.103.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be173f000, cur 1576153379 expire 1576153229 last 1576153152 Dec 12 04:22:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 04:29:40 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576153225/real 1576153225] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576153826 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:30:26 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 04:35:45 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:37:39 fir-md1-s1 kernel: LustreError: 40817:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005c-osc-MDT0000: cannot cleanup orphans: rc = -11 Dec 12 04:38:26 fir-md1-s1 kernel: LustreError: 40809:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0058-osc-MDT0000: cannot cleanup orphans: rc = -11 Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576153826/real 1576153826] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576154427 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:40:27 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 04:40:29 fir-md1-s1 kernel: LustreError: 40801:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0054-osc-MDT0000: cannot cleanup orphans: rc = -11 Dec 12 04:42:17 fir-md1-s1 kernel: LustreError: 40805:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0056-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:43:09 fir-md1-s1 kernel: LustreError: 40813:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005a-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:48:22 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:50:16 fir-md1-s1 kernel: LustreError: 40817:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005c-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576154427/real 1576154427] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576155028 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 04:50:28 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 04:53:06 fir-md1-s1 kernel: LustreError: 40801:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0054-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 04:53:06 fir-md1-s1 kernel: LustreError: 40801:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576155028/real 1576155028] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576155629 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:00:29 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 05:00:59 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 05:00:59 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages Dec 12 05:02:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 27dd63c4-0630-b8af-eb2d-2f38c1747230 (at 10.8.19.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba97b6400, cur 1576155721 expire 1576155571 last 1576155494 Dec 12 05:02:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576155629/real 1576155629] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576156230 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:10:30 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 05:13:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d4e78436-48cb-55f2-4bab-88419072f51d (at 10.9.103.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba97e8400, cur 1576156395 expire 1576156245 last 1576156168 Dec 12 05:13:15 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:13:36 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 05:13:36 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576156230/real 1576156230] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576156831 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:20:31 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 05:23:49 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client babd4767-8aaa-fdee-2202-d9471210976a (at 10.9.104.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885be4138800, cur 1576157029 expire 1576156879 last 1576156802 Dec 12 05:23:49 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:26:13 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 05:26:13 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 05:28:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client ee45735a-3c72-071c-fe40-2e82d3a751bd (at 10.8.7.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9067000, cur 1576157324 expire 1576157174 last 1576157097 Dec 12 05:28:44 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576156831/real 1576156831] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576157432 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:30:32 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 05:37:21 fir-md1-s1 kernel: INFO: task mdt01_016:39241 blocked for more than 120 seconds. Dec 12 05:37:21 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:37:21 fir-md1-s1 kernel: mdt01_016 D ffff887bbfeea080 0 39241 2 0x00000080 Dec 12 05:37:21 fir-md1-s1 kernel: Call Trace: Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lquota_disk_read+0xf2/0x390 [lquota] Dec 12 05:37:21 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:37:21 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:37:21 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? qsd_op_begin+0x262/0x4b0 [lquota] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? osd_declare_inode_qid+0x27b/0x430 [osd_ldiskfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lu_context_refill+0x19/0x50 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:37:21 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:37:21 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:37:21 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:37:21 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:37:21 fir-md1-s1 kernel: INFO: task mdt00_016:39323 blocked for more than 120 seconds. Dec 12 05:37:21 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:37:21 fir-md1-s1 kernel: mdt00_016 D ffff887bbf735140 0 39323 2 0x00000080 Dec 12 05:37:21 fir-md1-s1 kernel: Call Trace: Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lquota_disk_read+0xf2/0x390 [lquota] Dec 12 05:37:21 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:37:21 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? __radix_tree_lookup+0x84/0xf0 Dec 12 05:37:21 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:37:21 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? qsd_op_begin+0x262/0x4b0 [lquota] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? osd_declare_inode_qid+0x27b/0x430 [osd_ldiskfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lu_context_refill+0x19/0x50 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Dec 12 05:37:21 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? ldlm_lock_create+0xa4/0x9f0 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] Dec 12 05:37:21 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:37:21 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:37:21 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:37:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:37:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:37:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:37:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:37:22 fir-md1-s1 kernel: INFO: task mdt00_038:39399 blocked for more than 120 seconds. Dec 12 05:37:22 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:37:22 fir-md1-s1 kernel: mdt00_038 D ffff885bf330a080 0 39399 2 0x00000080 Dec 12 05:37:22 fir-md1-s1 kernel: Call Trace: Dec 12 05:37:22 fir-md1-s1 kernel: [] ? lquota_disk_read+0xf2/0x390 [lquota] Dec 12 05:37:22 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:37:22 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:37:22 fir-md1-s1 kernel: [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:37:22 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:37:22 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? qsd_op_begin+0x262/0x4b0 [lquota] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? osd_declare_inode_qid+0x27b/0x430 [osd_ldiskfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? lu_context_refill+0x19/0x50 [obdclass] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Dec 12 05:37:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] Dec 12 05:37:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:37:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:37:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:37:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:37:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:37:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:37:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:38:41 fir-md1-s1 kernel: LNet: Service thread pid 39241 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 05:38:41 fir-md1-s1 kernel: Pid: 39241, comm: mdt01_016 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:38:41 fir-md1-s1 kernel: Call Trace: Dec 12 05:38:41 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:38:41 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:38:41 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:38:41 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:38:41 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:38:41 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:38:41 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:38:41 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:38:41 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:38:41 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:38:41 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:38:41 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576157921.39241 Dec 12 05:38:50 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 05:38:50 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 05:39:04 fir-md1-s1 kernel: LNet: Service thread pid 39399 was inactive for 224.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 05:39:04 fir-md1-s1 kernel: Pid: 39399, comm: mdt00_038 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:39:04 fir-md1-s1 kernel: Call Trace: Dec 12 05:39:04 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 05:39:04 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:39:04 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:39:04 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:39:04 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:39:04 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:39:04 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:39:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576157944.39399 Dec 12 05:39:06 fir-md1-s1 kernel: LNet: Service thread pid 39399 completed after 226.90s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 05:39:06 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 05:39:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2d6a9cf7-46ee-4 (at 10.8.7.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886a73a17800, cur 1576157960 expire 1576157810 last 1576157733 Dec 12 05:39:20 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576157432/real 1576157432] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576158033 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:40:33 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 05:40:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 19c70918-a172-38a5-2512-02b987cb686f (at 10.9.116.8@o2ib4) in 152 seconds. I think it's dead, and I am evicting it. exp ffff888bed3da400, cur 1576158036 expire 1576157886 last 1576157884 Dec 12 05:40:36 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 05:41:51 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 02299f50-fd55-88da-2f6a-7032b0997b32 (at 10.9.116.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf5254000, cur 1576158111 expire 1576157961 last 1576157884 Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576158033/real 1576158033] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576158634 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 05:50:35 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 05:51:27 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 05:51:27 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 05:52:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 75c6d6d0-df4c-7543-716f-77a06d0b577a (at 10.9.103.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bed3d9800, cur 1576158727 expire 1576158577 last 1576158500 Dec 12 05:52:19 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client f12efaec-23ad-f3f4-5f09-9dc112b40215 (at 10.9.103.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bd0a2c800, cur 1576158739 expire 1576158589 last 1576158512 Dec 12 05:55:22 fir-md1-s1 kernel: INFO: task mdt01_016:39241 blocked for more than 120 seconds. Dec 12 05:55:22 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:22 fir-md1-s1 kernel: mdt01_016 D ffff887bbfeea080 0 39241 2 0x00000080 Dec 12 05:55:22 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Dec 12 05:55:22 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:22 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:22 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:22 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? try_to_del_timer_sync+0x5e/0x90 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? del_timer_sync+0x52/0x60 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? schedule_timeout+0x170/0x2d0 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lod_qos_statfs_update+0x3c/0x2b0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x375/0x780 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:22 fir-md1-s1 kernel: INFO: task mdt03_023:39352 blocked for more than 120 seconds. Dec 12 05:55:22 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:22 fir-md1-s1 kernel: mdt03_023 D ffff887bbf659040 0 39352 2 0x00000080 Dec 12 05:55:22 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:22 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:22 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:22 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:22 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ldlm_cli_enqueue_local+0x272/0x830 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:22 fir-md1-s1 kernel: INFO: task mdt01_031:39382 blocked for more than 120 seconds. Dec 12 05:55:22 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:23 fir-md1-s1 kernel: mdt01_031 D ffff887bbf325140 0 39382 2 0x00000080 Dec 12 05:55:23 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:23 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Dec 12 05:55:23 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:23 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:23 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:23 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? try_to_del_timer_sync+0x5e/0x90 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? del_timer_sync+0x52/0x60 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? schedule_timeout+0x170/0x2d0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_qos_statfs_update+0x3c/0x2b0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x375/0x780 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: INFO: task mdt01_032:39384 blocked for more than 120 seconds. Dec 12 05:55:23 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:23 fir-md1-s1 kernel: mdt01_032 D ffff887bbf368000 0 39384 2 0x00000080 Dec 12 05:55:23 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:23 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Dec 12 05:55:23 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:23 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:23 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:23 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? try_to_del_timer_sync+0x5e/0x90 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? del_timer_sync+0x52/0x60 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? schedule_timeout+0x170/0x2d0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_qos_statfs_update+0x3c/0x2b0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x375/0x780 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: INFO: task mdt00_038:39399 blocked for more than 120 seconds. Dec 12 05:55:23 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:23 fir-md1-s1 kernel: mdt00_038 D ffff885bf330a080 0 39399 2 0x00000080 Dec 12 05:55:23 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:23 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Dec 12 05:55:23 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:23 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:23 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:23 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? try_to_del_timer_sync+0x5e/0x90 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? del_timer_sync+0x52/0x60 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? schedule_timeout+0x170/0x2d0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_qos_statfs_update+0x3c/0x2b0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x375/0x780 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:23 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:23 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:23 fir-md1-s1 kernel: INFO: task mdt02_049:39433 blocked for more than 120 seconds. Dec 12 05:55:23 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:55:23 fir-md1-s1 kernel: mdt02_049 D ffff887bbee330c0 0 39433 2 0x00000080 Dec 12 05:55:23 fir-md1-s1 kernel: Call Trace: Dec 12 05:55:23 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Dec 12 05:55:23 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:55:23 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:55:23 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:55:23 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:55:23 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? try_to_del_timer_sync+0x5e/0x90 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? del_timer_sync+0x52/0x60 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? schedule_timeout+0x170/0x2d0 Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_qos_statfs_update+0x3c/0x2b0 [lod] Dec 12 05:55:23 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x375/0x780 [lod] Dec 12 05:55:24 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? ldlm_inodebits_alloc_lock+0x66/0x180 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Dec 12 05:55:24 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:55:24 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:55:24 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:55:24 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 05:55:24 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:55:24 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Dec 12 05:55:24 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:55:24 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:55:24 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:55:24 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:55:24 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:55:24 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:55:24 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:55:24 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:56:36 fir-md1-s1 kernel: LNet: Service thread pid 39399 was inactive for 200.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 05:56:36 fir-md1-s1 kernel: Pid: 39399, comm: mdt00_038 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:56:36 fir-md1-s1 kernel: Call Trace: Dec 12 05:56:36 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 05:56:36 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:56:36 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:56:36 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:56:36 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576158996.39399 Dec 12 05:56:36 fir-md1-s1 kernel: Pid: 39384, comm: mdt01_032 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:56:36 fir-md1-s1 kernel: Call Trace: Dec 12 05:56:36 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:56:36 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:56:36 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:56:36 fir-md1-s1 kernel: Pid: 39241, comm: mdt01_016 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:56:36 fir-md1-s1 kernel: Call Trace: Dec 12 05:56:36 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:56:36 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:56:36 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:56:36 fir-md1-s1 kernel: Pid: 39382, comm: mdt01_031 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:56:36 fir-md1-s1 kernel: Call Trace: Dec 12 05:56:36 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:56:36 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:56:36 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:56:36 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:56:42 fir-md1-s1 kernel: LNet: Service thread pid 39352 was inactive for 200.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 05:56:42 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 05:56:42 fir-md1-s1 kernel: Pid: 39352, comm: mdt03_023 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 05:56:42 fir-md1-s1 kernel: Call Trace: Dec 12 05:56:42 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:56:42 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:56:42 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:56:42 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 05:56:42 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 05:56:42 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 05:56:42 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 05:56:42 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 05:56:42 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 05:56:42 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:56:42 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:56:42 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:56:42 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:56:42 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 05:56:42 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159002.39352 Dec 12 05:56:50 fir-md1-s1 kernel: LNet: Service thread pid 39239 was inactive for 200.70s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 05:56:50 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159010.39239 Dec 12 05:57:24 fir-md1-s1 kernel: INFO: task mdt02_010:39239 blocked for more than 120 seconds. Dec 12 05:57:24 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 12 05:57:24 fir-md1-s1 kernel: mdt02_010 D ffff887bbfee8000 0 39239 2 0x00000080 Dec 12 05:57:24 fir-md1-s1 kernel: Call Trace: Dec 12 05:57:24 fir-md1-s1 kernel: [] ? lquota_disk_read+0xf2/0x390 [lquota] Dec 12 05:57:24 fir-md1-s1 kernel: [] schedule+0x29/0x70 Dec 12 05:57:24 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Dec 12 05:57:24 fir-md1-s1 kernel: [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? __radix_tree_lookup+0x84/0xf0 Dec 12 05:57:24 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 05:57:24 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Dec 12 05:57:24 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? qsd_op_begin+0x262/0x4b0 [lquota] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? osd_declare_inode_qid+0x27b/0x430 [osd_ldiskfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? lu_context_refill+0x19/0x50 [obdclass] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Dec 12 05:57:24 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] Dec 12 05:57:24 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Dec 12 05:57:24 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Dec 12 05:57:24 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Dec 12 05:57:24 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Dec 12 05:57:24 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 05:57:24 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:57:24 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 05:57:24 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 12 05:57:44 fir-md1-s1 kernel: LNet: Service thread pid 39269 was inactive for 200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 05:57:44 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159064.39269 Dec 12 05:58:09 fir-md1-s1 kernel: LNet: Service thread pid 39399 completed after 294.37s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 05:58:15 fir-md1-s1 kernel: LNet: Service thread pid 39255 was inactive for 200.36s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 05:58:15 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159095.39255 Dec 12 05:58:29 fir-md1-s1 kernel: LNet: Service thread pid 39367 was inactive for 200.02s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 05:58:29 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159109.39367 Dec 12 05:59:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 030cce72-3f78-2631-9a21-d2dac6dcbefa (at 10.8.19.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9029000, cur 1576159160 expire 1576159010 last 1576158933 Dec 12 06:00:35 fir-md1-s1 kernel: LustreError: 39375:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576158935, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887af3215580/0xc3c20c06c19a7142 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39375 timeout: 0 lvb_type: 0 Dec 12 06:00:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159235.39375 Dec 12 06:00:36 fir-md1-s1 kernel: LustreError: 39341:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576158936, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887acca37080/0xc3c20c06c19a718f lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39341 timeout: 0 lvb_type: 0 Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576158635/real 1576158635] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576159236 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:00:36 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 06:00:36 fir-md1-s1 kernel: LustreError: 39341:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Dec 12 06:00:37 fir-md1-s1 kernel: LustreError: 39323:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576158937, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852deede780/0xc3c20c06c19a74c9 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39323 timeout: 0 lvb_type: 0 Dec 12 06:00:37 fir-md1-s1 kernel: LustreError: 39323:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Dec 12 06:00:42 fir-md1-s1 kernel: LustreError: 39364:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576158942, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8874977bf080/0xc3c20c06c19a7a79 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39364 timeout: 0 lvb_type: 0 Dec 12 06:00:42 fir-md1-s1 kernel: LustreError: 39364:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 06:00:46 fir-md1-s1 kernel: LNet: Service thread pid 39236 was inactive for 310.36s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 06:00:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159246.39236 Dec 12 06:00:47 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159247.39395 Dec 12 06:00:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159248.39375 Dec 12 06:00:49 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159249.39341 Dec 12 06:00:50 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159250.39267 Dec 12 06:01:20 fir-md1-s1 kernel: LustreError: 39346:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576158980, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8877ae0d0b40/0xc3c20c06c19a96a3 lrc: 3/1,0 mode: --/PR res: [0x200029791:0x7f50:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39346 timeout: 0 lvb_type: 0 Dec 12 06:01:20 fir-md1-s1 kernel: LustreError: 39346:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 06:01:29 fir-md1-s1 kernel: LNet: Service thread pid 39384 completed after 494.34s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 06:01:45 fir-md1-s1 kernel: LNet: Service thread pid 39364 was inactive for 362.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 06:01:45 fir-md1-s1 kernel: Pid: 39364, comm: mdt00_030 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:01:45 fir-md1-s1 kernel: Call Trace: Dec 12 06:01:45 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 06:01:45 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:01:45 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 06:01:45 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 06:01:45 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:01:45 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:01:45 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:01:45 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:01:45 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:01:45 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159305.39364 Dec 12 06:02:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 4da0631e-0b9c-7c27-c6e2-66c5a8c0b673 (at 10.9.101.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be8230c00, cur 1576159353 expire 1576159203 last 1576159126 Dec 12 06:02:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 06:03:10 fir-md1-s1 kernel: Lustre: 39384:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff886bf21a8900 x1651216130790064/t0(0) o101->a1acf167-afde-6f5a-879d-1a7c0814f282@10.9.117.21@o2ib4:255/0 lens 376/1600 e 19 to 0 dl 1576159395 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:03:11 fir-md1-s1 kernel: LNet: Service thread pid 39419 was inactive for 410.69s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 06:03:11 fir-md1-s1 kernel: Pid: 39419, comm: mdt02_045 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:03:11 fir-md1-s1 kernel: Call Trace: Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:03:11 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:03:11 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:03:11 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159391.39419 Dec 12 06:03:11 fir-md1-s1 kernel: Pid: 39346, comm: mdt02_026 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:03:11 fir-md1-s1 kernel: Call Trace: Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:03:11 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:03:11 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:03:11 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:03:13 fir-md1-s1 kernel: Pid: 39407, comm: mdt01_036 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:03:13 fir-md1-s1 kernel: Call Trace: Dec 12 06:03:13 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 06:03:13 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 06:03:13 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 06:03:13 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 06:03:13 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 06:03:13 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 06:03:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:03:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:03:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:03:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:03:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:03:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576159393.39407 Dec 12 06:03:16 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a1acf167-afde-6f5a-879d-1a7c0814f282 (at 10.9.117.21@o2ib4) reconnecting Dec 12 06:03:17 fir-md1-s1 kernel: Lustre: 39266:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff888bf873cc80 x1649559130108000/t0(0) o101->a8d84424-9b8a-5525-fab4-b5243bf0dc64@10.9.104.22@o2ib4:262/0 lens 376/1600 e 19 to 0 dl 1576159402 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:03:17 fir-md1-s1 kernel: Lustre: 39266:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Dec 12 06:03:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a8d84424-9b8a-5525-fab4-b5243bf0dc64 (at 10.9.104.22@o2ib4) reconnecting Dec 12 06:03:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 06:03:24 fir-md1-s1 kernel: Lustre: 39277:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff887bf9fa3f00 x1649312727839232/t0(0) o101->e19e1947-897d-03aa-f267-2edb615db310@10.9.110.41@o2ib4:269/0 lens 1888/3288 e 19 to 0 dl 1576159409 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:03:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client e19e1947-897d-03aa-f267-2edb615db310 (at 10.9.110.41@o2ib4) reconnecting Dec 12 06:04:04 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 06:04:04 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 06:04:18 fir-md1-s1 kernel: Lustre: 39411:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff887ba5484c80 x1651840017167536/t0(0) o101->20841216-9d8b-7794-9459-ced18b617ae2@10.9.114.3@o2ib4:323/0 lens 1792/3288 e 7 to 0 dl 1576159463 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:04:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 20841216-9d8b-7794-9459-ced18b617ae2 (at 10.9.114.3@o2ib4) reconnecting Dec 12 06:04:49 fir-md1-s1 kernel: Lustre: 39220:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff886bf5ab9f80 x1648842511409920/t0(0) o101->68425483-9450-d7a7-cad3-736e62941d5a@10.9.110.18@o2ib4:354/0 lens 376/1600 e 5 to 0 dl 1576159494 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:04:49 fir-md1-s1 kernel: LNet: Service thread pid 39352 completed after 687.52s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 06:04:49 fir-md1-s1 kernel: LNet: Skipped 20 previous similar messages Dec 12 06:09:50 fir-md1-s1 kernel: LustreError: 39265:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576159490, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852dd9d33c0/0xc3c20c06c19ce3d7 lrc: 3/1,0 mode: --/PR res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39265 timeout: 0 lvb_type: 0 Dec 12 06:10:36 fir-md1-s1 kernel: Lustre: 40805:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576159080/real 1576159080] req@ffff887bbd98b180 x1652542930629712/t0(0) o5->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1576159836 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Dec 12 06:10:36 fir-md1-s1 kernel: Lustre: 40805:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Dec 12 06:10:37 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:10:37 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 06:10:37 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:10:37 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Dec 12 06:11:29 fir-md1-s1 kernel: LustreError: 39371:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576159589, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885d87215a00/0xc3c20c06c19e4d99 lrc: 3/1,0 mode: --/PR res: [0x200037a5a:0xbae0:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39371 timeout: 0 lvb_type: 0 Dec 12 06:11:29 fir-md1-s1 kernel: LustreError: 39371:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 06:12:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 7ac0db55-de36-c1c6-f1a9-d7191d6b9947 (at 10.9.103.29@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba91cfc00, cur 1576159924 expire 1576159774 last 1576159697 Dec 12 06:12:04 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 06:13:09 fir-md1-s1 kernel: LustreError: 39389:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576159689, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852deec3840/0xc3c20c06c19f6545 lrc: 3/1,0 mode: --/PR res: [0x2000389b9:0x11efe:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39389 timeout: 0 lvb_type: 0 Dec 12 06:13:09 fir-md1-s1 kernel: LustreError: 39389:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 06:14:49 fir-md1-s1 kernel: LustreError: 39343:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576159789, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff888bf3b71f80/0xc3c20c06c1a07b54 lrc: 3/0,1 mode: --/CW res: [0x200029791:0x7f50:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39343 timeout: 0 lvb_type: 0 Dec 12 06:14:49 fir-md1-s1 kernel: LustreError: 39343:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 8 previous similar messages Dec 12 06:14:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client dc8b0e50-2be4-ddc9-1be7-a287c814d044 (at 10.9.110.46@o2ib4) reconnecting Dec 12 06:14:49 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 06:16:41 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 06:16:41 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576159837/real 1576159837] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576160438 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:20:38 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 06:29:18 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 06:29:18 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576160438/real 1576160438] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576161039 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:30:39 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576161039/real 1576161039] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576161640 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:40:40 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 06:41:55 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 06:41:55 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 06:45:27 fir-md1-s1 kernel: LustreError: 39414:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576161627, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8855fafebf00/0xc3c20c06c1b29d18 lrc: 3/1,0 mode: --/PR res: [0x200029791:0x7f50:0x0].0x0 bits 0x13/0x0 rrc: 24 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39414 timeout: 0 lvb_type: 0 Dec 12 06:45:27 fir-md1-s1 kernel: LustreError: 39414:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 06:46:02 fir-md1-s1 kernel: LustreError: 38893:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576161662, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887bc6b87980/0xc3c20c06c1b2c0f1 lrc: 3/1,0 mode: --/PR res: [0x200029791:0x7f50:0x0].0x0 bits 0x13/0x0 rrc: 24 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 38893 timeout: 0 lvb_type: 0 Dec 12 06:46:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client de02546b-f416-f3b2-d476-06fb4a31366f (at 10.8.7.20@o2ib6) reconnecting Dec 12 06:47:21 fir-md1-s1 kernel: LustreError: 39432:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576161741, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8859c7d18480/0xc3c20c06c1b33264 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 42 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39432 timeout: 0 lvb_type: 0 Dec 12 06:47:21 fir-md1-s1 kernel: LustreError: 39432:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 24 previous similar messages Dec 12 06:48:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1d444526-0c94-9229-34be-9d214c0c6bbd (at 10.9.101.46@o2ib4) reconnecting Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576161640/real 1576161640] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576162241 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 06:50:42 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Dec 12 06:53:05 fir-md1-s1 kernel: Lustre: 38889:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff885b94335100 x1649443527617904/t0(0) o101->75ca7fbe-4dbb-5345-e1bf-3a337b10784c@10.9.117.38@o2ib4:230/0 lens 1800/3288 e 4 to 0 dl 1576162390 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 06:53:09 fir-md1-s1 kernel: LNet: Service thread pid 39329 was inactive for 599.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 06:53:09 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 06:53:09 fir-md1-s1 kernel: Pid: 39329, comm: mdt00_019 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:53:09 fir-md1-s1 kernel: Call Trace: Dec 12 06:53:09 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_object_local_lock+0x438/0xb20 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_reint_open+0x106a/0x3240 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:53:09 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:53:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:53:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:53:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:53:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576162389.39329 Dec 12 06:53:10 fir-md1-s1 kernel: LustreError: 39358:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576162090, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887ad67ebf00/0xc3c20c06c1b6f236 lrc: 3/0,1 mode: --/CW res: [0x200029791:0x7f50:0x0].0x0 bits 0x2/0x0 rrc: 29 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39358 timeout: 0 lvb_type: 0 Dec 12 06:53:10 fir-md1-s1 kernel: LustreError: 39358:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 18 previous similar messages Dec 12 06:53:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 02eb8135-4034-bcb2-8df8-77d00506e76a (at 10.8.7.15@o2ib6) reconnecting Dec 12 06:53:10 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 06:53:11 fir-md1-s1 kernel: LNet: Service thread pid 39346 was inactive for 601.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 06:53:11 fir-md1-s1 kernel: Pid: 39346, comm: mdt02_026 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:53:11 fir-md1-s1 kernel: Call Trace: Dec 12 06:53:11 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 06:53:11 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:53:11 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 06:53:11 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 06:53:11 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:53:11 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:53:11 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:53:11 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:53:11 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:53:11 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576162391.39346 Dec 12 06:53:12 fir-md1-s1 kernel: Pid: 39269, comm: mdt02_018 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:53:12 fir-md1-s1 kernel: Call Trace: Dec 12 06:53:12 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:53:12 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:53:12 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:53:12 fir-md1-s1 kernel: Pid: 39371, comm: mdt02_033 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:53:12 fir-md1-s1 kernel: Call Trace: Dec 12 06:53:12 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:53:12 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:53:12 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:53:12 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:53:12 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:54:32 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 06:54:32 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 06:55:53 fir-md1-s1 kernel: LNet: Service thread pid 38897 was inactive for 763.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 06:55:53 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 06:55:53 fir-md1-s1 kernel: Pid: 38897, comm: mdt03_001 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 06:55:53 fir-md1-s1 kernel: Call Trace: Dec 12 06:55:53 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 06:55:53 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 06:55:53 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 06:55:53 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 06:55:53 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 06:55:53 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 06:55:53 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 06:55:53 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 06:55:53 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 06:55:53 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576162553.38897 Dec 12 06:56:30 fir-md1-s1 kernel: LNet: Service thread pid 39371 completed after 799.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 06:56:30 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576162242/real 1576162242] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576162843 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 07:00:43 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Dec 12 07:04:50 fir-md1-s1 kernel: LustreError: 39229:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576162790, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8855f753d100/0xc3c20c06c1bcc2c7 lrc: 3/0,1 mode: --/PW res: [0x200039577:0x11b6:0x0].0x8ec40924 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39229 timeout: 0 lvb_type: 0 Dec 12 07:04:50 fir-md1-s1 kernel: LustreError: 39229:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 13 previous similar messages Dec 12 07:04:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client c104d961-ddd0-a5eb-3382-4ecbd88b591c (at 10.8.18.16@o2ib6) reconnecting Dec 12 07:04:50 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Dec 12 07:04:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4fb4463b-4df1-b2ca-bcaf-03821e29c498 (at 10.8.8.31@o2ib6) reconnecting Dec 12 07:04:51 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Dec 12 07:07:09 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 07:07:09 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576162843/real 1576162843] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576163444 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 07:10:44 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Dec 12 07:12:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3bd651a1-07e6-0cec-1800-45156860eb64 (at 10.9.110.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bdf49ec00, cur 1576163522 expire 1576163372 last 1576163295 Dec 12 07:12:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 07:12:02 fir-md1-s1 kernel: LustreError: 39384:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.110.39@o2ib4) failed to reply to blocking AST (req@ffff885db815e300 x1652542932057200 status 0 rc -5), evict it ns: mdt-fir-MDT0000_UUID lock: ffff885861b9e300/0xc3c20c06c1c07621 lrc: 4/0,0 mode: PR/PR res: [0x20003963a:0x2ae:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.110.39@o2ib4 remote: 0x3535cd2e4ab6440 expref: 419 pid: 39328 timeout: 177424 lvb_type: 0 Dec 12 07:12:02 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.110.39@o2ib4 was evicted due to a lock blocking callback time out: rc -5 Dec 12 07:12:10 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 29b6614b-d9b6-3c4b-cd6c-79cb079428c5 (at 10.9.110.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885b73820c00, cur 1576163530 expire 1576163380 last 1576163303 Dec 12 07:19:46 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 07:19:46 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576163444/real 1576163444] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576164045 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 07:20:45 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 07:23:29 fir-md1-s1 kernel: LustreError: 39230:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576163909, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8863aabea880/0xc3c20c06c1c583ca lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39230 timeout: 0 lvb_type: 0 Dec 12 07:23:29 fir-md1-s1 kernel: LustreError: 39230:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 19 previous similar messages Dec 12 07:24:05 fir-md1-s1 kernel: LustreError: 39326:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576163945, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff88792d670000/0xc3c20c06c1c5bab9 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39326 timeout: 0 lvb_type: 0 Dec 12 07:24:08 fir-md1-s1 kernel: LNet: Service thread pid 39341 was inactive for 360.81s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 07:24:08 fir-md1-s1 kernel: Pid: 39341, comm: mdt02_025 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 07:24:08 fir-md1-s1 kernel: Call Trace: Dec 12 07:24:08 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 07:24:08 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 07:24:08 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 07:24:08 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 07:24:08 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 07:24:08 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 07:24:08 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 07:24:08 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 07:24:08 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 07:24:08 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 07:24:08 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 07:24:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164248.39341 Dec 12 07:24:30 fir-md1-s1 kernel: LNet: Service thread pid 39258 was inactive for 361.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 07:24:30 fir-md1-s1 kernel: Pid: 39258, comm: mdt02_014 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 07:24:30 fir-md1-s1 kernel: Call Trace: Dec 12 07:24:30 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 07:24:30 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 07:24:30 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 07:24:30 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 07:24:30 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 07:24:30 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 07:24:30 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 07:24:31 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 07:24:31 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 07:24:31 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 07:24:31 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 07:24:31 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 07:24:31 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164271.39258 Dec 12 07:24:50 fir-md1-s1 kernel: LNet: Service thread pid 39341 completed after 403.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 07:25:05 fir-md1-s1 kernel: LNet: Service thread pid 39384 was inactive for 414.94s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 07:25:05 fir-md1-s1 kernel: Pid: 39384, comm: mdt01_032 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 07:25:05 fir-md1-s1 kernel: Call Trace: Dec 12 07:25:05 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 07:25:05 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 07:25:05 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 07:25:05 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 07:25:05 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 07:25:05 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 07:25:05 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 07:25:05 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164305.39384 Dec 12 07:25:16 fir-md1-s1 kernel: LNet: Service thread pid 39250 was inactive for 426.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 07:25:16 fir-md1-s1 kernel: Pid: 39250, comm: mdt00_010 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 07:25:16 fir-md1-s1 kernel: Call Trace: Dec 12 07:25:16 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 07:25:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 07:25:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 07:25:17 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 07:25:17 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 07:25:17 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 07:25:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 07:25:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 07:25:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 07:25:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 07:25:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 07:25:17 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164317.39250 Dec 12 07:25:19 fir-md1-s1 kernel: Pid: 39364, comm: mdt00_030 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 07:25:19 fir-md1-s1 kernel: Call Trace: Dec 12 07:25:19 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 07:25:19 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 07:25:19 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 07:25:19 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 07:25:19 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 07:25:19 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 07:25:19 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 07:25:19 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 07:25:19 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 07:25:19 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 07:25:19 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 07:25:19 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164319.39364 Dec 12 07:25:25 fir-md1-s1 kernel: LNet: Service thread pid 39230 was inactive for 415.28s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 07:25:25 fir-md1-s1 kernel: LNet: Skipped 9 previous similar messages Dec 12 07:25:25 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164325.39230 Dec 12 07:25:51 fir-md1-s1 kernel: LNet: Service thread pid 39268 was inactive for 361.02s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 07:25:51 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164351.39268 Dec 12 07:26:06 fir-md1-s1 kernel: LNet: Service thread pid 39382 was inactive for 414.45s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 07:26:06 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576164366.39382 Dec 12 07:26:21 fir-md1-s1 kernel: LustreError: 39328:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576164081, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885d6f11b600/0xc3c20c06c1c6f3d6 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 29 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39328 timeout: 0 lvb_type: 0 Dec 12 07:26:21 fir-md1-s1 kernel: LustreError: 39328:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 07:26:30 fir-md1-s1 kernel: LNet: Service thread pid 39364 completed after 499.27s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 07:26:30 fir-md1-s1 kernel: LNet: Skipped 6 previous similar messages Dec 12 07:27:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client c1504d4c-7504-c251-de3c-6f26c7b8e7d5 (at 10.9.102.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba92a7400, cur 1576164458 expire 1576164308 last 1576164231 Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576164045/real 1576164045] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576164646 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 07:30:46 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 07:32:23 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -107 Dec 12 07:32:23 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 07:39:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3c020cd0-089d-acb1-e879-86429192cebf (at 10.8.27.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba97b0000, cur 1576165170 expire 1576165020 last 1576164943 Dec 12 07:39:30 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576164646/real 1576164646] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576165247 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection to fir-OST0056 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: fir-OST0056-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 07:40:47 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 07:43:24 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 07:43:24 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.115@o2ib7 (6): c: 0, oc: 0, rc: 8 Dec 12 07:43:24 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 07:43:31 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.115@o2ib7: 1 seconds Dec 12 07:43:31 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 59 previous similar messages Dec 12 07:43:57 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.8.19@o2ib6, removing former export from same NID Dec 12 07:43:58 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.102.58@o2ib4, removing former export from same NID Dec 12 07:43:58 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 07:44:01 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.108.9@o2ib4, removing former export from same NID Dec 12 07:44:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.110.63@o2ib4, removing former export from same NID Dec 12 07:44:04 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 07:44:08 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.103.13@o2ib4, removing former export from same NID Dec 12 07:44:08 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 07:44:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 0713f3a9-f297-cd73-69ad-d70a0f44846f (at 10.9.104.62@o2ib4) reconnecting Dec 12 07:44:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 09403296-99cb-0352-a342-f41333f5025e (at 10.9.107.69@o2ib4) reconnecting Dec 12 07:44:16 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.105.9@o2ib4, removing former export from same NID Dec 12 07:44:16 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Dec 12 07:44:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client ee8a44d1-a255-3904-d785-781d851ce5cc (at 10.9.107.65@o2ib4) reconnecting Dec 12 07:44:24 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.211@o2ib7 added to recovery queue. Health = 900 Dec 12 07:44:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 91b198e6-ce9d-6e88-4a8a-d97e9eaae698 (at 10.8.26.36@o2ib6) reconnecting Dec 12 07:44:32 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.112.2@o2ib4, removing former export from same NID Dec 12 07:44:32 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Dec 12 07:44:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.108.14@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:44:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d021ee3d-37fa-4 (at 10.8.28.7@o2ib6) reconnecting Dec 12 07:44:37 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 07:44:39 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 Dec 12 07:44:43 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 07:44:43 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 6 previous similar messages Dec 12 07:44:48 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.115@o2ib7: 1 seconds Dec 12 07:44:48 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 19 previous similar messages Dec 12 07:44:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b9d3876a-bb59-06ed-126e-3827677a4444 (at 10.9.104.3@o2ib4) reconnecting Dec 12 07:44:51 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 07:44:54 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.212@o2ib7 added to recovery queue. Health = 900 Dec 12 07:44:54 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 1 previous similar message Dec 12 07:44:59 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 Dec 12 07:44:59 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 1 previous similar message Dec 12 07:45:00 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST005e-osc-MDT0000: cannot cleanup orphans: rc = -11 Dec 12 07:45:00 fir-md1-s1 kernel: LustreError: 40821:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Dec 12 07:45:04 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.105.65@o2ib4, removing former export from same NID Dec 12 07:45:04 fir-md1-s1 kernel: Lustre: Skipped 106 previous similar messages Dec 12 07:45:05 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 Dec 12 07:45:05 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 2 previous similar messages Dec 12 07:45:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3cd8d17d-c015-47f4-5929-0823e94a86fa (at 10.9.110.34@o2ib4) reconnecting Dec 12 07:45:08 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Dec 12 07:45:14 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.104.8@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:45:28 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.7.18@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:45:28 fir-md1-s1 kernel: LustreError: Skipped 1 previous similar message Dec 12 07:45:39 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.203@o2ib7 added to recovery queue. Health = 900 Dec 12 07:45:39 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 1 previous similar message Dec 12 07:45:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 54a4e18f-2dbf-9330-244f-d38b0011d1d4 (at 10.9.103.65@o2ib4) reconnecting Dec 12 07:45:41 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Dec 12 07:45:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.103.46@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:45:49 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 12 07:45:50 fir-md1-s1 kernel: LustreError: 96473:0:(ldlm_lib.c:3256:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff886bfd423850 x1648782579006384/t0(0) o256->1b4e0033-5092-73e2-ee39-e46ff9b43fe9@10.8.28.8@o2ib6:412/0 lens 304/240 e 0 to 0 dl 1576165592 ref 1 fl Interpret:/0/0 rc 0/0 Dec 12 07:45:51 fir-md1-s1 kernel: LustreError: 96472:0:(ldlm_lib.c:3256:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff888be926b850 x1649567732887344/t0(0) o256->351465b9-9a15-eaaf-e2ff-273afe28ffed@10.9.104.27@o2ib4:413/0 lens 304/240 e 0 to 0 dl 1576165593 ref 1 fl Interpret:/0/0 rc 0/0 Dec 12 07:45:51 fir-md1-s1 kernel: LustreError: 96472:0:(ldlm_lib.c:3256:target_bulk_io()) Skipped 1 previous similar message Dec 12 07:46:04 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 Dec 12 07:46:04 fir-md1-s1 kernel: LNetError: 38679:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 4 previous similar messages Dec 12 07:46:09 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.106.19@o2ib4, removing former export from same NID Dec 12 07:46:09 fir-md1-s1 kernel: Lustre: Skipped 206 previous similar messages Dec 12 07:46:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client fir-MDT0000-lwp-OST0054_UUID (at 10.0.10.115@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886be695e800, cur 1576165583 expire 1576165433 last 1576165356 Dec 12 07:46:23 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 07:46:27 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.108.50@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:46:27 fir-md1-s1 kernel: LustreError: Skipped 82 previous similar messages Dec 12 07:46:30 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b3aef711-fb13-218a-11cf-7e4e4d6f4a51 (at 10.0.10.115@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bdddea400, cur 1576165590 expire 1576165440 last 1576165363 Dec 12 07:46:30 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 07:46:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4e8251e5-eb6b-473d-1b55-6cf68aeb84d4 (at 10.9.105.59@o2ib4) reconnecting Dec 12 07:46:56 fir-md1-s1 kernel: Lustre: Skipped 44 previous similar messages Dec 12 07:47:17 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 07:47:17 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 8 previous similar messages Dec 12 07:47:19 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.115@o2ib7: 2 seconds Dec 12 07:47:19 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 45 previous similar messages Dec 12 07:49:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client cce2fc1a-d500-0fbb-5491-2d32b40f4df2 (at 10.8.20.10@o2ib6) reconnecting Dec 12 07:49:07 fir-md1-s1 kernel: Lustre: Skipped 64 previous similar messages Dec 12 07:50:15 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Dec 12 07:50:15 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.115@o2ib7 (7): c: 0, oc: 0, rc: 8 Dec 12 07:50:26 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.107.9@o2ib4, removing former export from same NID Dec 12 07:50:26 fir-md1-s1 kernel: Lustre: Skipped 92 previous similar messages Dec 12 07:50:48 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576165247/real 1576165247] req@ffff886bcde4ad00 x1652542919103808/t0(0) o6->fir-OST0056-osc-MDT0000@10.0.10.115@o2ib7:28/4 lens 544/432 e 4 to 1 dl 1576165848 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 07:50:48 fir-md1-s1 kernel: Lustre: 38698:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Dec 12 07:51:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.107.9@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:51:37 fir-md1-s1 kernel: LustreError: Skipped 118 previous similar messages Dec 12 07:51:51 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.107.9@o2ib4) Dec 12 07:51:51 fir-md1-s1 kernel: Lustre: Skipped 629 previous similar messages Dec 12 07:52:07 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e3242d1f-bdca-4a42-9a91-79e078549196 (at 10.9.103.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bdde1d400, cur 1576165927 expire 1576165777 last 1576165700 Dec 12 07:55:17 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.107.9@o2ib4, removing former export from same NID Dec 12 07:55:17 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 07:55:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d833ee08-9e03-4 (at 10.9.107.9@o2ib4) reconnecting Dec 12 07:55:23 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 07:55:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.107.9@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 07:55:29 fir-md1-s1 kernel: LustreError: Skipped 2 previous similar messages Dec 12 07:57:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 4c5e6f33-2d0c-f229-3fed-c30688bbed72 (at 10.9.116.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba902d000, cur 1576166276 expire 1576166126 last 1576166049 Dec 12 07:57:56 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 07:58:09 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5540cff7-da1b-d42f-e90b-ff6d64672f1b (at 10.9.102.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be08d5000, cur 1576166289 expire 1576166139 last 1576166062 Dec 12 07:58:09 fir-md1-s1 kernel: Lustre: Skipped 18 previous similar messages Dec 12 07:59:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 29e66763-b95c-3d3e-5532-53facc0d6b7a (at 10.9.109.32@o2ib4) in 220 seconds. I think it's dead, and I am evicting it. exp ffff887ba9607400, cur 1576166352 expire 1576166202 last 1576166132 Dec 12 07:59:12 fir-md1-s1 kernel: Lustre: Skipped 20 previous similar messages Dec 12 07:59:23 fir-md1-s1 kernel: LustreError: 42578:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166063, 300s ago); not entering recovery in server code, just going back to sleep ns: MGS lock: ffff888b827572c0/0xc3c20c06c1d5896f lrc: 3/0,1 mode: --/EX res: [0x726966:0x2:0x0].0x0 rrc: 1257 type: PLN flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 42578 timeout: 0 lvb_type: 0 Dec 12 07:59:29 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 07:59:29 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166069, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888bf5e3a400/0xc3c20c06c1d5a06e lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c1d5a075 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 07:59:29 fir-md1-s1 kernel: LustreError: 96734:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bb2664180) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:01:42 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.107.9@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 08:01:42 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Dec 12 08:01:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.107.9@o2ib4) Dec 12 08:01:51 fir-md1-s1 kernel: Lustre: Skipped 1254 previous similar messages Dec 12 08:02:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 646257db-4a10-1d7d-1435-2f2425d1bdb2 (at 10.8.18.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bdd4b2800, cur 1576166531 expire 1576166381 last 1576166304 Dec 12 08:02:11 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 08:04:33 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.8.23.20@o2ib6, removing former export from same NID Dec 12 08:04:33 fir-md1-s1 kernel: Lustre: Skipped 1242 previous similar messages Dec 12 08:04:34 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:04:34 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166374, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888bd0ec1200/0xc3c20c06c1d7745d lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c1d77464 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:04:34 fir-md1-s1 kernel: LustreError: 96855:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bb2664840) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:04:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d833ee08-9e03-4 (at 10.9.107.9@o2ib4) reconnecting Dec 12 08:04:49 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 08:09:40 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:09:40 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166680, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff887ad67e8480/0xc3c20c06c1d97ca9 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c1d97cb0 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:09:40 fir-md1-s1 kernel: LustreError: 96937:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bb2665ec0) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:10:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client db44fcc6-df61-0a83-7c51-af3e9a77d479 (at 10.8.7.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9776c00, cur 1576167005 expire 1576166855 last 1576166778 Dec 12 08:10:05 fir-md1-s1 kernel: Lustre: Skipped 29 previous similar messages Dec 12 08:10:42 fir-md1-s1 kernel: LustreError: 39253:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166742, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff886a6ee64800/0xc3c20c06c1da8f4d lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 32 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39253 timeout: 0 lvb_type: 0 Dec 12 08:11:12 fir-md1-s1 kernel: LustreError: 39416:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166772, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887ba4700d80/0xc3c20c06c1dab7f6 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 32 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39416 timeout: 0 lvb_type: 0 Dec 12 08:11:22 fir-md1-s1 kernel: LustreError: 39250:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166782, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852d79d4ec0/0xc3c20c06c1dabd8a lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 32 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39250 timeout: 0 lvb_type: 0 Dec 12 08:11:59 fir-md1-s1 kernel: Lustre: fir-OST0058-osc-MDT0000: Connection restored to 10.0.10.115@o2ib7 (at 10.0.10.115@o2ib7) Dec 12 08:11:59 fir-md1-s1 kernel: Lustre: Skipped 2499 previous similar messages Dec 12 08:12:01 fir-md1-s1 kernel: LustreError: 39265:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166821, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8877ada68000/0xc3c20c06c1db5f71 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 35 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39265 timeout: 0 lvb_type: 0 Dec 12 08:12:08 fir-md1-s1 kernel: LNet: Service thread pid 39268 was inactive for 411.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 08:12:08 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 08:12:08 fir-md1-s1 kernel: Pid: 39268, comm: mdt02_017 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 08:12:08 fir-md1-s1 kernel: Call Trace: Dec 12 08:12:08 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 08:12:08 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 08:12:08 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 08:12:08 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 08:12:08 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 08:12:08 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 08:12:08 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 08:12:08 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 08:12:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 08:12:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 08:12:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167129.39268 Dec 12 08:12:09 fir-md1-s1 kernel: Pid: 39269, comm: mdt02_018 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 08:12:09 fir-md1-s1 kernel: Call Trace: Dec 12 08:12:09 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 08:12:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 08:12:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 08:12:09 fir-md1-s1 kernel: Pid: 39375, comm: mdt02_034 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 08:12:09 fir-md1-s1 kernel: Call Trace: Dec 12 08:12:09 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 08:12:09 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 08:12:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 08:12:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 08:12:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 08:12:10 fir-md1-s1 kernel: LNet: Service thread pid 39382 was inactive for 412.93s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 08:12:10 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 08:12:10 fir-md1-s1 kernel: Pid: 39382, comm: mdt01_031 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 08:12:10 fir-md1-s1 kernel: Call Trace: Dec 12 08:12:10 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 08:12:10 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 08:12:10 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 08:12:10 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 08:12:10 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 08:12:10 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 08:12:10 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 08:12:11 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 08:12:11 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 08:12:11 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167131.39382 Dec 12 08:12:11 fir-md1-s1 kernel: Pid: 39367, comm: mdt03_028 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 08:12:11 fir-md1-s1 kernel: Call Trace: Dec 12 08:12:11 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 08:12:11 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 08:12:11 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 08:12:11 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 08:12:11 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 08:12:11 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 08:12:11 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 08:12:11 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 08:12:11 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 08:12:16 fir-md1-s1 kernel: LNet: Service thread pid 39268 completed after 418.39s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 08:12:16 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 08:12:17 fir-md1-s1 kernel: LNet: Service thread pid 38895 was inactive for 411.25s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:12:17 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167137.38895 Dec 12 08:12:18 fir-md1-s1 kernel: LNet: Service thread pid 39258 was inactive for 410.89s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:12:18 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167138.39258 Dec 12 08:12:25 fir-md1-s1 kernel: LNet: Service thread pid 39339 was inactive for 413.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:12:25 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167145.39339 Dec 12 08:12:28 fir-md1-s1 kernel: LNet: Service thread pid 39375 completed after 430.17s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 08:12:28 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 08:12:35 fir-md1-s1 kernel: LNet: Service thread pid 39253 was inactive for 412.57s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:12:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167155.39253 Dec 12 08:12:51 fir-md1-s1 kernel: LNet: Service thread pid 39367 completed after 453.38s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 08:13:03 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.107.9@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 08:13:03 fir-md1-s1 kernel: LustreError: Skipped 10 previous similar messages Dec 12 08:13:04 fir-md1-s1 kernel: LNet: Service thread pid 39416 was inactive for 411.28s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:13:04 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 08:13:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167184.39416 Dec 12 08:13:15 fir-md1-s1 kernel: LNet: Service thread pid 39250 was inactive for 412.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:13:15 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167195.39250 Dec 12 08:13:45 fir-md1-s1 kernel: LNet: Service thread pid 39341 was inactive for 411.51s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:13:45 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167225.39341 Dec 12 08:13:48 fir-md1-s1 kernel: LustreError: 39378:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166928, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8875b2a03f00/0xc3c20c06c1dc286b lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 36 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39378 timeout: 0 lvb_type: 0 Dec 12 08:13:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167232.39265 Dec 12 08:14:01 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167241.39346 Dec 12 08:14:03 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167243.39217 Dec 12 08:14:24 fir-md1-s1 kernel: LNet: Service thread pid 38891 was inactive for 413.14s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:14:24 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 08:14:24 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167264.38891 Dec 12 08:14:25 fir-md1-s1 kernel: Lustre: 39215:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576167258/real 1576167258] req@ffff88785e337080 x1652542933315888/t0(0) o104->fir-MDT0000@10.9.115.1@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576167265 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Dec 12 08:14:25 fir-md1-s1 kernel: Lustre: 39215:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Dec 12 08:14:31 fir-md1-s1 kernel: LNet: Service thread pid 39258 completed after 544.49s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 08:14:42 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.107.9@o2ib4, removing former export from same NID Dec 12 08:14:42 fir-md1-s1 kernel: Lustre: Skipped 2496 previous similar messages Dec 12 08:14:50 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:14:50 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576166990, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888b89be4ec0/0xc3c20c06c1dccd0e lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c1dccd15 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:14:50 fir-md1-s1 kernel: LustreError: 97158:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888b8aab4240) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:15:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d59b4a25-94cd-9118-509c-0144bd0df5bb (at 10.9.109.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886be3ac3000, cur 1576167311 expire 1576167161 last 1576167084 Dec 12 08:15:11 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 08:15:20 fir-md1-s1 kernel: Lustre: 39330:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88799bae8480 x1649614765426528/t0(0) o101->5b41e348-8633-a21d-46d9-7918979d9d25@10.9.104.19@o2ib4:635/0 lens 376/1600 e 14 to 0 dl 1576167325 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 08:15:20 fir-md1-s1 kernel: Lustre: 39330:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Dec 12 08:15:26 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 5b41e348-8633-a21d-46d9-7918979d9d25 (at 10.9.104.19@o2ib4) reconnecting Dec 12 08:15:26 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 08:15:27 fir-md1-s1 kernel: Lustre: 39351:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff885b8bbfad00 x1649498306499712/t0(0) o101->cf0dcba8-ff55-c75d-2ce2-0d11bb83fb82@10.9.102.21@o2ib4:642/0 lens 376/1600 e 14 to 0 dl 1576167332 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 08:15:37 fir-md1-s1 kernel: Lustre: 39323:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff886becfb9b00 x1648439898262768/t0(0) o101->5e845a5e-9c00-cc58-79df-d4d75fd3c1a1@10.8.27.4@o2ib6:652/0 lens 1792/3288 e 11 to 0 dl 1576167342 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 08:15:39 fir-md1-s1 kernel: LNet: Service thread pid 39378 was inactive for 411.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 08:15:39 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167339.39378 Dec 12 08:15:40 fir-md1-s1 kernel: LustreError: 39325:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576167040, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885bd5349b00/0xc3c20c06c1dd425c lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 38 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39325 timeout: 0 lvb_type: 0 Dec 12 08:15:40 fir-md1-s1 kernel: LustreError: 39325:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Dec 12 08:15:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576167356.39371 Dec 12 08:16:08 fir-md1-s1 kernel: Lustre: 39242:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff887b9b6eb600 x1648417756486160/t0(0) o101->4e97c29c-283b-4253-402d-db9d46beedd7@10.9.101.39@o2ib4:682/0 lens 600/3264 e 6 to 0 dl 1576167372 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 08:16:08 fir-md1-s1 kernel: Lustre: 39242:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Dec 12 08:16:11 fir-md1-s1 kernel: LNet: Service thread pid 39339 completed after 639.67s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 08:16:11 fir-md1-s1 kernel: LNet: Skipped 11 previous similar messages Dec 12 08:19:58 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 742b2cbc-624e-86be-da90-400c9fd59825 (at 10.9.114.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc6bf5000, cur 1576167598 expire 1576167448 last 1576167371 Dec 12 08:19:58 fir-md1-s1 kernel: Lustre: Skipped 67 previous similar messages Dec 12 08:19:58 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:19:58 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576167298, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888b8278a640/0xc3c20c06c1e92ca3 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c1e92caa expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:19:58 fir-md1-s1 kernel: LustreError: 97304:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bca7fb500) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:20:40 fir-md1-s1 kernel: Lustre: 38715:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576167633/real 1576167633] req@ffff887534b33180 x1652542933462384/t0(0) o41->fir-MDT0003-osp-MDT0000@10.0.10.54@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1576167640 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Dec 12 08:20:40 fir-md1-s1 kernel: Lustre: 38715:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Dec 12 08:20:40 fir-md1-s1 kernel: Lustre: fir-MDT0003-osp-MDT0000: Connection to fir-MDT0003 (at 10.0.10.54@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 08:20:40 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Dec 12 08:21:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 70ca4d0d-57d2-4178-fe01-a31f45306b60 (at 10.9.112.16@o2ib4) Dec 12 08:21:59 fir-md1-s1 kernel: Lustre: Skipped 2446 previous similar messages Dec 12 08:22:27 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Dec 12 08:22:27 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (106): c: 4, oc: 0, rc: 8 Dec 12 08:22:27 fir-md1-s1 kernel: LNetError: 38671:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.54@o2ib7 added to recovery queue. Health = 900 Dec 12 08:22:28 fir-md1-s1 kernel: LNetError: 96587:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 08:22:28 fir-md1-s1 kernel: LNetError: 96587:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 12 previous similar messages Dec 12 08:23:04 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.104.25@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 08:23:04 fir-md1-s1 kernel: LustreError: Skipped 143 previous similar messages Dec 12 08:23:09 fir-md1-s1 kernel: LNetError: 96587:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 08:23:09 fir-md1-s1 kernel: LNetError: 96587:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 4 previous similar messages Dec 12 08:23:19 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1576167797/real 1576167799] req@ffff8858ccf1d100 x1652542933542448/t0(0) o105->MGS@10.0.10.54@o2ib7:15/16 lens 304/224 e 0 to 1 dl 1576167804 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Dec 12 08:23:29 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 1 seconds Dec 12 08:23:29 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 56 previous similar messages Dec 12 08:23:53 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Dec 12 08:23:53 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (9): c: 0, oc: 0, rc: 8 Dec 12 08:24:04 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 08:24:04 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (10): c: 0, oc: 0, rc: 8 Dec 12 08:24:13 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 08:24:13 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (5): c: 0, oc: 0, rc: 8 Dec 12 08:24:27 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 08:24:27 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (5): c: 0, oc: 0, rc: 8 Dec 12 08:24:27 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 08:24:27 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 12 previous similar messages Dec 12 08:24:42 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Dec 12 08:24:42 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (6): c: 0, oc: 0, rc: 8 Dec 12 08:25:10 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Dec 12 08:25:10 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Skipped 1 previous similar message Dec 12 08:25:10 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (6): c: 0, oc: 0, rc: 8 Dec 12 08:25:10 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Skipped 1 previous similar message Dec 12 08:25:11 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 2 seconds Dec 12 08:25:11 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 14 previous similar messages Dec 12 08:25:47 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 08:25:47 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Dec 12 08:25:47 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (7): c: 0, oc: 0, rc: 8 Dec 12 08:25:47 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Skipped 2 previous similar messages Dec 12 08:26:56 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 08:26:56 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Skipped 4 previous similar messages Dec 12 08:26:56 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (6): c: 0, oc: 0, rc: 8 Dec 12 08:26:56 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Skipped 4 previous similar messages Dec 12 08:27:11 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 08:27:11 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 12 previous similar messages Dec 12 08:28:20 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576168093/real 1576168093] req@ffff88560703b600 x1652542933636560/t0(0) o105->MGS@10.0.10.54@o2ib7:15/16 lens 304/224 e 0 to 1 dl 1576168100 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 08:28:20 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 382 previous similar messages Dec 12 08:28:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.54@o2ib7, removing former export from same NID Dec 12 08:28:42 fir-md1-s1 kernel: Lustre: Skipped 2426 previous similar messages Dec 12 08:28:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client a1acf167-afde-6f5a-879d-1a7c0814f282 (at 10.9.117.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9601c00, cur 1576168127 expire 1576167977 last 1576167900 Dec 12 08:28:47 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 12 08:31:16 fir-md1-s1 kernel: LustreError: 42578:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.107.9@o2ib4) failed to reply to blocking AST (req@ffff888b75b51f80 x1652542933790336 status 0 rc -110), evict it ns: MGS lock: ffff888bf2303840/0xc3c20c06c206bf5b lrc: 4/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 1227 type: PLN flags: 0x40000400000020 nid: 10.9.107.9@o2ib4 remote: 0x1f531de89b55b22a expref: 17 pid: 42501 timeout: 0 lvb_type: 0 Dec 12 08:31:16 fir-md1-s1 kernel: LustreError: 138-a: MGS: A client on nid 10.9.107.9@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Dec 12 08:31:16 fir-md1-s1 kernel: LustreError: 38883:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1576168276s: evicting client at 10.9.107.9@o2ib4 ns: MGS lock: ffff888bf2303840/0xc3c20c06c206bf5b lrc: 4/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 1228 type: PLN flags: 0x40000400000020 nid: 10.9.107.9@o2ib4 remote: 0x1f531de89b55b22a expref: 18 pid: 42501 timeout: 0 lvb_type: 0 Dec 12 08:32:59 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.104.42@o2ib4) Dec 12 08:32:59 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages Dec 12 08:33:49 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:33:49 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576168129, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888be0300480/0xc3c20c06c2758fad lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c2758fb4 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:33:49 fir-md1-s1 kernel: LustreError: 97965:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888b813400c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:36:17 fir-md1-s1 kernel: LustreError: 42578:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576168276, 300s ago); not entering recovery in server code, just going back to sleep ns: MGS lock: ffff888bde6b2ac0/0xc3c20c06c2750c27 lrc: 3/0,1 mode: --/EX res: [0x726966:0x2:0x0].0x0 rrc: 2443 type: PLN flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 42578 timeout: 0 lvb_type: 0 Dec 12 08:38:20 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576168693/real 1576168693] req@ffff88571b6bbf00 x1652542933546944/t0(0) o105->MGS@10.9.107.9@o2ib4:15/16 lens 304/224 e 0 to 1 dl 1576168700 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 08:38:20 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 551 previous similar messages Dec 12 08:38:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client dec5062c-f101-0dc5-128b-72e40bd60a5a (at 10.9.112.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bde256400, cur 1576168731 expire 1576168581 last 1576168504 Dec 12 08:38:51 fir-md1-s1 kernel: Lustre: Skipped 7 previous similar messages Dec 12 08:38:52 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.105.17@o2ib4, removing former export from same NID Dec 12 08:38:52 fir-md1-s1 kernel: Lustre: Skipped 1218 previous similar messages Dec 12 08:38:57 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:38:57 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576168436, 301s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8863c6899b00/0xc3c20c06c28a681c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c28a6823 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:38:57 fir-md1-s1 kernel: LustreError: 98064:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bb17a7ec0) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:43:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.117.11@o2ib4) Dec 12 08:43:06 fir-md1-s1 kernel: Lustre: Skipped 2480 previous similar messages Dec 12 08:44:06 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:44:06 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576168746, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888bbaf9dc40/0xc3c20c06c2adfd45 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c2adfd4c expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:44:06 fir-md1-s1 kernel: LustreError: 98237:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888bf8ed7080) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:48:22 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576169295/real 1576169295] req@ffff88571b6bbf00 x1652542933546944/t0(0) o105->MGS@10.9.107.9@o2ib4:15/16 lens 304/224 e 0 to 1 dl 1576169302 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Dec 12 08:48:22 fir-md1-s1 kernel: Lustre: 42120:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 429 previous similar messages Dec 12 08:48:54 fir-md1-s1 kernel: Lustre: MGS: Received new LWP connection from 10.9.101.63@o2ib4, removing former export from same NID Dec 12 08:48:54 fir-md1-s1 kernel: Lustre: Skipped 2462 previous similar messages Dec 12 08:49:11 fir-md1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Dec 12 08:49:11 fir-md1-s1 kernel: LustreError: 38884:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576169051, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff888becfaf2c0/0xc3c20c06c2c33140 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc3c20c06c2c33147 expref: -99 pid: 38884 timeout: 0 lvb_type: 0 Dec 12 08:49:11 fir-md1-s1 kernel: LustreError: 98325:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff888b86b8e0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: 42120:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.107.9@o2ib4) returned error from completion AST (req@ffff88571b6bb600 x1652542933546976 status -107 rc -107), evict it ns: MGS lock: ffff888bf5dd1680/0xc3c20c06c20386c0 lrc: 3/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 6164 type: PLN flags: 0x40000400000020 nid: 10.9.107.9@o2ib4 remote: 0x1f531de89b55b1f9 expref: 17 pid: 42580 timeout: 0 lvb_type: 0 Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: 42120:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 4 previous similar messages Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: 138-a: MGS: A client on nid 10.9.107.9@o2ib4 was evicted due to a lock completion callback time out: rc -107 Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: 38883:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1576169435s: evicting client at 10.9.107.9@o2ib4 ns: MGS lock: ffff888bf5dd1680/0xc3c20c06c20386c0 lrc: 3/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 6165 type: PLN flags: 0x40000400000020 nid: 10.9.107.9@o2ib4 remote: 0x1f531de89b55b1f9 expref: 18 pid: 42580 timeout: 0 lvb_type: 0 Dec 12 08:50:35 fir-md1-s1 kernel: LustreError: 38883:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Dec 12 08:53:18 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.116.11@o2ib4) Dec 12 08:53:18 fir-md1-s1 kernel: Lustre: Skipped 2531 previous similar messages Dec 12 08:58:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 295209bb-0224-d868-bd7c-cd75c3b19a1c (at 10.8.18.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9359000, cur 1576169881 expire 1576169731 last 1576169654 Dec 12 08:58:01 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 09:04:49 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.115.12@o2ib4) Dec 12 09:04:49 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Dec 12 09:15:01 fir-md1-s1 kernel: Lustre: MGS: Connection restored to b9b67222-dc5d-c9e8-945b-377220afc943 (at 10.8.20.25@o2ib6) Dec 12 09:15:01 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 09:22:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 75167b5d-e2d7-d704-ea07-95d8feb377a6 (at 10.9.102.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba90a5800, cur 1576171361 expire 1576171211 last 1576171134 Dec 12 09:22:41 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 09:25:16 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 55529a98-2a28-a963-7acd-1b84cd50762d (at 10.8.7.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf5a04400, cur 1576171516 expire 1576171366 last 1576171289 Dec 12 09:25:16 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 09:29:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3fa61b7b-3364-0c3e-efb9-55ce1343c799 (at 10.8.23.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888c3fc72c00, cur 1576171791 expire 1576171641 last 1576171564 Dec 12 09:29:51 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 09:32:10 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 295209bb-0224-d868-bd7c-cd75c3b19a1c (at 10.8.18.20@o2ib6) Dec 12 09:32:10 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 09:40:11 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 9c024261-121c-46e9-5dee-41ee02e3e326 (at 10.8.18.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885be4453400, cur 1576172411 expire 1576172261 last 1576172184 Dec 12 09:40:11 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 09:51:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client ef78dfe0-80b9-391e-81c2-9236655a36fe (at 10.9.103.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bed3da800, cur 1576173069 expire 1576172919 last 1576172842 Dec 12 09:51:09 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 09:54:13 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.104.7@o2ib4) Dec 12 09:54:13 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 10:01:25 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 75167b5d-e2d7-d704-ea07-95d8feb377a6 (at 10.9.102.1@o2ib4) Dec 12 10:01:25 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 10:04:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 3fa61b7b-3364-0c3e-efb9-55ce1343c799 (at 10.8.23.34@o2ib6) Dec 12 10:04:55 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 10:06:33 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d84068f9-facb-1706-cbe7-745525e4a5c1 (at 10.8.27.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc761c400, cur 1576173993 expire 1576173843 last 1576173766 Dec 12 10:06:33 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 10:16:12 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ee8a8d10-65c2-ae96-bc67-9f6bae32e110 (at 10.8.18.18@o2ib6) Dec 12 10:16:12 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 10:26:37 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 5708211f-0df6-e95b-8bc0-a86ba2362e40 (at 10.9.101.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be1739400, cur 1576175197 expire 1576175047 last 1576174970 Dec 12 10:26:37 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 10:41:26 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 227d7a25-50be-a469-9b6d-83846499cd76 (at 10.8.27.14@o2ib6) Dec 12 10:41:26 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 10:49:38 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.116.14@o2ib4) Dec 12 10:49:38 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 10:50:23 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 5d110741-f52f-a556-c0fd-775bc1eebbda (at 10.9.105.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba92e7400, cur 1576176623 expire 1576176473 last 1576176396 Dec 12 10:50:23 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Dec 12 10:52:52 fir-md1-s1 kernel: LustreError: 39258:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576176472, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887bd17e18c0/0xc3c20c06c69f66cf lrc: 3/1,0 mode: --/PR res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x13/0x0 rrc: 27 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39258 timeout: 0 lvb_type: 0 Dec 12 10:53:30 fir-md1-s1 kernel: LustreError: 38892:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576176510, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885b969398c0/0xc3c20c06c6a1e167 lrc: 3/1,0 mode: --/PR res: [0x200037a5a:0xbae0:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 38892 timeout: 0 lvb_type: 0 Dec 12 10:53:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to ce5ee768-37d0-d480-6e14-e3a25f5ac36c (at 10.9.117.30@o2ib4) Dec 12 10:53:31 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 10:53:49 fir-md1-s1 kernel: LustreError: 88947:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576176529, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff88687b7960c0/0xc3c20c06c6a35857 lrc: 3/0,1 mode: --/CW res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x2/0x0 rrc: 27 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 88947 timeout: 0 lvb_type: 0 Dec 12 10:53:49 fir-md1-s1 kernel: LustreError: 88947:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Dec 12 10:54:22 fir-md1-s1 kernel: LustreError: 97383:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576176562, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff886bf7e2bcc0/0xc3c20c06c6a59714 lrc: 3/0,1 mode: --/CW res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x2/0x0 rrc: 28 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97383 timeout: 0 lvb_type: 0 Dec 12 10:54:31 fir-md1-s1 kernel: LNet: Service thread pid 39258 was inactive for 399.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 10:54:31 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 10:54:31 fir-md1-s1 kernel: Pid: 39258, comm: mdt02_014 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 10:54:31 fir-md1-s1 kernel: Call Trace: Dec 12 10:54:31 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 10:54:31 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 10:54:31 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 10:54:31 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 10:54:31 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 10:54:31 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 10:54:31 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 10:54:31 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 10:54:31 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 10:54:31 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576176871.39258 Dec 12 10:54:49 fir-md1-s1 kernel: LNet: Service thread pid 39336 was inactive for 398.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 10:54:49 fir-md1-s1 kernel: Pid: 39336, comm: mdt02_024 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 10:54:49 fir-md1-s1 kernel: Call Trace: Dec 12 10:54:49 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 10:54:49 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 10:54:49 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 10:54:49 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 10:54:49 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 10:54:49 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 10:54:49 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 10:54:49 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 10:54:49 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 10:54:49 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 10:54:49 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 10:54:49 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576176889.39336 Dec 12 10:54:51 fir-md1-s1 kernel: LNet: Service thread pid 39336 completed after 399.97s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 10:54:51 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 10:54:57 fir-md1-s1 kernel: LustreError: 98146:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576176597, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885b977a1200/0xc3c20c06c6a7b6f2 lrc: 3/1,0 mode: --/PR res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x13/0x0 rrc: 30 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 98146 timeout: 0 lvb_type: 0 Dec 12 10:56:02 fir-md1-s1 kernel: LNet: Service thread pid 39232 was inactive for 450.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 10:56:02 fir-md1-s1 kernel: Pid: 39232, comm: mdt01_014 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 10:56:02 fir-md1-s1 kernel: Call Trace: Dec 12 10:56:02 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 10:56:02 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 10:56:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 10:56:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 10:56:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 10:56:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 10:56:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 10:56:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576176962.39232 Dec 12 10:56:31 fir-md1-s1 kernel: LNet: Service thread pid 39232 completed after 480.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 11:02:41 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.115.12@o2ib4) Dec 12 11:02:41 fir-md1-s1 kernel: Lustre: Skipped 13 previous similar messages Dec 12 11:13:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client e8e18d90-dcac-7195-a7b7-bbaf10be70ce (at 10.9.103.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886be3ac0c00, cur 1576178027 expire 1576177877 last 1576177800 Dec 12 11:13:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 11:20:54 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 5d110741-f52f-a556-c0fd-775bc1eebbda (at 10.9.105.33@o2ib4) Dec 12 11:20:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 11:40:55 fir-md1-s1 kernel: Lustre: MGS: Connection restored to e8e18d90-dcac-7195-a7b7-bbaf10be70ce (at 10.9.103.52@o2ib4) Dec 12 11:40:55 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 11:43:38 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 9a79f7a1-9fac-50b7-c195-aa7bdae4f43f (at 10.8.7.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bf8834400, cur 1576179818 expire 1576179668 last 1576179591 Dec 12 11:43:38 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 11:46:17 fir-md1-s1 kernel: Lustre: 97348:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576179970/real 1576179970] req@ffff886bcde4ec00 x1652542944106224/t0(0) o104->fir-MDT0000@10.9.101.46@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1576179977 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Dec 12 11:46:17 fir-md1-s1 kernel: Lustre: 97348:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 99 previous similar messages Dec 12 11:46:59 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ebc1ca27-139b-33a6-84f1-99529f6e5ea6 (at 10.9.101.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc6bdf800, cur 1576180019 expire 1576179869 last 1576179792 Dec 12 11:46:59 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 11:47:00 fir-md1-s1 kernel: LustreError: 97348:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.101.46@o2ib4) failed to reply to blocking AST (req@ffff886bcde4ec00 x1652542944106224 status 0 rc -5), evict it ns: mdt-fir-MDT0000_UUID lock: ffff885d51e0fbc0/0xc3c20c06c74b6efb lrc: 4/0,0 mode: PR/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 19 type: IBT flags: 0x60200400000020 nid: 10.9.101.46@o2ib4 remote: 0x363c394bde5a12c6 expref: 1369 pid: 88948 timeout: 194034 lvb_type: 0 Dec 12 11:47:00 fir-md1-s1 kernel: LustreError: 97348:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 4 previous similar messages Dec 12 11:47:00 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.101.46@o2ib4 was evicted due to a lock blocking callback time out: rc -5 Dec 12 11:47:00 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Dec 12 11:51:07 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ab6cce31-df0e-ae34-d69d-c23500355ff1 (at 10.9.101.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885beabcec00, cur 1576180267 expire 1576180117 last 1576180040 Dec 12 11:51:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:05:19 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client acf051dc-6a1e-bbe8-7a61-8e031fd79e86 (at 10.8.7.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bf5a01000, cur 1576181119 expire 1576180969 last 1576180892 Dec 12 12:05:19 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:12:48 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c804f06b-97c0-205b-aa77-e2392ade35bd (at 10.8.7.7@o2ib6) Dec 12 12:12:48 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:24:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 1d444526-0c94-9229-34be-9d214c0c6bbd (at 10.9.101.46@o2ib4) Dec 12 12:24:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:30:30 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7126efc2-9676-1db9-94d0-ae09c1520697 (at 10.9.101.26@o2ib4) Dec 12 12:30:30 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:34:06 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 02eb8135-4034-bcb2-8df8-77d00506e76a (at 10.8.7.15@o2ib6) Dec 12 12:34:06 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:34:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.22.31@o2ib6) Dec 12 12:34:57 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:36:19 fir-md1-s1 kernel: Lustre: 97355:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1576182972/real 1576182972] req@ffff886bd3fe8d80 x1652542946361344/t0(0) o1000->fir-MDT0001-osp-MDT0000@10.0.10.52@o2ib7:24/4 lens 304/4320 e 0 to 1 dl 1576182979 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Dec 12 12:36:19 fir-md1-s1 kernel: Lustre: 97355:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Dec 12 12:36:19 fir-md1-s1 kernel: Lustre: fir-MDT0001-osp-MDT0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Dec 12 12:36:44 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 12:36:44 fir-md1-s1 kernel: LustreError: Skipped 1109 previous similar messages Dec 12 12:38:00 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.110.51@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 12:38:00 fir-md1-s1 kernel: LustreError: Skipped 120 previous similar messages Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Skipped 3 previous similar messages Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (105): c: 4, oc: 0, rc: 8 Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Skipped 3 previous similar messages Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 38668:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 101736:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 12:38:05 fir-md1-s1 kernel: LNetError: 101736:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages Dec 12 12:38:49 fir-md1-s1 kernel: LNetError: 101736:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 12:38:49 fir-md1-s1 kernel: LNetError: 101736:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 5 previous similar messages Dec 12 12:39:10 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Dec 12 12:39:10 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 2 previous similar messages Dec 12 12:39:26 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Dec 12 12:39:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.103.28@o2ib4) Dec 12 12:39:37 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:39:43 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b6936c9e-bc4f-ad29-5bfd-ac26e88c91e0 (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bd0acd000, cur 1576183183 expire 1576183033 last 1576182956 Dec 12 12:39:43 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:39:46 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Dec 12 12:39:46 fir-md1-s1 kernel: LNet: 38662:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Skipped 1 previous similar message Dec 12 12:40:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.27.18@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 12:40:31 fir-md1-s1 kernel: LustreError: Skipped 182 previous similar messages Dec 12 12:41:25 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 12:41:25 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (20): c: 0, oc: 0, rc: 8 Dec 12 12:41:25 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 Dec 12 12:41:25 fir-md1-s1 kernel: LNetError: 38662:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 4 previous similar messages Dec 12 12:43:03 fir-md1-s1 kernel: LNet: Service thread pid 97355 was inactive for 411.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 12:43:03 fir-md1-s1 kernel: Pid: 97355, comm: mdt00_051 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:43:03 fir-md1-s1 kernel: Call Trace: Dec 12 12:43:03 fir-md1-s1 kernel: [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] osp_remote_sync+0xd3/0x200 [osp] Dec 12 12:43:03 fir-md1-s1 kernel: [] osp_attr_get+0x463/0x730 [osp] Dec 12 12:43:03 fir-md1-s1 kernel: [] osp_object_init+0x16d/0x2d0 [osp] Dec 12 12:43:03 fir-md1-s1 kernel: [] lu_object_start.isra.35+0x8b/0x120 [obdclass] Dec 12 12:43:03 fir-md1-s1 kernel: [] lu_object_find_at+0x1e1/0xa60 [obdclass] Dec 12 12:43:03 fir-md1-s1 kernel: [] lu_object_find_slice+0x1f/0x90 [obdclass] Dec 12 12:43:03 fir-md1-s1 kernel: [] mdd_object_find+0x10/0x70 [mdd] Dec 12 12:43:03 fir-md1-s1 kernel: [] obf_lookup+0x2c9/0x350 [mdd] Dec 12 12:43:03 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0xf7c/0x1c30 [mdt] Dec 12 12:43:03 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 12:43:03 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:43:03 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:43:03 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:43:04 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:43:04 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:43:04 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:43:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183384.97355 Dec 12 12:43:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Dec 12 12:43:05 fir-md1-s1 kernel: LNetError: 38662:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (15): c: 0, oc: 0, rc: 8 Dec 12 12:45:31 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Dec 12 12:45:31 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 12:45:32 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.27.18@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Dec 12 12:45:32 fir-md1-s1 kernel: LustreError: Skipped 550 previous similar messages Dec 12 12:45:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID Dec 12 12:45:32 fir-md1-s1 kernel: Lustre: Skipped 1222 previous similar messages Dec 12 12:46:07 fir-md1-s1 kernel: Lustre: 39399:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8852966f9200 x1652162130109376/t0(0) o101->ae1d0080-04fa-5436-e145-ffdf0db9990d@10.0.10.3@o2ib7:272/0 lens 600/3264 e 14 to 0 dl 1576183572 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 12:46:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client ae1d0080-04fa-5436-e145-ffdf0db9990d (at 10.0.10.3@o2ib7) reconnecting Dec 12 12:46:13 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 12:47:16 fir-md1-s1 kernel: LNet: Service thread pid 97355 completed after 663.77s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 12:48:55 fir-md1-s1 kernel: LNet: Service thread pid 39417 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 12:48:55 fir-md1-s1 kernel: Pid: 39417, comm: mdt03_042 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:48:55 fir-md1-s1 kernel: Call Trace: Dec 12 12:48:55 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 12:48:55 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 12:48:55 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 12:48:55 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 12:48:55 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 12:48:55 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 12:48:55 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:48:55 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:48:55 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:48:55 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:48:55 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:48:55 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183735.39417 Dec 12 12:48:56 fir-md1-s1 kernel: LNet: Service thread pid 39358 was inactive for 200.58s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 12:48:56 fir-md1-s1 kernel: Pid: 39358, comm: mdt03_025 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:48:56 fir-md1-s1 kernel: Call Trace: Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:48:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:48:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:48:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183736.39358 Dec 12 12:48:56 fir-md1-s1 kernel: Pid: 39275, comm: mdt02_020 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:48:56 fir-md1-s1 kernel: Call Trace: Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:48:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:48:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:48:56 fir-md1-s1 kernel: Pid: 39411, comm: mdt02_042 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:48:56 fir-md1-s1 kernel: Call Trace: Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:48:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:48:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:48:56 fir-md1-s1 kernel: Pid: 39244, comm: mdt03_008 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 12:48:56 fir-md1-s1 kernel: Call Trace: Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 12:48:56 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 12:48:56 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 12:48:56 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 12:48:56 fir-md1-s1 kernel: LNet: Service thread pid 97455 was inactive for 200.76s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 12:48:56 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 12:48:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183737.97383 Dec 12 12:48:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183738.38898 Dec 12 12:49:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183740.39426 Dec 12 12:49:11 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183751.39254 Dec 12 12:49:13 fir-md1-s1 kernel: LNet: Service thread pid 38895 was inactive for 200.12s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 12:49:13 fir-md1-s1 kernel: LNet: Skipped 35 previous similar messages Dec 12 12:49:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183753.38895 Dec 12 12:50:12 fir-md1-s1 kernel: LNet: Service thread pid 39417 completed after 277.29s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 12:50:12 fir-md1-s1 kernel: LNet: Skipped 39 previous similar messages Dec 12 12:50:12 fir-md1-s1 kernel: LNet: Service thread pid 39232 was inactive for 200.61s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 12:50:12 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183812.39232 Dec 12 12:50:32 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183832.97378 Dec 12 12:50:41 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576183841.39238 Dec 12 12:51:52 fir-md1-s1 kernel: LNet: Service thread pid 38894 completed after 299.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 12:51:52 fir-md1-s1 kernel: LNet: Skipped 5 previous similar messages Dec 12 12:52:35 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Thu Dec 12 12:52:35 2019 Dec 12 13:12:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f97f048d-b027-4 (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff886a69bac000, cur 1576185157 expire 1576185007 last 1576184930 Dec 12 13:41:03 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 88e30dc0-2493-6815-27c6-7300a4eebf30 (at 10.8.28.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bd0ac2800, cur 1576186863 expire 1576186713 last 1576186636 Dec 12 13:41:03 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 13:45:21 fir-md1-s1 kernel: LNet: Service thread pid 97407 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 13:45:21 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 13:45:21 fir-md1-s1 kernel: Pid: 97407, comm: mdt02_055 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 13:45:21 fir-md1-s1 kernel: Call Trace: Dec 12 13:45:21 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 13:45:21 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 13:45:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 13:45:22 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 13:45:22 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 13:45:22 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 13:45:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 13:45:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 13:45:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 13:45:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 13:45:22 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 13:45:22 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187122.97407 Dec 12 13:45:39 fir-md1-s1 kernel: LNet: Service thread pid 39408 was inactive for 200.43s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 13:45:39 fir-md1-s1 kernel: Pid: 39408, comm: mdt02_041 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 13:45:39 fir-md1-s1 kernel: Call Trace: Dec 12 13:45:39 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 13:45:39 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 13:45:39 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 13:45:39 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187139.39408 Dec 12 13:45:39 fir-md1-s1 kernel: Pid: 39363, comm: mdt02_030 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 13:45:39 fir-md1-s1 kernel: Call Trace: Dec 12 13:45:39 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 13:45:39 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 13:45:39 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 13:45:39 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 13:45:39 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 13:46:09 fir-md1-s1 kernel: LNet: Service thread pid 39360 was inactive for 200.45s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 13:46:09 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 13:46:09 fir-md1-s1 kernel: Pid: 39360, comm: mdt02_029 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 13:46:09 fir-md1-s1 kernel: Call Trace: Dec 12 13:46:09 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 13:46:09 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 13:46:09 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 13:46:09 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 13:46:09 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 13:46:09 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 13:46:09 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 13:46:09 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 13:46:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 13:46:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 13:46:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 13:46:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187169.39360 Dec 12 13:46:58 fir-md1-s1 kernel: LNet: Service thread pid 39408 completed after 280.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 13:46:58 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 13:47:01 fir-md1-s1 kernel: LNet: Service thread pid 38895 was inactive for 200.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 13:47:01 fir-md1-s1 kernel: Pid: 38895, comm: mdt02_002 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 13:47:01 fir-md1-s1 kernel: Call Trace: Dec 12 13:47:01 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 13:47:01 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 13:47:01 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 13:47:01 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 13:47:01 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 13:47:01 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 13:47:01 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 13:47:01 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 13:47:01 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 13:47:01 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 13:47:01 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 13:47:01 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187221.38895 Dec 12 13:47:35 fir-md1-s1 kernel: LNet: Service thread pid 39330 was inactive for 200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 13:47:35 fir-md1-s1 kernel: LNet: Skipped 5 previous similar messages Dec 12 13:47:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187255.39330 Dec 12 13:48:38 fir-md1-s1 kernel: LNet: Service thread pid 38895 completed after 297.88s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 13:48:38 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 13:48:39 fir-md1-s1 kernel: LNet: Service thread pid 39329 was inactive for 200.11s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 13:48:39 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576187319.39329 Dec 12 13:50:18 fir-md1-s1 kernel: LNet: Service thread pid 39329 completed after 300.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 13:50:18 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 14:03:00 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.108.12@o2ib4) Dec 12 14:03:00 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Dec 12 14:07:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 4359a6d6-39f4-3744-7f0f-dc517a2bb4c6 (at 10.8.28.3@o2ib6) Dec 12 14:07:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 15:09:47 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 8b08dc27-d2aa-93f7-25fb-507b587de732 (at 10.9.101.71@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887d4c8b2400, cur 1576192187 expire 1576192037 last 1576191960 Dec 12 15:09:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 15:29:11 fir-md1-s1 kernel: LustreError: 39232:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193051, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8869bcfeb180/0xc3c20c06d2f5b395 lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39232 timeout: 0 lvb_type: 0 Dec 12 15:29:11 fir-md1-s1 kernel: LustreError: 39232:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 15 previous similar messages Dec 12 15:29:25 fir-md1-s1 kernel: LustreError: 39336:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193065, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8875a4fb6300/0xc3c20c06d2f8d5af lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39336 timeout: 0 lvb_type: 0 Dec 12 15:29:43 fir-md1-s1 kernel: LustreError: 39374:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193083, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff887bbe51e0c0/0xc3c20c06d2fcef9a lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39374 timeout: 0 lvb_type: 0 Dec 12 15:29:43 fir-md1-s1 kernel: LustreError: 39374:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 5 previous similar messages Dec 12 15:29:54 fir-md1-s1 kernel: LustreError: 39252:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193094, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff888bf5e6ad00/0xc3c20c06d2ff1fee lrc: 3/1,0 mode: --/PR res: [0x2000013a6:0x62d4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39252 timeout: 0 lvb_type: 0 Dec 12 15:30:17 fir-md1-s1 kernel: LNet: Service thread pid 97355 was inactive for 365.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 15:30:17 fir-md1-s1 kernel: Pid: 97355, comm: mdt00_051 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 15:30:17 fir-md1-s1 kernel: Call Trace: Dec 12 15:30:17 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 15:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 15:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 15:30:17 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193417.97355 Dec 12 15:30:17 fir-md1-s1 kernel: Pid: 39428, comm: mdt00_042 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 15:30:17 fir-md1-s1 kernel: Call Trace: Dec 12 15:30:17 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 15:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 15:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 15:30:17 fir-md1-s1 kernel: Pid: 39432, comm: mdt00_044 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 15:30:17 fir-md1-s1 kernel: Call Trace: Dec 12 15:30:17 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 15:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 15:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 15:30:17 fir-md1-s1 kernel: Pid: 97382, comm: mdt01_060 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 15:30:17 fir-md1-s1 kernel: Call Trace: Dec 12 15:30:17 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 15:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 15:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 15:30:17 fir-md1-s1 kernel: LNet: Service thread pid 97348 was inactive for 365.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 15:30:17 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 15:30:17 fir-md1-s1 kernel: Pid: 97348, comm: mdt01_050 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 15:30:17 fir-md1-s1 kernel: Call Trace: Dec 12 15:30:17 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 15:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 15:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 15:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 15:30:17 fir-md1-s1 kernel: LNet: Service thread pid 39257 was inactive for 364.96s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:30:17 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 15:30:18 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193418.97362 Dec 12 15:30:19 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193419.39349 Dec 12 15:30:20 fir-md1-s1 kernel: LNet: Service thread pid 97405 was inactive for 366.92s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:30:20 fir-md1-s1 kernel: LNet: Skipped 19 previous similar messages Dec 12 15:30:20 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193420.97405 Dec 12 15:30:22 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193422.39394 Dec 12 15:30:23 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193423.39258 Dec 12 15:30:27 fir-md1-s1 kernel: LNet: Service thread pid 98146 was inactive for 364.72s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:30:27 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 15:30:27 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193427.98146 Dec 12 15:30:29 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193429.39247 Dec 12 15:30:30 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193430.39384 Dec 12 15:30:35 fir-md1-s1 kernel: LNet: Service thread pid 97389 was inactive for 364.98s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:30:35 fir-md1-s1 kernel: LNet: Skipped 10 previous similar messages Dec 12 15:30:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193435.97389 Dec 12 15:30:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193448.39374 Dec 12 15:30:51 fir-md1-s1 kernel: LNet: Service thread pid 39264 completed after 399.18s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 15:30:51 fir-md1-s1 kernel: LNet: Skipped 5 previous similar messages Dec 12 15:30:58 fir-md1-s1 kernel: LNet: Service thread pid 98176 was inactive for 364.86s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:30:58 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 15:30:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193458.98176 Dec 12 15:31:01 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193461.39252 Dec 12 15:31:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193468.39444 Dec 12 15:31:10 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193470.39270 Dec 12 15:31:16 fir-md1-s1 kernel: LustreError: 97386:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193176, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8855e0f30240/0xc3c20c06d311b8d5 lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97386 timeout: 0 lvb_type: 0 Dec 12 15:31:16 fir-md1-s1 kernel: LustreError: 97386:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Dec 12 15:31:49 fir-md1-s1 kernel: LustreError: 39241:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193209, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff88639b4421c0/0xc3c20c06d31919d2 lrc: 3/0,1 mode: --/CW res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x2/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39241 timeout: 0 lvb_type: 0 Dec 12 15:31:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 15:31:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 15:31:51 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 15:31:56 fir-md1-s1 kernel: LNet: Service thread pid 39407 was inactive for 364.71s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 15:31:56 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 15:31:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193516.39407 Dec 12 15:32:31 fir-md1-s1 kernel: LNet: Service thread pid 38890 completed after 499.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 15:32:31 fir-md1-s1 kernel: LNet: Skipped 13 previous similar messages Dec 12 15:32:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193566.38891 Dec 12 15:32:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193568.39233 Dec 12 15:32:49 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193569.39214 Dec 12 15:32:51 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193571.39266 Dec 12 15:32:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576193572.97386 Dec 12 15:34:06 fir-md1-s1 kernel: Lustre: 39436:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff885bc250cc80 x1649315785578896/t0(0) o101->b4206b2f-67a2-cb01-c899-d99205e22b23@10.9.108.61@o2ib4:536/0 lens 1832/3288 e 13 to 0 dl 1576193651 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 15:34:12 fir-md1-s1 kernel: LustreError: 38892:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193351, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8869cecc6c00/0xc3c20c06d338026b lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 72 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 38892 timeout: 0 lvb_type: 0 Dec 12 15:34:12 fir-md1-s1 kernel: LustreError: 97387:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193351, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852f1bac380/0xc3c20c06d338023a lrc: 3/0,1 mode: --/CW res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x2/0x0 rrc: 70 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 97387 timeout: 0 lvb_type: 0 Dec 12 15:34:12 fir-md1-s1 kernel: LustreError: 97387:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Dec 12 15:34:12 fir-md1-s1 kernel: LNet: Service thread pid 98176 completed after 557.81s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 15:34:12 fir-md1-s1 kernel: Lustre: 97348:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff886bea4aa400 x1649317747289552/t592806742019(0) o101->3532db27-3550-1319-6c1b-3d6651c2c9af@10.9.108.62@o2ib4:536/0 lens 1840/904 e 13 to 0 dl 1576193651 ref 1 fl Complete:/0/0 rc 0/0 Dec 12 15:34:12 fir-md1-s1 kernel: LustreError: 38892:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages Dec 12 15:39:12 fir-md1-s1 kernel: LustreError: 39241:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576193652, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff888bac6eb3c0/0xc3c20c06d37b7b70 lrc: 3/0,1 mode: --/CW res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x2/0x0 rrc: 82 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39241 timeout: 0 lvb_type: 0 Dec 12 15:39:12 fir-md1-s1 kernel: LustreError: 39241:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages Dec 12 15:39:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 15:39:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 15:48:05 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.101.71@o2ib4) Dec 12 15:49:12 fir-md1-s1 kernel: LustreError: 39258:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576194252, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff885853baca40/0xc3c20c06d40fb711 lrc: 3/1,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 84 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39258 timeout: 0 lvb_type: 0 Dec 12 15:49:12 fir-md1-s1 kernel: LustreError: 39258:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 35 previous similar messages Dec 12 15:49:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 15:49:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 15:49:12 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 15:59:12 fir-md1-s1 kernel: LustreError: 39341:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576194852, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8852cc2ccec0/0xc3c20c06d467104b lrc: 3/1,0 mode: --/PR res: [0x20003957b:0x1410:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39341 timeout: 0 lvb_type: 0 Dec 12 15:59:12 fir-md1-s1 kernel: LustreError: 39341:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 38 previous similar messages Dec 12 16:00:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 16:00:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 16:05:47 fir-md1-s1 kernel: Lustre: 39215:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff887405e96300 x1649340861684560/t0(0) o101->d5336f36-1352-ddc7-e966-e696298bb1ae@10.9.106.53@o2ib4:172/0 lens 376/1600 e 5 to 0 dl 1576195552 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:05:47 fir-md1-s1 kernel: Lustre: 39215:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages Dec 12 16:05:48 fir-md1-s1 kernel: Lustre: 38896:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff888bd0e69f80 x1649046576943376/t0(0) o101->f9f503f0-6ff6-698f-9a8d-14bd128a6d42@10.9.101.27@o2ib4:173/0 lens 1792/3288 e 5 to 0 dl 1576195553 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:05:48 fir-md1-s1 kernel: Lustre: 38896:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages Dec 12 16:05:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 78c1e3f1-dd0b-4 (at 10.8.18.18@o2ib6) reconnecting Dec 12 16:05:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to ee8a8d10-65c2-ae96-bc67-9f6bae32e110 (at 10.8.18.18@o2ib6) Dec 12 16:05:55 fir-md1-s1 kernel: LNet: Service thread pid 39389 was inactive for 601.16s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:05:55 fir-md1-s1 kernel: Pid: 39389, comm: mdt03_034 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:05:55 fir-md1-s1 kernel: Call Trace: Dec 12 16:05:55 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:05:55 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:05:55 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:05:55 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195555.39389 Dec 12 16:05:55 fir-md1-s1 kernel: Pid: 98146, comm: mdt00_068 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:05:55 fir-md1-s1 kernel: Call Trace: Dec 12 16:05:55 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 16:05:55 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:05:55 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:05:55 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:05:55 fir-md1-s1 kernel: Pid: 39431, comm: mdt00_043 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:05:55 fir-md1-s1 kernel: Call Trace: Dec 12 16:05:55 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:05:55 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:05:55 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:05:55 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:05:56 fir-md1-s1 kernel: Lustre: 39245:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8852857e0900 x1649314302985776/t0(0) o101->75af6c9a-e740-8c0d-465f-820e82ef6338@10.9.108.60@o2ib4:181/0 lens 1784/3288 e 4 to 0 dl 1576195561 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:05:56 fir-md1-s1 kernel: Lustre: 39245:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 26 previous similar messages Dec 12 16:05:58 fir-md1-s1 kernel: Lustre: 106785:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8873fca03a80 x1649309816130960/t0(0) o101->1431f338-e19b-6337-4b33-ec6ebaff454a@10.8.18.22@o2ib6:183/0 lens 1840/3288 e 5 to 0 dl 1576195563 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:05:58 fir-md1-s1 kernel: Lustre: 106785:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Dec 12 16:06:03 fir-md1-s1 kernel: LNet: Service thread pid 39347 was inactive for 601.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:06:03 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 16:06:03 fir-md1-s1 kernel: Pid: 39347, comm: mdt00_023 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:06:03 fir-md1-s1 kernel: Call Trace: Dec 12 16:06:03 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:06:03 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:06:03 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:06:03 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:06:03 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:06:03 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:06:03 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:06:03 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:06:03 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:06:03 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:06:03 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:06:03 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195563.39347 Dec 12 16:06:04 fir-md1-s1 kernel: Lustre: 39245:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8852cc790d80 x1652733017378752/t0(0) o101->851b742b-36ee-4@10.9.107.13@o2ib4:189/0 lens 1896/3288 e 4 to 0 dl 1576195569 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:06:04 fir-md1-s1 kernel: Lustre: 39245:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages Dec 12 16:06:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1431f338-e19b-6337-4b33-ec6ebaff454a (at 10.8.18.22@o2ib6) reconnecting Dec 12 16:06:05 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Dec 12 16:06:09 fir-md1-s1 kernel: LNet: Service thread pid 97354 was inactive for 600.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:06:09 fir-md1-s1 kernel: Pid: 97354, comm: mdt00_050 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:06:09 fir-md1-s1 kernel: Call Trace: Dec 12 16:06:09 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:06:09 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:06:09 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:06:09 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:06:09 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:06:09 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:06:09 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:06:09 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:06:09 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:06:09 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:06:09 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:06:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195569.97354 Dec 12 16:06:12 fir-md1-s1 kernel: Lustre: 97350:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88528625c380 x1649559265975568/t0(0) o101->a8d84424-9b8a-5525-fab4-b5243bf0dc64@10.9.104.22@o2ib4:197/0 lens 1904/3288 e 4 to 0 dl 1576195577 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:06:12 fir-md1-s1 kernel: Lustre: 97350:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Dec 12 16:06:13 fir-md1-s1 kernel: LNet: Service thread pid 39436 was inactive for 601.61s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:06:13 fir-md1-s1 kernel: LNet: Skipped 9 previous similar messages Dec 12 16:06:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195573.39436 Dec 12 16:06:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 5860417b-a563-2455-9c94-86226f905ab9 (at 10.8.27.9@o2ib6) Dec 12 16:06:15 fir-md1-s1 kernel: Lustre: Skipped 19 previous similar messages Dec 12 16:06:15 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195575.39351 Dec 12 16:06:19 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195579.97380 Dec 12 16:06:27 fir-md1-s1 kernel: LNet: Service thread pid 39417 was inactive for 600.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:06:27 fir-md1-s1 kernel: LNet: Skipped 5 previous similar messages Dec 12 16:06:27 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195587.39417 Dec 12 16:06:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d2bd0014-3bea-4 (at 10.9.114.7@o2ib4) reconnecting Dec 12 16:06:28 fir-md1-s1 kernel: Lustre: Skipped 14 previous similar messages Dec 12 16:06:41 fir-md1-s1 kernel: Lustre: 38892:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff886978788000 x1652736187566016/t0(0) o101->d9364eb2-511c-4@10.8.27.10@o2ib6:225/0 lens 600/3264 e 2 to 0 dl 1576195605 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:06:41 fir-md1-s1 kernel: Lustre: 38892:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Dec 12 16:07:27 fir-md1-s1 kernel: Lustre: 39339:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8852bf750900 x1649327288667664/t0(0) o101->a7c6c322-7850-feae-097c-a35b332d6e36@10.9.108.67@o2ib4:272/0 lens 376/1600 e 1 to 0 dl 1576195652 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:07:32 fir-md1-s1 kernel: LNet: Service thread pid 98146 completed after 698.30s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:07:32 fir-md1-s1 kernel: LNet: Skipped 36 previous similar messages Dec 12 16:07:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 16:07:32 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 16:07:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 16:07:32 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Dec 12 16:07:33 fir-md1-s1 kernel: LNet: Service thread pid 97405 was inactive for 601.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:07:33 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195653.97405 Dec 12 16:07:43 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195663.98144 Dec 12 16:07:45 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195665.97386 Dec 12 16:07:47 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195667.39442 Dec 12 16:08:48 fir-md1-s1 kernel: Lustre: 107131:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff887879306c00 x1649484815686192/t0(0) o101->5627d86f-0964-ad4d-2769-f014ccc68300@10.8.17.16@o2ib6:353/0 lens 600/3264 e 2 to 0 dl 1576195733 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:08:48 fir-md1-s1 kernel: Lustre: 107131:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 18 previous similar messages Dec 12 16:08:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 5627d86f-0964-ad4d-2769-f014ccc68300 (at 10.8.17.16@o2ib6) reconnecting Dec 12 16:08:54 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Dec 12 16:08:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 5627d86f-0964-ad4d-2769-f014ccc68300 (at 10.8.17.16@o2ib6) Dec 12 16:08:54 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Dec 12 16:08:57 fir-md1-s1 kernel: LNet: Service thread pid 39261 was inactive for 600.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:08:57 fir-md1-s1 kernel: LNet: Skipped 12 previous similar messages Dec 12 16:08:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195737.39261 Dec 12 16:09:05 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195745.39380 Dec 12 16:09:12 fir-md1-s1 kernel: LNet: Service thread pid 98142 completed after 780.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:09:12 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 16:09:15 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195755.39263 Dec 12 16:09:24 fir-md1-s1 kernel: LustreError: 107018:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576195464, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff88643a14f740/0xc3c20c06d4cc5c26 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 14 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 107018 timeout: 0 lvb_type: 0 Dec 12 16:09:24 fir-md1-s1 kernel: LustreError: 107018:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 51 previous similar messages Dec 12 16:09:32 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195772.39402 Dec 12 16:09:36 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195776.106852 Dec 12 16:09:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195792.39333 Dec 12 16:10:07 fir-md1-s1 kernel: LNet: Service thread pid 106854 was inactive for 801.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:10:07 fir-md1-s1 kernel: LNet: Skipped 9 previous similar messages Dec 12 16:10:07 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195807.106854 Dec 12 16:10:52 fir-md1-s1 kernel: LNet: Service thread pid 39402 completed after 879.96s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:10:52 fir-md1-s1 kernel: LNet: Skipped 15 previous similar messages Dec 12 16:10:54 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195854.39425 Dec 12 16:11:00 fir-md1-s1 kernel: LNet: Service thread pid 39253 was inactive for 801.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:11:00 fir-md1-s1 kernel: Pid: 39253, comm: mdt01_019 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:11:00 fir-md1-s1 kernel: Call Trace: Dec 12 16:11:00 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:11:00 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:11:00 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:11:00 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:11:00 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:11:00 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:11:00 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:11:00 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:11:00 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:11:00 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:11:00 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:11:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195860.39253 Dec 12 16:11:02 fir-md1-s1 kernel: Pid: 97344, comm: mdt01_046 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:11:02 fir-md1-s1 kernel: Call Trace: Dec 12 16:11:02 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:11:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:11:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:11:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195862.97344 Dec 12 16:11:02 fir-md1-s1 kernel: Pid: 98236, comm: mdt01_065 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:11:02 fir-md1-s1 kernel: Call Trace: Dec 12 16:11:02 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:11:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:11:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:11:02 fir-md1-s1 kernel: Pid: 38891, comm: mdt01_001 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:11:02 fir-md1-s1 kernel: Call Trace: Dec 12 16:11:02 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:11:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:11:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:11:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:11:04 fir-md1-s1 kernel: LNet: Service thread pid 39265 was inactive for 801.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:11:04 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Dec 12 16:11:04 fir-md1-s1 kernel: Pid: 39265, comm: mdt02_016 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:11:04 fir-md1-s1 kernel: Call Trace: Dec 12 16:11:04 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:11:04 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:11:04 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:11:04 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:11:04 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:11:04 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:11:04 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:11:04 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:11:04 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:11:04 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:11:04 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:11:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195864.39265 Dec 12 16:11:06 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195866.97376 Dec 12 16:11:23 fir-md1-s1 kernel: Lustre: 38887:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-31), not sending early reply req@ffff8852d22f6c00 x1649494866337200/t0(0) o101->a39c942a-14d0-8a42-662a-6515c9201963@10.9.102.5@o2ib4:508/0 lens 576/3264 e 0 to 0 dl 1576195888 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:11:23 fir-md1-s1 kernel: Lustre: 38887:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages Dec 12 16:11:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 97102c2b-e0e2-553a-c933-88dc912145da (at 10.9.115.11@o2ib4) reconnecting Dec 12 16:11:29 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Dec 12 16:11:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 97102c2b-e0e2-553a-c933-88dc912145da (at 10.9.115.11@o2ib4) Dec 12 16:11:29 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 12 16:12:18 fir-md1-s1 kernel: LNet: Service thread pid 106898 was inactive for 801.94s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:12:18 fir-md1-s1 kernel: LNet: Skipped 11 previous similar messages Dec 12 16:12:18 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195938.106898 Dec 12 16:12:28 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195948.106917 Dec 12 16:12:32 fir-md1-s1 kernel: LNet: Service thread pid 97355 completed after 900.06s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:12:32 fir-md1-s1 kernel: LNet: Skipped 7 previous similar messages Dec 12 16:12:40 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195960.39438 Dec 12 16:12:44 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195964.98340 Dec 12 16:12:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195972.97377 Dec 12 16:12:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195976.39246 Dec 12 16:13:03 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195983.39360 Dec 12 16:13:05 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576195985.39346 Dec 12 16:13:42 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196022.39241 Dec 12 16:13:44 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196024.39257 Dec 12 16:13:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client af8d5000-0c68-4 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9684400, cur 1576196026 expire 1576195876 last 1576195799 Dec 12 16:13:46 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 16:13:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196026.97383 Dec 12 16:13:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196032.39232 Dec 12 16:13:54 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196034.39407 Dec 12 16:13:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196038.106857 Dec 12 16:14:12 fir-md1-s1 kernel: LNet: Service thread pid 39426 completed after 998.83s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:14:12 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Dec 12 16:14:18 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196058.39324 Dec 12 16:15:16 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196116.39404 Dec 12 16:15:52 fir-md1-s1 kernel: LNet: Service thread pid 39265 completed after 1089.26s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:15:52 fir-md1-s1 kernel: LNet: Skipped 7 previous similar messages Dec 12 16:15:59 fir-md1-s1 kernel: Lustre: 107132:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-398), not sending early reply req@ffff886928d1ba80 x1649340861729072/t0(0) o101->d5336f36-1352-ddc7-e966-e696298bb1ae@10.9.106.53@o2ib4:29/0 lens 1784/3288 e 0 to 0 dl 1576196164 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:15:59 fir-md1-s1 kernel: Lustre: 107132:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 45 previous similar messages Dec 12 16:16:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 469c9fed-7a4e-a33d-2f08-51ca338b69fb (at 10.9.108.68@o2ib4) reconnecting Dec 12 16:16:31 fir-md1-s1 kernel: Lustre: Skipped 35 previous similar messages Dec 12 16:16:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 469c9fed-7a4e-a33d-2f08-51ca338b69fb (at 10.9.108.68@o2ib4) Dec 12 16:16:31 fir-md1-s1 kernel: Lustre: Skipped 36 previous similar messages Dec 12 16:16:42 fir-md1-s1 kernel: LNet: Service thread pid 106909 was inactive for 960.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:16:42 fir-md1-s1 kernel: Pid: 106909, comm: mdt02_068 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:16:42 fir-md1-s1 kernel: Call Trace: Dec 12 16:16:42 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:16:42 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:16:42 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:16:42 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:16:42 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:16:42 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:16:42 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:16:42 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:16:42 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:16:42 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196202.106909 Dec 12 16:16:46 fir-md1-s1 kernel: Pid: 38898, comm: mdt03_002 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:16:46 fir-md1-s1 kernel: Call Trace: Dec 12 16:16:46 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:16:46 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:16:46 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:16:46 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:16:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:16:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:16:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:16:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:16:46 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:16:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196206.38898 Dec 12 16:17:32 fir-md1-s1 kernel: LNet: Service thread pid 39325 completed after 1187.57s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:17:32 fir-md1-s1 kernel: LNet: Skipped 14 previous similar messages Dec 12 16:17:43 fir-md1-s1 kernel: LNet: Service thread pid 39414 was inactive for 1011.58s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:17:43 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 16:17:43 fir-md1-s1 kernel: Pid: 39414, comm: mdt00_040 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:17:43 fir-md1-s1 kernel: Call Trace: Dec 12 16:17:43 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:17:43 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:17:43 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:17:43 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:17:43 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196263.39414 Dec 12 16:17:43 fir-md1-s1 kernel: Pid: 88947, comm: mdt01_042 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:17:43 fir-md1-s1 kernel: Call Trace: Dec 12 16:17:43 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:17:43 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:17:44 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:17:44 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:17:44 fir-md1-s1 kernel: Pid: 106931, comm: mdt01_093 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:17:44 fir-md1-s1 kernel: Call Trace: Dec 12 16:17:44 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:17:44 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:17:44 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:17:44 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:17:44 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:17:44 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:17:44 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:17:44 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:17:44 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:17:44 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:17:44 fir-md1-s1 kernel: LNet: Service thread pid 98332 was inactive for 1012.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 16:17:44 fir-md1-s1 kernel: LNet: Skipped 42 previous similar messages Dec 12 16:17:47 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196267.39385 Dec 12 16:17:51 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196271.39273 Dec 12 16:17:53 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196273.39348 Dec 12 16:17:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196276.97459 Dec 12 16:18:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196280.39255 Dec 12 16:18:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196284.39369 Dec 12 16:18:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196288.98176 Dec 12 16:18:24 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196304.39416 Dec 12 16:18:28 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196308.38941 Dec 12 16:18:41 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196321.97359 Dec 12 16:18:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196337.39236 Dec 12 16:19:05 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196345.106855 Dec 12 16:19:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196349.39349 Dec 12 16:19:12 fir-md1-s1 kernel: LNet: Service thread pid 39336 completed after 1178.37s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:19:12 fir-md1-s1 kernel: LNet: Skipped 10 previous similar messages Dec 12 16:19:24 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196364.97401 Dec 12 16:19:26 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196366.39227 Dec 12 16:19:30 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196370.39247 Dec 12 16:19:32 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196372.97408 Dec 12 16:19:34 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196374.39421 Dec 12 16:20:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196402.107023 Dec 12 16:20:15 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196415.97434 Dec 12 16:20:20 fir-md1-s1 kernel: LustreError: 39340:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576196120, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff888b8d7eaf40/0xc3c20c06d5315c19 lrc: 3/1,0 mode: --/PR res: [0x2000016f2:0x7:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39340 timeout: 0 lvb_type: 0 Dec 12 16:20:20 fir-md1-s1 kernel: LustreError: 39340:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 32 previous similar messages Dec 12 16:20:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196448.39411 Dec 12 16:20:52 fir-md1-s1 kernel: LNet: Service thread pid 39377 completed after 1100.06s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:20:52 fir-md1-s1 kernel: LNet: Skipped 37 previous similar messages Dec 12 16:21:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196464.39403 Dec 12 16:22:05 fir-md1-s1 kernel: LNet: Service thread pid 97345 was inactive for 1063.42s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:22:05 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 16:22:05 fir-md1-s1 kernel: Pid: 97345, comm: mdt01_047 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:22:05 fir-md1-s1 kernel: Call Trace: Dec 12 16:22:05 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:22:05 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:22:05 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:22:05 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:22:05 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:22:05 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:22:05 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:22:05 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:22:05 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:22:06 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:22:06 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:22:06 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196526.97345 Dec 12 16:22:06 fir-md1-s1 kernel: Pid: 39323, comm: mdt00_016 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:22:06 fir-md1-s1 kernel: Call Trace: Dec 12 16:22:06 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:22:06 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 16:22:06 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 16:22:06 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:22:06 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:22:06 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:22:06 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:22:06 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:22:06 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:22:06 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:22:06 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:22:32 fir-md1-s1 kernel: Lustre: 39411:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (636:481s); client may timeout. req@ffff887775c93180 x1652555110615360/t0(0) o101->cfe93466-ba97-4@10.9.0.62@o2ib4:691/0 lens 584/536 e 0 to 0 dl 1576196071 ref 1 fl Complete:/0/0 rc 0/0 Dec 12 16:22:32 fir-md1-s1 kernel: Lustre: 39411:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 3 previous similar messages Dec 12 16:24:57 fir-md1-s1 kernel: Lustre: 106831:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-845), not sending early reply req@ffff8864fd63f980 x1649313082320112/t0(0) o101->7aa12007-79f9-a9cc-9090-a11975521a91@10.9.108.63@o2ib4:567/0 lens 1832/3288 e 0 to 0 dl 1576196702 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:24:57 fir-md1-s1 kernel: Lustre: 106831:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 84 previous similar messages Dec 12 16:25:47 fir-md1-s1 kernel: LNet: Service thread pid 39192 was inactive for 1204.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 16:25:47 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Dec 12 16:25:47 fir-md1-s1 kernel: Pid: 39192, comm: mdt03_003 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:25:47 fir-md1-s1 kernel: Call Trace: Dec 12 16:25:47 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:25:47 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:25:47 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:25:47 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 16:25:47 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 16:25:47 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 16:25:47 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 16:25:47 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 16:25:47 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 16:25:47 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:25:47 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:25:47 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:25:47 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:25:47 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:25:47 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196747.39192 Dec 12 16:26:03 fir-md1-s1 kernel: Pid: 97382, comm: mdt01_060 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:26:03 fir-md1-s1 kernel: Call Trace: Dec 12 16:26:03 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:26:03 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:26:03 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:26:03 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196763.97382 Dec 12 16:26:03 fir-md1-s1 kernel: Pid: 39221, comm: mdt01_007 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:26:03 fir-md1-s1 kernel: Call Trace: Dec 12 16:26:03 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:26:03 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:26:03 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:26:03 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:26:03 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:26:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 55c89a19-c2de-4 (at 10.8.0.82@o2ib6) reconnecting Dec 12 16:26:51 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Dec 12 16:26:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 55c89a19-c2de-4 (at 10.8.0.82@o2ib6) Dec 12 16:26:51 fir-md1-s1 kernel: Lustre: Skipped 43 previous similar messages Dec 12 16:27:33 fir-md1-s1 kernel: Pid: 106834, comm: mdt01_078 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:27:33 fir-md1-s1 kernel: Call Trace: Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:27:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:27:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:27:33 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576196853.106834 Dec 12 16:27:33 fir-md1-s1 kernel: Pid: 39384, comm: mdt01_032 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:27:33 fir-md1-s1 kernel: Call Trace: Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:27:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:27:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:27:33 fir-md1-s1 kernel: Pid: 39334, comm: mdt00_020 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:27:33 fir-md1-s1 kernel: Call Trace: Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:27:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:27:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:27:33 fir-md1-s1 kernel: Pid: 97442, comm: mdt02_059 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 16:27:33 fir-md1-s1 kernel: Call Trace: Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_getattr+0x2b5/0x480 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 16:27:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 16:27:34 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 16:27:34 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 16:27:34 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 16:27:34 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 16:30:52 fir-md1-s1 kernel: LustreError: 97347:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576196752, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8869c46f8900/0xc3c20c06d597b463 lrc: 3/1,0 mode: --/PR res: [0x200000406:0x1b2:0x0].0x0 bits 0x13/0x0 rrc: 44 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 97347 timeout: 0 lvb_type: 0 Dec 12 16:30:52 fir-md1-s1 kernel: LustreError: 97347:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 52 previous similar messages Dec 12 16:30:52 fir-md1-s1 kernel: LNet: Service thread pid 39428 completed after 1700.14s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 16:30:52 fir-md1-s1 kernel: LNet: Skipped 21 previous similar messages Dec 12 16:37:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) reconnecting Dec 12 16:37:32 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Dec 12 16:37:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 03dd52b8-a4fc-4 (at 10.9.0.61@o2ib4) Dec 12 16:37:32 fir-md1-s1 kernel: Lustre: Skipped 16 previous similar messages Dec 12 16:41:51 fir-md1-s1 kernel: LustreError: 39268:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576197411, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8878de2818c0/0xc3c20c06d6189ee6 lrc: 3/1,0 mode: --/PR res: [0x200039577:0x1b1a:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 39268 timeout: 0 lvb_type: 0 Dec 12 16:41:51 fir-md1-s1 kernel: LustreError: 39268:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 13 previous similar messages Dec 12 16:47:28 fir-md1-s1 kernel: Lustre: 39429:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff888bafbcc800 x1649050425102112/t0(0) o101->4c1f7414-081e-38fa-7245-fdc2400de56e@10.9.101.49@o2ib4:408/0 lens 1792/3288 e 0 to 0 dl 1576198053 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:47:28 fir-md1-s1 kernel: Lustre: 39429:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages Dec 12 16:47:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 4c1f7414-081e-38fa-7245-fdc2400de56e (at 10.9.101.49@o2ib4) reconnecting Dec 12 16:47:34 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 16:47:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.49@o2ib4) Dec 12 16:47:34 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 16:52:19 fir-md1-s1 kernel: LustreError: 39229:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576198038, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff88643a3a6780/0xc3c20c06d676e9be lrc: 3/0,1 mode: --/CW res: [0x2000376b8:0x1706e:0x0].0x0 bits 0x2/0x0 rrc: 28 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39229 timeout: 0 lvb_type: 0 Dec 12 16:52:19 fir-md1-s1 kernel: LustreError: 39229:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 38 previous similar messages Dec 12 16:53:22 fir-md1-s1 kernel: Lustre: 39370:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff8874d5eb5e80 x1650958631802240/t0(0) o101->bdc6a669-f745-2944-1b74-3762ff7d0bf8@10.9.101.36@o2ib4:7/0 lens 584/3264 e 0 to 0 dl 1576198407 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:53:22 fir-md1-s1 kernel: Lustre: 39370:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Dec 12 16:58:22 fir-md1-s1 kernel: Lustre: 39389:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff888b6fa8b600 x1649420627536176/t0(0) o101->970bc850-7648-f96d-fc2b-8b8c64ce0bd4@10.9.101.52@o2ib4:307/0 lens 584/3264 e 0 to 0 dl 1576198707 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 16:58:22 fir-md1-s1 kernel: Lustre: 39389:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Dec 12 16:58:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 970bc850-7648-f96d-fc2b-8b8c64ce0bd4 (at 10.9.101.52@o2ib4) reconnecting Dec 12 16:58:29 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Dec 12 16:58:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.52@o2ib4) Dec 12 16:58:29 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Dec 12 17:07:32 fir-md1-s1 kernel: LustreError: 39357:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576198952, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff888be543d100/0xc3c20c06d6b8f03a lrc: 3/0,1 mode: --/CW res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x2/0x0 rrc: 39 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 39357 timeout: 0 lvb_type: 0 Dec 12 17:07:32 fir-md1-s1 kernel: LustreError: 39357:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 45 previous similar messages Dec 12 17:17:32 fir-md1-s1 kernel: LustreError: 106832:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1576199552, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff886be6e7a1c0/0xc3c20c06d6ee5d4c lrc: 3/0,1 mode: --/CW res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x2/0x0 rrc: 37 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 106832 timeout: 0 lvb_type: 0 Dec 12 17:17:32 fir-md1-s1 kernel: LustreError: 106832:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 10 previous similar messages Dec 12 17:19:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 62f117dd-237d-c074-d679-5244422357ce (at 10.9.103.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887ba9557c00, cur 1576199979 expire 1576199829 last 1576199752 Dec 12 17:19:39 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 17:22:27 fir-md1-s1 kernel: Lustre: 106857:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8865386b2d00 x1649340862596688/t0(0) o101->d5336f36-1352-ddc7-e966-e696298bb1ae@10.9.106.53@o2ib4:242/0 lens 584/3264 e 3 to 0 dl 1576200152 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 17:22:27 fir-md1-s1 kernel: Lustre: 106857:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Dec 12 17:22:32 fir-md1-s1 kernel: LNet: Service thread pid 39349 was inactive for 600.52s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 12 17:22:32 fir-md1-s1 kernel: LNet: Skipped 6 previous similar messages Dec 12 17:22:32 fir-md1-s1 kernel: Pid: 39349, comm: mdt03_022 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 17:22:32 fir-md1-s1 kernel: Call Trace: Dec 12 17:22:32 fir-md1-s1 kernel: [] osp_precreate_reserve+0x2e8/0x800 [osp] Dec 12 17:22:32 fir-md1-s1 kernel: [] osp_declare_create+0x199/0x5b0 [osp] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_sub_declare_create+0xdf/0x210 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_qos_declare_object_on+0xbe/0x3a0 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x10f4/0x1840 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 17:22:32 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 17:22:32 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 17:22:32 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 17:22:32 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 17:22:32 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 17:22:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 17:22:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 17:22:33 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200153.39349 Dec 12 17:22:33 fir-md1-s1 kernel: Pid: 39385, comm: mdt03_033 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 17:22:33 fir-md1-s1 kernel: Call Trace: Dec 12 17:22:33 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 17:22:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 17:22:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 17:22:33 fir-md1-s1 kernel: Pid: 106832, comm: mdt01_076 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 17:22:33 fir-md1-s1 kernel: Call Trace: Dec 12 17:22:33 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client be4565a9-8448-ebff-ec7a-065a9a83593c (at 10.8.18.19@o2ib6) reconnecting Dec 12 17:22:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 17:22:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.18.19@o2ib6) Dec 12 17:22:33 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 17:22:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 17:22:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 17:22:33 fir-md1-s1 kernel: Pid: 39324, comm: mdt03_015 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 17:22:33 fir-md1-s1 kernel: Call Trace: Dec 12 17:22:33 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_prepare_create+0x215/0x2e0 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_striped_create+0x1ee/0x980 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] lod_declare_create+0x204/0x590 [lod] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_declare_create+0x4c/0xcb0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdd_create+0x847/0x14e0 [mdd] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_open+0x224f/0x3240 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_open+0x82/0x3a0 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 17:22:33 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 17:22:33 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 17:22:33 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 17:22:34 fir-md1-s1 kernel: Pid: 39429, comm: mdt03_046 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 Dec 12 17:22:34 fir-md1-s1 kernel: Call Trace: Dec 12 17:22:34 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Dec 12 17:22:34 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Dec 12 17:22:34 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Dec 12 17:22:34 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Dec 12 17:22:34 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Dec 12 17:22:34 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Dec 12 17:22:34 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Dec 12 17:22:34 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Dec 12 17:22:34 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Dec 12 17:22:34 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Dec 12 17:22:35 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Dec 12 17:22:35 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Dec 12 17:22:35 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Dec 12 17:22:35 fir-md1-s1 kernel: [] 0xffffffffffffffff Dec 12 17:22:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200155.39429 Dec 12 17:22:45 fir-md1-s1 kernel: LNet: Service thread pid 39352 was inactive for 600.96s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 17:22:45 fir-md1-s1 kernel: LNet: Skipped 86 previous similar messages Dec 12 17:22:45 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200165.39352 Dec 12 17:22:55 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200175.39438 Dec 12 17:23:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200193.97455 Dec 12 17:23:15 fir-md1-s1 kernel: Lustre: 39428:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-23), not sending early reply req@ffff88527f7c0480 x1649314426856784/t0(0) o101->1f72d546-482b-ba22-9634-964c4dc9701a@10.9.108.56@o2ib4:290/0 lens 1888/3288 e 0 to 0 dl 1576200200 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 17:23:15 fir-md1-s1 kernel: Lustre: 39428:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Dec 12 17:24:12 fir-md1-s1 kernel: LNet: Service thread pid 39349 completed after 700.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 17:24:12 fir-md1-s1 kernel: LNet: Skipped 22 previous similar messages Dec 12 17:24:13 fir-md1-s1 kernel: LNet: Service thread pid 39368 was inactive for 600.78s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 17:24:13 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Dec 12 17:24:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200253.39368 Dec 12 17:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 0c302cf4-1147-d945-dfa2-e9bc796b3175 (at 10.9.101.32@o2ib4) reconnecting Dec 12 17:24:13 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 12 17:24:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.32@o2ib4) Dec 12 17:24:13 fir-md1-s1 kernel: Lustre: Skipped 11 previous similar messages Dec 12 17:24:35 fir-md1-s1 kernel: Lustre: 107149:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-23), not sending early reply req@ffff88637305b600 x1649050425908336/t0(0) o101->4c1f7414-081e-38fa-7245-fdc2400de56e@10.9.101.49@o2ib4:370/0 lens 584/3264 e 0 to 0 dl 1576200280 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 17:24:35 fir-md1-s1 kernel: Lustre: 107149:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Dec 12 17:24:45 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200285.39229 Dec 12 17:25:10 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200310.107141 Dec 12 17:25:29 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200328.39440 Dec 12 17:25:35 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200335.39273 Dec 12 17:25:52 fir-md1-s1 kernel: LNet: Service thread pid 106930 completed after 700.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 17:25:52 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Dec 12 17:25:53 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200353.39244 Dec 12 17:25:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200357.39358 Dec 12 17:25:59 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200359.39406 Dec 12 17:26:01 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200361.39217 Dec 12 17:26:14 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200374.106870 Dec 12 17:26:16 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200376.39247 Dec 12 17:26:24 fir-md1-s1 kernel: LNet: Service thread pid 39437 was inactive for 601.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Dec 12 17:26:24 fir-md1-s1 kernel: LNet: Skipped 28 previous similar messages Dec 12 17:26:24 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200384.39437 Dec 12 17:26:30 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Dec 12 17:26:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.8.23.22@o2ib6 (stopping) Dec 12 17:26:30 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Dec 12 17:26:30 fir-md1-s1 kernel: LustreError: 97383:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffff886be3ac0800 ns: mdt-fir-MDT0000_UUID lock: ffff885dc50b0900/0xc3c20c06d71114b3 lrc: 3/0,0 mode: --/PR res: [0x20003ac50:0x7f36:0x0].0x0 bits 0x13/0x0 rrc: 32 type: IBT flags: 0x50306400000020 nid: 10.9.106.15@o2ib4 remote: 0x50fb205d05874a29 expref: 7 pid: 97383 timeout: 0 lvb_type: 0 Dec 12 17:26:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.106.66@o2ib4 (stopping) Dec 12 17:26:30 fir-md1-s1 kernel: Lustre: Skipped 38 previous similar messages Dec 12 17:26:30 fir-md1-s1 kernel: LustreError: 43237:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.8@o2ib4 arrived at 1576200390 with bad export cookie 14105850204140236714 Dec 12 17:26:30 fir-md1-s1 kernel: LustreError: 43237:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Dec 12 17:26:30 fir-md1-s1 kernel: LustreError: 43237:0:(ldlm_lock.c:2710:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffff887bbdb13cc0/0xc3c20c06d566926d lrc: 3/0,0 mode: PR/PR res: [0x200000406:0xb3:0x0].0x0 bits 0x13/0x0 rrc: 378 type: IBT flags: 0x40200000000000 nid: 10.9.101.8@o2ib4 remote: 0x61759e98ea0e2f24 expref: 2 pid: 97398 timeout: 0 lvb_type: 0 Dec 12 17:26:31 fir-md1-s1 kernel: LustreError: 39405:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffff887ba9550c00 ns: mdt-fir-MDT0000_UUID lock: ffff88688aa10fc0/0xc3c20c06d71700f0 lrc: 3/0,0 mode: --/PR res: [0x200039577:0x11b6:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x50306400000000 nid: 10.8.27.8@o2ib6 remote: 0x335e15af6b6351f4 expref: 16 pid: 39405 timeout: 0 lvb_type: 0 Dec 12 17:26:31 fir-md1-s1 kernel: LustreError: 39405:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 9 previous similar messages Dec 12 17:26:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.108.30@o2ib4 (stopping) Dec 12 17:26:31 fir-md1-s1 kernel: Lustre: Skipped 72 previous similar messages Dec 12 17:26:31 fir-md1-s1 kernel: LustreError: 45692:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.112@o2ib7 arrived at 1576200391 with bad export cookie 14105850204140815068 Dec 12 17:26:31 fir-md1-s1 kernel: LustreError: 45692:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 8 previous similar messages Dec 12 17:26:31 fir-md1-s1 kernel: LustreError: 108863:0:(ldlm_resource.c:1147:ldlm_resource_complain()) mdt-fir-MDT0000_UUID: namespace resource [0x200038534:0x1ace:0x0].0x0 (ffff8856ccabbbc0) refcount nonzero (2) after lock cleanup; forcing cleanup. Dec 12 17:26:33 fir-md1-s1 kernel: LustreError: 43239:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.105@o2ib7 arrived at 1576200393 with bad export cookie 14105850204140656658 Dec 12 17:26:33 fir-md1-s1 kernel: LustreError: 43239:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 22 previous similar messages Dec 12 17:26:33 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.8.22.24@o2ib6 (stopping) Dec 12 17:26:33 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Dec 12 17:26:36 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200396.97457 Dec 12 17:26:36 fir-md1-s1 kernel: LustreError: 43239:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.110@o2ib7 arrived at 1576200396 with bad export cookie 14105850204140426456 Dec 12 17:26:36 fir-md1-s1 kernel: LustreError: 43239:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 42 previous similar messages Dec 12 17:26:37 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.108.61@o2ib4 (stopping) Dec 12 17:26:37 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Dec 12 17:26:42 fir-md1-s1 kernel: LustreError: 38879:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.111@o2ib7 arrived at 1576200402 with bad export cookie 14105850204141199347 Dec 12 17:26:42 fir-md1-s1 kernel: LustreError: 38879:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 21 previous similar messages Dec 12 17:26:45 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.105.38@o2ib4 (stopping) Dec 12 17:26:45 fir-md1-s1 kernel: Lustre: Skipped 194 previous similar messages Dec 12 17:26:53 fir-md1-s1 kernel: LustreError: 38875:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.109@o2ib7 arrived at 1576200413 with bad export cookie 14105850204140880259 Dec 12 17:26:53 fir-md1-s1 kernel: LustreError: 38875:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 25 previous similar messages Dec 12 17:26:57 fir-md1-s1 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-fir-MDT0000_UUID namespace with 103 resources in use, (rc=-110) Dec 12 17:26:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200417.39266 Dec 12 17:27:02 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.8.0.65@o2ib6 (stopping) Dec 12 17:27:02 fir-md1-s1 kernel: Lustre: Skipped 860 previous similar messages Dec 12 17:27:17 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1576200437.107183 Dec 12 17:27:19 fir-md1-s1 kernel: Lustre: 39344:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-23), not sending early reply req@ffff888bc7722400 x1652733444556736/t0(0) o101->d2bd0014-3bea-4@10.9.114.7@o2ib4:534/0 lens 1792/3288 e 0 to 0 dl 1576200444 ref 2 fl Interpret:/0/0 rc 0/0 Dec 12 17:27:19 fir-md1-s1 kernel: Lustre: 39344:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages Dec 12 17:27:22 fir-md1-s1 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-fir-MDT0000_UUID namespace with 103 resources in use, (rc=-110) Dec 12 17:27:32 fir-md1-s1 kernel: LustreError: 108933:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x20000:0x0], rc:-108 Dec 12 17:27:32 fir-md1-s1 kernel: LustreError: 108933:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 2 previous similar messages Dec 12 17:27:32 fir-md1-s1 kernel: LNet: Service thread pid 39358 completed after 695.19s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Dec 12 17:27:32 fir-md1-s1 kernel: Lustre: 39266:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (628:8s); client may timeout. req@ffff888bc7722400 x1652733444556736/t0(0) o101->d2bd0014-3bea-4@10.9.114.7@o2ib4:534/0 lens 1792/560 e 0 to 0 dl 1576200444 ref 1 fl Complete:/0/0 rc -19/-19 Dec 12 17:27:32 fir-md1-s1 kernel: LNet: Skipped 27 previous similar messages Dec 12 17:28:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.0.10.3@o2ib7 (stopping) Dec 12 17:28:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Dec 12 17:28:19 fir-md1-s1 kernel: LustreError: 108956:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-108 Dec 12 17:28:19 fir-md1-s1 kernel: LustreError: 108956:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 2 previous similar messages Dec 12 17:28:32 fir-md1-s1 kernel: LustreError: 108960:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x20000:0x0], rc:-108 Dec 12 17:28:32 fir-md1-s1 kernel: LustreError: 108960:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 2 previous similar messages Dec 12 17:28:59 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Dec 12 17:29:05 fir-md1-s1 kernel: sched: RT throttling activated Dec 12 17:29:39 fir-md1-s1 kernel: LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 Dec 12 17:29:39 fir-md1-s1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Dec 12 17:29:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Dec 12 17:29:42 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Dec 12 17:29:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Dec 12 17:29:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1271 clients reconnect Dec 12 17:29:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 704e8622-7442-8eb3-b4e3-c86a69ef45af (at 10.8.20.21@o2ib6) Dec 12 17:29:47 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Dec 12 17:29:53 fir-md1-s1 kernel: LustreError: 109508:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff885d5d2e5e80 x1652747909508464/t0(0) o601->fir-MDT0000-lwp-MDT0001_UUID@10.0.10.52@o2ib7:683/0 lens 336/0 e 0 to 0 dl 1576201348 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Dec 12 17:29:53 fir-md1-s1 kernel: LustreError: 109508:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 18 previous similar messages Dec 12 17:29:55 fir-md1-s1 kernel: Lustre: 109440:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Dec 12 17:29:55 fir-md1-s1 kernel: Lustre: 109440:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 36 previous similar messages Dec 12 17:29:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 0:08, of 1271 clients 1271 recovered and 0 were evicted. Dec 12 17:36:54 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 46ad863f-f227-deee-59d2-4b6842f8fe21 (at 10.9.102.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bfa311c00, cur 1576201014 expire 1576200864 last 1576200787 Dec 12 17:36:54 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 17:51:37 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 62f117dd-237d-c074-d679-5244422357ce (at 10.9.103.27@o2ib4) Dec 12 17:51:37 fir-md1-s1 kernel: Lustre: Skipped 1369 previous similar messages Dec 12 18:15:50 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.102.53@o2ib4) Dec 12 18:15:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 18:21:36 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 6784b256-083b-783f-ab9f-d610fc101c63 (at 10.9.104.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888bd0a2a400, cur 1576203696 expire 1576203546 last 1576203469 Dec 12 18:21:36 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 18:36:02 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e6faa273-d68e-7054-d60c-905379aaf1ac (at 10.9.101.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff885bc4e8b000, cur 1576204562 expire 1576204412 last 1576204335 Dec 12 18:36:02 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 19:00:57 fir-md1-s1 kernel: Lustre: MGS: Connection restored to c463879e-71d6-cfb3-b583-923d4925c479 (at 10.9.104.28@o2ib4) Dec 12 19:00:57 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 19:12:35 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 7de2709b-434b-c2b2-ee11-fe99c3a9d16f (at 10.9.101.51@o2ib4) Dec 12 19:12:35 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 20:44:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 935b75df-613a-c7ad-95b7-8cbfb8326a67 (at 10.9.101.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff887bbf04fc00, cur 1576212245 expire 1576212095 last 1576212018 Dec 12 20:44:05 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 20:44:17 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 62818541-2f9e-3fbf-37a6-6cd1b5c2b596 (at 10.9.101.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff888be0888000, cur 1576212257 expire 1576212107 last 1576212030 Dec 12 21:17:42 fir-md1-s1 kernel: Lustre: MGS: Connection restored to fe46e801-2d86-9439-0b24-b78514ed5486 (at 10.9.109.8@o2ib4) Dec 12 21:17:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Dec 12 21:21:24 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 935b75df-613a-c7ad-95b7-8cbfb8326a67 (at 10.9.101.28@o2ib4) Dec 12 21:21:24 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message