-- Logs begin at Mon 2019-11-04 15:38:49 PST, end at Mon 2019-11-04 18:41:01 PST. -- Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys cpuset Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys cpu Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys cpuacct Nov 04 15:38:49 fir-md1-s1 kernel: Linux version 3.10.0-957.27.2.el7_lustre.pl1.x86_64 (sthiell@oak-io4-s2) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 15:38:49 fir-md1-s1 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl1.x86_64 root=UUID=ad8ce4a1-fe7e-4012-89cf-243e2fe869ea ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Nov 04 15:38:49 fir-md1-s1 kernel: e820: BIOS-provided physical RAM map: Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000000100000-0x000000004f882fff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000004f883000-0x000000005788bfff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000005788c000-0x000000006cacefff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006cacf000-0x000000006efcefff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000107f37ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000001080000000-0x000000207ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000002080000000-0x000000307ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x0000003080000000-0x000000407ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: NX (Execute Disable) protection: active Nov 04 15:38:49 fir-md1-s1 kernel: e820: update [mem 0x3705a020-0x3708bc5f] usable ==> usable Nov 04 15:38:49 fir-md1-s1 kernel: e820: update [mem 0x37028020-0x37059c5f] usable ==> usable Nov 04 15:38:49 fir-md1-s1 kernel: e820: update [mem 0x3701f020-0x3702705f] usable ==> usable Nov 04 15:38:49 fir-md1-s1 kernel: e820: update [mem 0x37006020-0x3701e65f] usable ==> usable Nov 04 15:38:49 fir-md1-s1 kernel: extended physical RAM map: Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000000000-0x000000000008efff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000090000-0x000000000009ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000000100000-0x000000003700601f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037006020-0x000000003701e65f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003701e660-0x000000003701f01f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003701f020-0x000000003702705f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037027060-0x000000003702801f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037028020-0x0000000037059c5f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000037059c60-0x000000003705a01f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003705a020-0x000000003708bc5f] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000003708bc60-0x000000004f882fff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000004f883000-0x000000005788bfff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000005788c000-0x000000006cacefff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006cacf000-0x000000006efcefff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x00000000fed80000-0x00000000fed80fff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000000100000000-0x000000107f37ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000107f380000-0x000000107fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000001080000000-0x000000207ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000207ff80000-0x000000207fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000002080000000-0x000000307ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000307ff80000-0x000000307fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x0000003080000000-0x000000407ff7ffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: reserve setup_data: [mem 0x000000407ff80000-0x000000407fffffff] reserved Nov 04 15:38:49 fir-md1-s1 kernel: efi: EFI v2.50 by Dell Inc. Nov 04 15:38:49 fir-md1-s1 kernel: efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x6eab5000 SMBIOS 3.0=0x6eab3000 Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem01: type=2, attr=0xf, range=[0x0000000000001000-0x0000000000002000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem02: type=7, attr=0xf, range=[0x0000000000002000-0x0000000000010000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem03: type=3, attr=0xf, range=[0x0000000000010000-0x0000000000014000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem04: type=7, attr=0xf, range=[0x0000000000014000-0x0000000000063000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem05: type=3, attr=0xf, range=[0x0000000000063000-0x000000000008f000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem06: type=10, attr=0xf, range=[0x000000000008f000-0x0000000000090000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem07: type=3, attr=0xf, range=[0x0000000000090000-0x00000000000a0000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem08: type=4, attr=0xf, range=[0x0000000000100000-0x0000000000120000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem09: type=7, attr=0xf, range=[0x0000000000120000-0x0000000000c00000) (10MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem10: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem11: type=2, attr=0xf, range=[0x0000000001000000-0x000000000267b000) (22MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem12: type=7, attr=0xf, range=[0x000000000267b000-0x0000000004000000) (25MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem13: type=4, attr=0xf, range=[0x0000000004000000-0x000000000403b000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem14: type=7, attr=0xf, range=[0x000000000403b000-0x0000000037006000) (815MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem15: type=2, attr=0xf, range=[0x0000000037006000-0x000000004eee6000) (382MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem16: type=7, attr=0xf, range=[0x000000004eee6000-0x000000004eeea000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem17: type=2, attr=0xf, range=[0x000000004eeea000-0x000000004eeec000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem18: type=1, attr=0xf, range=[0x000000004eeec000-0x000000004f009000) (1MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem19: type=2, attr=0xf, range=[0x000000004f009000-0x000000004f128000) (1MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem20: type=1, attr=0xf, range=[0x000000004f128000-0x000000004f237000) (1MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem21: type=3, attr=0xf, range=[0x000000004f237000-0x000000004f883000) (6MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem22: type=0, attr=0xf, range=[0x000000004f883000-0x000000005788c000) (128MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem23: type=3, attr=0xf, range=[0x000000005788c000-0x000000005796e000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem24: type=4, attr=0xf, range=[0x000000005796e000-0x000000005b4cf000) (59MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem25: type=3, attr=0xf, range=[0x000000005b4cf000-0x000000005b8cf000) (4MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem26: type=7, attr=0xf, range=[0x000000005b8cf000-0x0000000067b63000) (194MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem27: type=4, attr=0xf, range=[0x0000000067b63000-0x0000000067b70000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem28: type=7, attr=0xf, range=[0x0000000067b70000-0x0000000067b74000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem29: type=4, attr=0xf, range=[0x0000000067b74000-0x00000000681aa000) (6MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem30: type=7, attr=0xf, range=[0x00000000681aa000-0x00000000681ab000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem31: type=4, attr=0xf, range=[0x00000000681ab000-0x00000000681b5000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem32: type=7, attr=0xf, range=[0x00000000681b5000-0x00000000681b6000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem33: type=4, attr=0xf, range=[0x00000000681b6000-0x00000000681ba000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem34: type=7, attr=0xf, range=[0x00000000681ba000-0x00000000681bb000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem35: type=4, attr=0xf, range=[0x00000000681bb000-0x00000000681cc000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem36: type=7, attr=0xf, range=[0x00000000681cc000-0x00000000681cd000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem37: type=4, attr=0xf, range=[0x00000000681cd000-0x00000000681d2000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem38: type=7, attr=0xf, range=[0x00000000681d2000-0x00000000681d3000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem39: type=4, attr=0xf, range=[0x00000000681d3000-0x00000000681db000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem40: type=7, attr=0xf, range=[0x00000000681db000-0x00000000681dc000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem41: type=4, attr=0xf, range=[0x00000000681dc000-0x00000000681de000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem42: type=7, attr=0xf, range=[0x00000000681de000-0x00000000681df000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem43: type=4, attr=0xf, range=[0x00000000681df000-0x00000000681f0000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem44: type=7, attr=0xf, range=[0x00000000681f0000-0x00000000681f1000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem45: type=4, attr=0xf, range=[0x00000000681f1000-0x00000000681f4000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem46: type=7, attr=0xf, range=[0x00000000681f4000-0x00000000681f6000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem47: type=4, attr=0xf, range=[0x00000000681f6000-0x00000000681ff000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem48: type=7, attr=0xf, range=[0x00000000681ff000-0x0000000068200000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem49: type=4, attr=0xf, range=[0x0000000068200000-0x0000000068202000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem50: type=7, attr=0xf, range=[0x0000000068202000-0x0000000068203000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem51: type=4, attr=0xf, range=[0x0000000068203000-0x0000000068207000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem52: type=7, attr=0xf, range=[0x0000000068207000-0x0000000068208000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem53: type=4, attr=0xf, range=[0x0000000068208000-0x000000006853d000) (3MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem54: type=7, attr=0xf, range=[0x000000006853d000-0x000000006853e000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem55: type=4, attr=0xf, range=[0x000000006853e000-0x0000000068552000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem56: type=7, attr=0xf, range=[0x0000000068552000-0x0000000068554000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem57: type=4, attr=0xf, range=[0x0000000068554000-0x0000000068564000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem58: type=7, attr=0xf, range=[0x0000000068564000-0x0000000068565000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem59: type=4, attr=0xf, range=[0x0000000068565000-0x000000006857a000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem60: type=7, attr=0xf, range=[0x000000006857a000-0x000000006857b000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem61: type=4, attr=0xf, range=[0x000000006857b000-0x000000006858b000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem62: type=7, attr=0xf, range=[0x000000006858b000-0x000000006858c000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem63: type=4, attr=0xf, range=[0x000000006858c000-0x00000000685b4000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem64: type=7, attr=0xf, range=[0x00000000685b4000-0x00000000685b5000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem65: type=4, attr=0xf, range=[0x00000000685b5000-0x00000000685cf000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem66: type=7, attr=0xf, range=[0x00000000685cf000-0x00000000685d0000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem67: type=4, attr=0xf, range=[0x00000000685d0000-0x00000000685eb000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem68: type=7, attr=0xf, range=[0x00000000685eb000-0x00000000685ec000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem69: type=4, attr=0xf, range=[0x00000000685ec000-0x000000006862f000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem70: type=7, attr=0xf, range=[0x000000006862f000-0x0000000068630000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem71: type=4, attr=0xf, range=[0x0000000068630000-0x0000000068641000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem72: type=7, attr=0xf, range=[0x0000000068641000-0x0000000068643000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem73: type=4, attr=0xf, range=[0x0000000068643000-0x0000000068648000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem74: type=7, attr=0xf, range=[0x0000000068648000-0x0000000068649000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem75: type=4, attr=0xf, range=[0x0000000068649000-0x0000000068658000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem76: type=7, attr=0xf, range=[0x0000000068658000-0x0000000068659000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem77: type=4, attr=0xf, range=[0x0000000068659000-0x000000006867a000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem78: type=7, attr=0xf, range=[0x000000006867a000-0x000000006867b000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem79: type=4, attr=0xf, range=[0x000000006867b000-0x00000000686da000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem80: type=7, attr=0xf, range=[0x00000000686da000-0x00000000686db000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem81: type=4, attr=0xf, range=[0x00000000686db000-0x00000000686de000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem82: type=7, attr=0xf, range=[0x00000000686de000-0x00000000686df000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem83: type=4, attr=0xf, range=[0x00000000686df000-0x00000000686e5000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem84: type=7, attr=0xf, range=[0x00000000686e5000-0x00000000686e6000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem85: type=4, attr=0xf, range=[0x00000000686e6000-0x00000000686e8000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem86: type=7, attr=0xf, range=[0x00000000686e8000-0x00000000686e9000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem87: type=4, attr=0xf, range=[0x00000000686e9000-0x00000000686ed000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem88: type=7, attr=0xf, range=[0x00000000686ed000-0x00000000686ee000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem89: type=4, attr=0xf, range=[0x00000000686ee000-0x00000000686f6000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem90: type=7, attr=0xf, range=[0x00000000686f6000-0x00000000686f7000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem91: type=4, attr=0xf, range=[0x00000000686f7000-0x0000000068701000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem92: type=7, attr=0xf, range=[0x0000000068701000-0x0000000068702000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem93: type=4, attr=0xf, range=[0x0000000068702000-0x0000000068704000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem94: type=7, attr=0xf, range=[0x0000000068704000-0x0000000068705000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem95: type=4, attr=0xf, range=[0x0000000068705000-0x0000000068722000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem96: type=7, attr=0xf, range=[0x0000000068722000-0x0000000068723000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem97: type=4, attr=0xf, range=[0x0000000068723000-0x000000006b8cf000) (49MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem98: type=7, attr=0xf, range=[0x000000006b8cf000-0x000000006b8d0000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem99: type=3, attr=0xf, range=[0x000000006b8d0000-0x000000006cacf000) (17MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem100: type=6, attr=0x800000000000000f, range=[0x000000006cacf000-0x000000006cbcf000) (1MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem101: type=5, attr=0x800000000000000f, range=[0x000000006cbcf000-0x000000006cdcf000) (2MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem102: type=0, attr=0xf, range=[0x000000006cdcf000-0x000000006efcf000) (34MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem103: type=10, attr=0xf, range=[0x000000006efcf000-0x000000006fdff000) (14MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem104: type=9, attr=0xf, range=[0x000000006fdff000-0x000000006ffff000) (2MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem105: type=4, attr=0xf, range=[0x000000006ffff000-0x0000000070000000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem106: type=7, attr=0xf, range=[0x0000000100000000-0x000000107f380000) (63475MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem107: type=7, attr=0xf, range=[0x0000001080000000-0x000000207ff80000) (65535MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem108: type=7, attr=0xf, range=[0x0000002080000000-0x000000307ff80000) (65535MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem109: type=7, attr=0xf, range=[0x0000003080000000-0x000000407ff80000) (65535MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem110: type=0, attr=0x9, range=[0x0000000070000000-0x0000000080000000) (256MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem111: type=11, attr=0x800000000000000f, range=[0x0000000080000000-0x0000000090000000) (256MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem112: type=11, attr=0x800000000000000f, range=[0x00000000fec10000-0x00000000fec11000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem113: type=11, attr=0x800000000000000f, range=[0x00000000fed80000-0x00000000fed81000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem114: type=0, attr=0x0, range=[0x000000107f380000-0x0000001080000000) (12MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem115: type=0, attr=0x0, range=[0x000000207ff80000-0x0000002080000000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem116: type=0, attr=0x0, range=[0x000000307ff80000-0x0000003080000000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: efi: mem117: type=0, attr=0x0, range=[0x000000407ff80000-0x0000004080000000) (0MB) Nov 04 15:38:49 fir-md1-s1 kernel: SMBIOS 3.2.0 present. Nov 04 15:38:49 fir-md1-s1 kernel: DMI: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.10.6 08/15/2019 Nov 04 15:38:49 fir-md1-s1 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Nov 04 15:38:49 fir-md1-s1 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Nov 04 15:38:49 fir-md1-s1 kernel: e820: last_pfn = 0x407ff80 max_arch_pfn = 0x400000000 Nov 04 15:38:49 fir-md1-s1 kernel: MTRR default type: uncachable Nov 04 15:38:49 fir-md1-s1 kernel: MTRR fixed ranges enabled: Nov 04 15:38:49 fir-md1-s1 kernel: 00000-9FFFF write-back Nov 04 15:38:49 fir-md1-s1 kernel: A0000-FFFFF uncachable Nov 04 15:38:49 fir-md1-s1 kernel: MTRR variable ranges enabled: Nov 04 15:38:49 fir-md1-s1 kernel: 0 base 0000FF000000 mask FFFFFF000000 write-protect Nov 04 15:38:49 fir-md1-s1 kernel: 1 base 000000000000 mask FFFF80000000 write-back Nov 04 15:38:49 fir-md1-s1 kernel: 2 base 000070000000 mask FFFFF0000000 uncachable Nov 04 15:38:49 fir-md1-s1 kernel: 3 disabled Nov 04 15:38:49 fir-md1-s1 kernel: 4 disabled Nov 04 15:38:49 fir-md1-s1 kernel: 5 disabled Nov 04 15:38:49 fir-md1-s1 kernel: 6 disabled Nov 04 15:38:49 fir-md1-s1 kernel: 7 disabled Nov 04 15:38:49 fir-md1-s1 kernel: TOM2: 0000004080000000 aka 264192M Nov 04 15:38:49 fir-md1-s1 kernel: PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC Nov 04 15:38:49 fir-md1-s1 kernel: e820: last_pfn = 0x70000 max_arch_pfn = 0x400000000 Nov 04 15:38:49 fir-md1-s1 kernel: Base memory trampoline at [ffffa0fe00099000] 99000 size 24576 Nov 04 15:38:49 fir-md1-s1 kernel: Using GB pages for direct mapping Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650053000, 0x3650053fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650054000, 0x3650054fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650055000, 0x3650055fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650056000, 0x3650056fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650057000, 0x3650057fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650058000, 0x3650058fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x3650059000, 0x3650059fff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x365005a000, 0x365005afff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x365005b000, 0x365005bfff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x365005c000, 0x365005cfff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x365005d000, 0x365005dfff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: BRK [0x365005e000, 0x365005efff] PGTABLE Nov 04 15:38:49 fir-md1-s1 kernel: RAMDISK: [mem 0x3708c000-0x383d1fff] Nov 04 15:38:49 fir-md1-s1 kernel: Early table checksum verification disabled Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: RSDP 000000006fffe014 00024 (v02 DELL ) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: XSDT 000000006fffd0e8 000AC (v01 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: FACP 000000006fff0000 00114 (v06 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: DSDT 000000006ffdc000 1038C (v02 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: FACS 000000006fdd3000 00040 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SSDT 000000006fffc000 000D2 (v02 DELL PE_SC3 00000002 MSFT 04000000) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: BERT 000000006fffb000 00030 (v01 DELL BERT 00000001 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: HEST 000000006fffa000 006DC (v01 DELL HEST 00000001 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SSDT 000000006fff9000 00294 (v01 DELL PE_SC3 00000001 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SRAT 000000006fff8000 00420 (v03 DELL PE_SC3 00000001 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: MSCT 000000006fff7000 0004E (v01 DELL PE_SC3 00000000 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SLIT 000000006fff6000 0003C (v01 DELL PE_SC3 00000001 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: CRAT 000000006fff3000 02DC0 (v01 DELL PE_SC3 00000001 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: EINJ 000000006fff2000 00150 (v01 DELL PE_SC3 00000001 AMD 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SLIC 000000006fff1000 00024 (v01 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: HPET 000000006ffef000 00038 (v01 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: APIC 000000006ffee000 004B2 (v03 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: MCFG 000000006ffed000 0003C (v01 DELL PE_SC3 00000002 DELL 00000001) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SSDT 000000006ffdb000 00629 (v02 DELL xhc_port 00000001 INTL 20170119) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IVRS 000000006ffda000 00210 (v02 DELL PE_SC3 00000001 AMD 00000000) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: SSDT 000000006ffd8000 01658 (v01 AMD CPMCMN 00000001 INTL 20170119) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x01 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x02 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x03 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x04 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x05 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x08 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x09 -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0a -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0b -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0c -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 0 -> APIC 0x0d -> Node 0 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x10 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x11 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x12 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x13 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x14 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x15 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x18 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x19 -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1a -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1b -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1c -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 1 -> APIC 0x1d -> Node 1 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x20 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x21 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x22 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x23 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x24 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x25 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x28 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x29 -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2a -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2b -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2c -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 2 -> APIC 0x2d -> Node 2 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x30 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x31 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x32 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x33 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x34 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x35 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x38 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x39 -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3a -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3b -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3c -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: PXM 3 -> APIC 0x3d -> Node 3 Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x00100000-0x7fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 2 PXM 2 [mem 0x2080000000-0x307fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: SRAT: Node 3 PXM 3 [mem 0x3080000000-0x407fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: NUMA: Initialized distance table, cnt=4 Nov 04 15:38:49 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: NODE_DATA(0) allocated [mem 0x107f359000-0x107f37ffff] Nov 04 15:38:49 fir-md1-s1 kernel: NODE_DATA(1) allocated [mem 0x207ff59000-0x207ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: NODE_DATA(2) allocated [mem 0x307ff59000-0x307ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: NODE_DATA(3) allocated [mem 0x407ff58000-0x407ff7efff] Nov 04 15:38:49 fir-md1-s1 kernel: Reserving 176MB of memory at 704MB for crashkernel (System RAM: 261692MB) Nov 04 15:38:49 fir-md1-s1 kernel: Zone ranges: Nov 04 15:38:49 fir-md1-s1 kernel: DMA [mem 0x00001000-0x00ffffff] Nov 04 15:38:49 fir-md1-s1 kernel: DMA32 [mem 0x01000000-0xffffffff] Nov 04 15:38:49 fir-md1-s1 kernel: Normal [mem 0x100000000-0x407ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: Movable zone start for each node Nov 04 15:38:49 fir-md1-s1 kernel: Early memory node ranges Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x00001000-0x0008efff] Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x00090000-0x0009ffff] Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x00100000-0x4f882fff] Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x5788c000-0x6cacefff] Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x6ffff000-0x6fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: node 0: [mem 0x100000000-0x107f37ffff] Nov 04 15:38:49 fir-md1-s1 kernel: node 1: [mem 0x1080000000-0x207ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: node 2: [mem 0x2080000000-0x307ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: node 3: [mem 0x3080000000-0x407ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: Initmem setup node 0 [mem 0x00001000-0x107f37ffff] Nov 04 15:38:49 fir-md1-s1 kernel: On node 0 totalpages: 16661989 Nov 04 15:38:49 fir-md1-s1 kernel: DMA zone: 64 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: DMA zone: 1126 pages reserved Nov 04 15:38:49 fir-md1-s1 kernel: DMA zone: 3998 pages, LIFO batch:0 Nov 04 15:38:49 fir-md1-s1 kernel: DMA32 zone: 6380 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: DMA32 zone: 408263 pages, LIFO batch:31 Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 253902 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 16249728 pages, LIFO batch:31 Nov 04 15:38:49 fir-md1-s1 kernel: Initmem setup node 1 [mem 0x1080000000-0x207ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: On node 1 totalpages: 16777088 Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Nov 04 15:38:49 fir-md1-s1 kernel: Initmem setup node 2 [mem 0x2080000000-0x307ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: On node 2 totalpages: 16777088 Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Nov 04 15:38:49 fir-md1-s1 kernel: Initmem setup node 3 [mem 0x3080000000-0x407ff7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: On node 3 totalpages: 16777088 Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 262142 pages used for memmap Nov 04 15:38:49 fir-md1-s1 kernel: Normal zone: 16777088 pages, LIFO batch:31 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PM-Timer IO Port: 0x408 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Local APIC address 0xfee00000 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x30] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x18] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x28] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x38] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x02] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x12] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x22] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0a] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2a] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3a] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x04] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x14] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x24] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x34] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x0c] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x1c] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2c] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x3c] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x01] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x11] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x21] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x31] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x09] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x19] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x29] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x39] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0x03] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0x13] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0x23] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0x33] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0x0b] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0x1b] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0x2b] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0x3b] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0x05] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0x15] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x25] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x35] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x0d] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x1d] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2d] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x3d] enabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x40] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x41] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x42] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x43] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x44] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x45] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x46] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x47] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x48] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x49] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4a] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4b] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4c] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4d] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4e] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x4f] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x50] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x51] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x52] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x53] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x54] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x55] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x56] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x57] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x58] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x59] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5a] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5b] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5c] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5d] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5e] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x5f] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x60] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x61] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x62] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x63] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x64] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x65] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x66] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x67] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x68] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x69] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6a] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6b] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6c] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6d] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6e] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x6f] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x70] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x71] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x72] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x73] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x74] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x75] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x76] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x77] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x78] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x79] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7a] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7b] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7c] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7d] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7e] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC (acpi_id[0x7f] lapic_id[0x00] disabled) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x80] address[0xfec00000] gsi_base[0]) Nov 04 15:38:49 fir-md1-s1 kernel: IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x81] address[0xfd880000] gsi_base[24]) Nov 04 15:38:49 fir-md1-s1 kernel: IOAPIC[1]: apic_id 129, version 33, address 0xfd880000, GSI 24-55 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x82] address[0xe0900000] gsi_base[56]) Nov 04 15:38:49 fir-md1-s1 kernel: IOAPIC[2]: apic_id 130, version 33, address 0xe0900000, GSI 56-87 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x83] address[0xc5900000] gsi_base[88]) Nov 04 15:38:49 fir-md1-s1 kernel: IOAPIC[3]: apic_id 131, version 33, address 0xc5900000, GSI 88-119 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IOAPIC (id[0x84] address[0xaa900000] gsi_base[120]) Nov 04 15:38:49 fir-md1-s1 kernel: IOAPIC[4]: apic_id 132, version 33, address 0xaa900000, GSI 120-151 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IRQ0 used by override. Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: IRQ9 used by override. Nov 04 15:38:49 fir-md1-s1 kernel: Using ACPI (MADT) for SMP configuration information Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: HPET id: 0x10228201 base: 0xfed00000 Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Allowing 128 CPUs, 80 hotplug CPUs Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x0008f000-0x0008ffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37006000-0x37006fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3701e000-0x3701efff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3701f000-0x3701ffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37027000-0x37027fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37028000-0x37028fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x37059000-0x37059fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3705a000-0x3705afff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x3708b000-0x3708bfff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x4f883000-0x5788bfff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6cacf000-0x6efcefff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6efcf000-0x6fdfefff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x6fdff000-0x6fffefff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x70000000-0x8fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x90000000-0xfec0ffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec10000-0xfec10fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfec11000-0xfed7ffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed80000-0xfed80fff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0xfed81000-0xffffffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x107f380000-0x107fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x207ff80000-0x207fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registered nosave memory: [mem 0x307ff80000-0x307fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: [mem 0x90000000-0xfec0ffff] available for PCI devices Nov 04 15:38:49 fir-md1-s1 kernel: Booting paravirtualized kernel on bare hardware Nov 04 15:38:49 fir-md1-s1 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:4 Nov 04 15:38:49 fir-md1-s1 kernel: PERCPU: Embedded 38 pages/cpu @ffffa10e3ee00000 s118784 r8192 d28672 u262144 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [0] 000 004 008 012 016 020 024 028 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [0] 032 036 040 044 048 052 056 060 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [0] 064 068 072 076 080 084 088 092 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [0] 096 100 104 108 112 116 120 124 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [1] 001 005 009 013 017 021 025 029 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [1] 033 037 041 045 049 053 057 061 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [1] 065 069 073 077 081 085 089 093 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [1] 097 101 105 109 113 117 121 125 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [2] 002 006 010 014 018 022 026 030 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [2] 034 038 042 046 050 054 058 062 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [2] 066 070 074 078 082 086 090 094 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [2] 098 102 106 110 114 118 122 126 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [3] 003 007 011 015 019 023 027 031 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [3] 035 039 043 047 051 055 059 063 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [3] 067 071 075 079 083 087 091 095 Nov 04 15:38:49 fir-md1-s1 kernel: pcpu-alloc: [3] 099 103 107 111 115 119 123 127 Nov 04 15:38:49 fir-md1-s1 kernel: Built 4 zonelists in Zone order, mobility grouping on. Total pages: 65945355 Nov 04 15:38:49 fir-md1-s1 kernel: Policy zone: Normal Nov 04 15:38:49 fir-md1-s1 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl1.x86_64 root=UUID=ad8ce4a1-fe7e-4012-89cf-243e2fe869ea ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Nov 04 15:38:49 fir-md1-s1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 Nov 04 15:38:49 fir-md1-s1 kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form Nov 04 15:38:49 fir-md1-s1 kernel: Memory: 9613424k/270532096k available (7676k kernel code, 2559084k absent, 4654536k reserved, 6045k data, 1876k init) Nov 04 15:38:49 fir-md1-s1 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=4 Nov 04 15:38:49 fir-md1-s1 kernel: Hierarchical RCU implementation. Nov 04 15:38:49 fir-md1-s1 kernel: RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. Nov 04 15:38:49 fir-md1-s1 kernel: NR_IRQS:327936 nr_irqs:3624 0 Nov 04 15:38:49 fir-md1-s1 kernel: Console: colour dummy device 80x25 Nov 04 15:38:49 fir-md1-s1 kernel: console [ttyS0] enabled Nov 04 15:38:49 fir-md1-s1 kernel: allocated 1072693248 bytes of page_cgroup Nov 04 15:38:49 fir-md1-s1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Nov 04 15:38:49 fir-md1-s1 kernel: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl Nov 04 15:38:49 fir-md1-s1 kernel: hpet clockevent registered Nov 04 15:38:49 fir-md1-s1 kernel: tsc: Fast TSC calibration using PIT Nov 04 15:38:49 fir-md1-s1 kernel: tsc: Detected 1996.203 MHz processor Nov 04 15:38:49 fir-md1-s1 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3992.40 BogoMIPS (lpj=1996203) Nov 04 15:38:49 fir-md1-s1 kernel: pid_max: default: 131072 minimum: 1024 Nov 04 15:38:49 fir-md1-s1 kernel: Security Framework initialized Nov 04 15:38:49 fir-md1-s1 kernel: SELinux: Initializing. Nov 04 15:38:49 fir-md1-s1 kernel: SELinux: Starting in permissive mode Nov 04 15:38:49 fir-md1-s1 kernel: Yama: becoming mindful. Nov 04 15:38:49 fir-md1-s1 kernel: Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys memory Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys devices Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys freezer Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys net_cls Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys blkio Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys perf_event Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys hugetlb Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys pids Nov 04 15:38:49 fir-md1-s1 kernel: Initializing cgroup subsys net_prio Nov 04 15:38:49 fir-md1-s1 kernel: tseg: 0070000000 Nov 04 15:38:49 fir-md1-s1 kernel: LVT offset 2 assigned for vector 0xf4 Nov 04 15:38:49 fir-md1-s1 kernel: Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 Nov 04 15:38:49 fir-md1-s1 kernel: Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768 Nov 04 15:38:49 fir-md1-s1 kernel: tlb_flushall_shift: 6 Nov 04 15:38:49 fir-md1-s1 kernel: Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp Nov 04 15:38:49 fir-md1-s1 kernel: FEATURE SPEC_CTRL Not Present Nov 04 15:38:49 fir-md1-s1 kernel: FEATURE IBPB_SUPPORT Present Nov 04 15:38:49 fir-md1-s1 kernel: Spectre V2 : Enabling Indirect Branch Prediction Barrier Nov 04 15:38:49 fir-md1-s1 kernel: Spectre V2 : Mitigation: Full retpoline Nov 04 15:38:49 fir-md1-s1 kernel: Freeing SMP alternatives: 28k freed Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Core revision 20130517 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: All ACPI Tables successfully acquired Nov 04 15:38:49 fir-md1-s1 kernel: ftrace: allocating 29215 entries in 115 pages Nov 04 15:38:49 fir-md1-s1 kernel: Switched APIC routing to physical flat. Nov 04 15:38:49 fir-md1-s1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: CPU0: AMD EPYC 7401P 24-Core Processor (fam: 17, model: 01, stepping: 02) Nov 04 15:38:49 fir-md1-s1 kernel: random: fast init done Nov 04 15:38:49 fir-md1-s1 kernel: APIC calibration not consistent with PM-Timer: 101ms instead of 100ms Nov 04 15:38:49 fir-md1-s1 kernel: APIC delta adjusted to PM-Timer: 623826 (636296) Nov 04 15:38:49 fir-md1-s1 kernel: Performance Events: Fam17h core perfctr, AMD PMU driver. Nov 04 15:38:49 fir-md1-s1 kernel: ... version: 0 Nov 04 15:38:49 fir-md1-s1 kernel: ... bit width: 48 Nov 04 15:38:49 fir-md1-s1 kernel: ... generic registers: 6 Nov 04 15:38:49 fir-md1-s1 kernel: ... value mask: 0000ffffffffffff Nov 04 15:38:49 fir-md1-s1 kernel: ... max period: 00007fffffffffff Nov 04 15:38:49 fir-md1-s1 kernel: ... fixed-purpose events: 0 Nov 04 15:38:49 fir-md1-s1 kernel: ... event mask: 000000000000003f Nov 04 15:38:49 fir-md1-s1 kernel: NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #1 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #2 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #3 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #4 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #5 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #6 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #7 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #8 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #9 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #10 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #11 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #12 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #13 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #14 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #15 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #16 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #17 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #18 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #19 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #20 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #21 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #22 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #23 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #24 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #25 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #26 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #27 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #28 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #29 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #30 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #31 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #32 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #33 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #34 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #35 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #36 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #37 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #38 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #39 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #40 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #41 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #42 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #43 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 0, Processors #44 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 1, Processors #45 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 2, Processors #46 OK Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Booting Node 3, Processors #47 Nov 04 15:38:49 fir-md1-s1 kernel: Brought up 48 CPUs Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Max logical packages: 3 Nov 04 15:38:49 fir-md1-s1 kernel: smpboot: Total of 48 processors activated (191635.48 BogoMIPS) Nov 04 15:38:49 fir-md1-s1 kernel: node 0 initialised, 15462980 pages in 273ms Nov 04 15:38:49 fir-md1-s1 kernel: node 1 initialised, 15989367 pages in 277ms Nov 04 15:38:49 fir-md1-s1 kernel: node 2 initialised, 15989367 pages in 278ms Nov 04 15:38:49 fir-md1-s1 kernel: node 3 initialised, 15984547 pages in 284ms Nov 04 15:38:49 fir-md1-s1 kernel: devtmpfs: initialized Nov 04 15:38:49 fir-md1-s1 kernel: EVM: security.selinux Nov 04 15:38:49 fir-md1-s1 kernel: EVM: security.ima Nov 04 15:38:49 fir-md1-s1 kernel: EVM: security.capability Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: PM: Registering ACPI NVS region [mem 0x6efcf000-0x6fdfefff] (14876672 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: atomic64 test passed for x86-64 platform with CX8 and with SSE Nov 04 15:38:49 fir-md1-s1 kernel: pinctrl core: initialized pinctrl subsystem Nov 04 15:38:49 fir-md1-s1 kernel: RTC time: 23:38:44, date: 11/04/19 Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 16 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: bus type PCI registered Nov 04 15:38:49 fir-md1-s1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Nov 04 15:38:49 fir-md1-s1 kernel: PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) Nov 04 15:38:49 fir-md1-s1 kernel: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 Nov 04 15:38:49 fir-md1-s1 kernel: PCI: Using configuration type 1 for base access Nov 04 15:38:49 fir-md1-s1 kernel: PCI: Dell System detected, enabling pci=bfsort. Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Added _OSI(Module Device) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Device) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Added _OSI(3.0 _SCP Extensions) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Added _OSI(Processor Aggregator Device) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Added _OSI(Linux-Dell-Video) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: EC: Look up EC in DSDT Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Executed 2 blocks of module-level executable AML code Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Interpreter enabled Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: (supports S0 S5) Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Using IOAPIC for interrupt routing Nov 04 15:38:49 fir-md1-s1 kernel: HEST: Table parsing has been initialized. Nov 04 15:38:49 fir-md1-s1 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Enabled 1 GPEs in block 00 to 1F Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-3f]) Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:00: PCIe AER handled by firmware Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration Nov 04 15:38:49 fir-md1-s1 kernel: PCI host bridge to bus 0000:00 Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [io 0x0d00-0x3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0xe1000000-0xfebfffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [mem 0x10000000000-0x2bf3fffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: root bus resource [bus 00-3f] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:00.0: [1022:1450] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:00.2: [1022:1451] type 00 class 0x080600 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:01.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:02.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: [1022:1453] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:04.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.0: [1022:1460] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.1: [1022:1461] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.2: [1022:1462] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.3: [1022:1463] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.4: [1022:1464] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.5: [1022:1465] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.6: [1022:1466] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:18.7: [1022:1467] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.0: [1022:1460] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.1: [1022:1461] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.2: [1022:1462] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.3: [1022:1463] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.4: [1022:1464] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.5: [1022:1465] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.6: [1022:1466] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:19.7: [1022:1467] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.0: [1022:1460] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.1: [1022:1461] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.2: [1022:1462] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.3: [1022:1463] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.4: [1022:1464] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.5: [1022:1465] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.6: [1022:1466] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1a.7: [1022:1467] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.0: [1022:1460] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.1: [1022:1461] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.2: [1022:1462] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.3: [1022:1463] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.4: [1022:1464] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.5: [1022:1465] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.6: [1022:1466] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:1b.7: [1022:1467] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: [15b3:101b] type 00 class 0x020700 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x10: [mem 0xe2000000-0xe3ffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: PME# supported from D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.0: [1022:145a] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.2: [1022:1456] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x18: [mem 0xf7300000-0xf73fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.2: reg 0x24: [mem 0xf7400000-0xf7401fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.3: [1022:145f] type 00 class 0x0c0330 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.3: reg 0x10: [mem 0xf7200000-0xf72fffff 64bit] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.3: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.0: [1022:1455] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.1: [1022:1468] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x18: [mem 0xf7000000-0xf70fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.1: reg 0x24: [mem 0xf7100000-0xf7101fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: on NUMA node 0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 40-7f]) Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:01: PCIe AER handled by firmware Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration Nov 04 15:38:49 fir-md1-s1 kernel: PCI host bridge to bus 0000:40 Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0xc6000000-0xe0ffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [mem 0x2bf40000000-0x47e7fffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: root bus resource [bus 40-7f] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:00.0: [1022:1450] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:00.2: [1022:1451] type 00 class 0x080600 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:01.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:02.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:03.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:04.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.0: [1022:145a] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.2: [1022:1456] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x18: [mem 0xdb300000-0xdb3fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.2: reg 0x24: [mem 0xdb400000-0xdb401fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.3: [1022:145f] type 00 class 0x0c0330 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.3: reg 0x10: [mem 0xdb200000-0xdb2fffff 64bit] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.3: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.0: [1022:1455] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.1: [1022:1468] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x18: [mem 0xdb000000-0xdb0fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.1: reg 0x24: [mem 0xdb100000-0xdb101fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: on NUMA node 1 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 80-bf]) Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:02: PCIe AER handled by firmware Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration Nov 04 15:38:49 fir-md1-s1 kernel: PCI host bridge to bus 0000:80 Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0xab000000-0xc5ffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [mem 0x47e80000000-0x63dbfffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: root bus resource [bus 80-bf] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:00.0: [1022:1450] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:00.2: [1022:1451] type 00 class 0x080600 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: [1022:1453] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: [1022:1453] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:02.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: [1022:1453] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:04.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: [14e4:165f] type 00 class 0x020000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x10: [mem 0xac230000-0xac23ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x18: [mem 0xac240000-0xac24ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x20: [mem 0xac250000-0xac25ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: [14e4:165f] type 00 class 0x020000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x10: [mem 0xac200000-0xac20ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x18: [mem 0xac210000-0xac21ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x20: [mem 0xac220000-0xac22ffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: [1556:be00] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: [102b:0536] type 00 class 0x030000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x10: [mem 0xab000000-0xabffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x14: [mem 0xc0808000-0xc080bfff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: reg 0x18: [mem 0xc0000000-0xc07fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: [1000:00d1] type 00 class 0x010700 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x10: [mem 0xac000000-0xac0fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x18: [mem 0xac100000-0xac1fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x20: [mem 0xc0d00000-0xc0dfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x24: [io 0x8000-0x80ff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: supports D1 D2 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.0: [1022:145a] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.2: [1022:1456] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x18: [mem 0xc0b00000-0xc0bfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.2: reg 0x24: [mem 0xc0c00000-0xc0c01fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.0: [1022:1455] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.1: [1022:1468] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x18: [mem 0xc0900000-0xc09fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.1: reg 0x24: [mem 0xc0a00000-0xc0a01fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.2: [1022:7901] type 00 class 0x010601 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.2: reg 0x24: [mem 0xc0a02000-0xc0a02fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.2: PME# supported from D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: on NUMA node 2 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: PCI Root Bridge [PC03] (domain 0000 [bus c0-ff]) Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: PCIe AER handled by firmware Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration Nov 04 15:38:49 fir-md1-s1 kernel: acpi PNP0A08:03: host bridge window [mem 0x63dc0000000-0xffffffffffff window] ([0x80000000000-0xffffffffffff] ignored, not CPU addressable) Nov 04 15:38:49 fir-md1-s1 kernel: PCI host bridge to bus 0000:c0 Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x90000000-0xaaffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [mem 0x63dc0000000-0x7ffffffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: root bus resource [bus c0-ff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:00.0: [1022:1450] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:00.2: [1022:1451] type 00 class 0x080600 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: [1022:1453] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:02.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:03.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:04.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.0: [1022:1452] type 00 class 0x060000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: [1022:1454] type 01 class 0x060400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: [1000:005f] type 00 class 0x010400 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x10: [io 0xc000-0xc0ff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x14: [mem 0xa5500000-0xa550ffff 64bit] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x1c: [mem 0xa5400000-0xa54fffff 64bit] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: supports D1 D2 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.0: [1022:145a] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.2: [1022:1456] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x18: [mem 0xa5200000-0xa52fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.2: reg 0x24: [mem 0xa5300000-0xa5301fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.0: [1022:1455] type 00 class 0x130000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.1: [1022:1468] type 00 class 0x108000 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x18: [mem 0xa5000000-0xa50fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.1: reg 0x24: [mem 0xa5100000-0xa5101fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: on NUMA node 3 Nov 04 15:38:49 fir-md1-s1 kernel: vgaarb: device added: PCI:0000:83:00.0,decodes=io+mem,owns=io+mem,locks=none Nov 04 15:38:49 fir-md1-s1 kernel: vgaarb: loaded Nov 04 15:38:49 fir-md1-s1 kernel: vgaarb: bridge control possible 0000:83:00.0 Nov 04 15:38:49 fir-md1-s1 kernel: SCSI subsystem initialized Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: bus type USB registered Nov 04 15:38:49 fir-md1-s1 kernel: usbcore: registered new interface driver usbfs Nov 04 15:38:49 fir-md1-s1 kernel: usbcore: registered new interface driver hub Nov 04 15:38:49 fir-md1-s1 kernel: usbcore: registered new device driver usb Nov 04 15:38:49 fir-md1-s1 kernel: EDAC MC: Ver: 3.0.0 Nov 04 15:38:49 fir-md1-s1 kernel: PCI: Using ACPI for IRQ routing Nov 04 15:38:49 fir-md1-s1 kernel: PCI: pci_cache_line_size set to 64 bytes Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x37006020-0x37ffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x3701f020-0x37ffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x37028020-0x37ffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x3705a020-0x37ffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x4f883000-0x4fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x6cacf000-0x6fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x107f380000-0x107fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x207ff80000-0x207fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x307ff80000-0x307fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: e820: reserve RAM buffer [mem 0x407ff80000-0x407fffffff] Nov 04 15:38:49 fir-md1-s1 kernel: NetLabel: Initializing Nov 04 15:38:49 fir-md1-s1 kernel: NetLabel: domain hash size = 128 Nov 04 15:38:49 fir-md1-s1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Nov 04 15:38:49 fir-md1-s1 kernel: NetLabel: unlabeled traffic allowed by default Nov 04 15:38:49 fir-md1-s1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Nov 04 15:38:49 fir-md1-s1 kernel: hpet0: 3 comparators, 32-bit 14.318180 MHz counter Nov 04 15:38:49 fir-md1-s1 kernel: Switched to clocksource hpet Nov 04 15:38:49 fir-md1-s1 kernel: pnp: PnP ACPI init Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: bus type PNP registered Nov 04 15:38:49 fir-md1-s1 kernel: system 00:00: [mem 0x80000000-0x8fffffff] has been reserved Nov 04 15:38:49 fir-md1-s1 kernel: system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) Nov 04 15:38:49 fir-md1-s1 kernel: pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) Nov 04 15:38:49 fir-md1-s1 kernel: pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) Nov 04 15:38:49 fir-md1-s1 kernel: pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) Nov 04 15:38:49 fir-md1-s1 kernel: pnp: PnP ACPI: found 4 devices Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: bus type PNP unregistered Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: BAR 14: assigned [mem 0xe1000000-0xe10fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: BAR 6: assigned [mem 0xe1000000-0xe10fffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe10fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: PCI bridge to [bus 02] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: PCI bridge to [bus 03] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 10 [mem 0x000d0000-0x000d3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 11 [mem 0x000d4000-0x000d7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 12 [mem 0x000d8000-0x000dbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 13 [mem 0x000dc000-0x000dffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 14 [mem 0x000e0000-0x000e3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 18 [mem 0x000f0000-0x000fffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 19 [io 0x0d00-0x3fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 20 [mem 0xe1000000-0xfebfffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:00: resource 21 [mem 0x10000000000-0x2bf3fffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:01: resource 1 [mem 0xe1000000-0xe10fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:01: resource 2 [mem 0xe2000000-0xe3ffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:02: resource 1 [mem 0xf7200000-0xf74fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:03: resource 1 [mem 0xf7000000-0xf71fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: PCI bridge to [bus 41] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: PCI bridge to [bus 42] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: resource 4 [io 0x4000-0x7fff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: resource 5 [mem 0xc6000000-0xe0ffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:40: resource 6 [mem 0x2bf40000000-0x47e7fffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:41: resource 1 [mem 0xdb200000-0xdb4fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:42: resource 1 [mem 0xdb000000-0xdb1fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: BAR 14: assigned [mem 0xac300000-0xac3fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: BAR 6: assigned [mem 0xac300000-0xac33ffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: BAR 6: assigned [mem 0xac340000-0xac37ffff pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: PCI bridge to [bus 81] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac300000-0xac3fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: PCI bridge to [bus 83] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: PCI bridge to [bus 82-83] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: BAR 6: no space for [mem size 0x00040000 pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: BAR 6: failed to assign [mem size 0x00040000 pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: PCI bridge to [bus 84] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: PCI bridge to [bus 85] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: PCI bridge to [bus 86] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: resource 5 [mem 0x000a0000-0x000bffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: resource 6 [io 0x8000-0xbfff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: resource 7 [mem 0xab000000-0xc5ffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:80: resource 8 [mem 0x47e80000000-0x63dbfffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:81: resource 1 [mem 0xac300000-0xac3fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:81: resource 2 [mem 0xac200000-0xac2fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:82: resource 1 [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:82: resource 2 [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:83: resource 1 [mem 0xc0000000-0xc08fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:83: resource 2 [mem 0xab000000-0xabffffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:84: resource 0 [io 0x8000-0x8fff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:84: resource 1 [mem 0xc0d00000-0xc0dfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:84: resource 2 [mem 0xac000000-0xac1fffff 64bit pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:85: resource 1 [mem 0xc0b00000-0xc0cfffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:86: resource 1 [mem 0xc0900000-0xc0afffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: no space for [mem size 0x00100000 pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: PCI bridge to [bus c1] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: PCI bridge to [bus c2] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: PCI bridge to [bus c3] Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: resource 5 [mem 0x90000000-0xaaffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c0: resource 6 [mem 0x63dc0000000-0x7ffffffffff window] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c1: resource 0 [io 0xc000-0xcfff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c1: resource 1 [mem 0xa5400000-0xa55fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c2: resource 1 [mem 0xa5200000-0xa53fffff] Nov 04 15:38:49 fir-md1-s1 kernel: pci_bus 0000:c3: resource 1 [mem 0xa5000000-0xa51fffff] Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 2 Nov 04 15:38:49 fir-md1-s1 kernel: TCP established hash table entries: 524288 (order: 10, 4194304 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: TCP: Hash tables configured (established 524288 bind 65536) Nov 04 15:38:49 fir-md1-s1 kernel: TCP: reno registered Nov 04 15:38:49 fir-md1-s1 kernel: UDP hash table entries: 65536 (order: 9, 2097152 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 1 Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: Boot video device Nov 04 15:38:49 fir-md1-s1 kernel: PCI: CLS 64 bytes, default 64 Nov 04 15:38:49 fir-md1-s1 kernel: Unpacking initramfs... Nov 04 15:38:49 fir-md1-s1 kernel: Freeing initrd memory: 19736k freed Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: IOMMU performance counters supported Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:01.0 to group 0 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:02.0 to group 1 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.0 to group 2 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:03.1 to group 3 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:04.0 to group 4 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.0 to group 5 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:07.1 to group 6 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.0 to group 7 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:08.1 to group 8 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.0 to group 9 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:14.3 to group 9 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.0 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.1 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.2 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.3 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.4 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.5 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.6 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:18.7 to group 10 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.0 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.1 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.2 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.3 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.4 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.5 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.6 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:19.7 to group 11 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.0 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.1 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.2 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.3 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.4 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.5 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.6 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1a.7 to group 12 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.0 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.1 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.2 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.3 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.4 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.5 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.6 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:00:1b.7 to group 13 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:01:00.0 to group 14 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.0 to group 15 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.2 to group 16 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:02:00.3 to group 17 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.0 to group 18 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:03:00.1 to group 19 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:01.0 to group 20 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:02.0 to group 21 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:03.0 to group 22 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:04.0 to group 23 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.0 to group 24 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:07.1 to group 25 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.0 to group 26 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:40:08.1 to group 27 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.0 to group 28 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.2 to group 29 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:41:00.3 to group 30 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.0 to group 31 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:42:00.1 to group 32 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.0 to group 33 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.1 to group 34 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:01.2 to group 35 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:02.0 to group 36 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.0 to group 37 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:03.1 to group 38 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:04.0 to group 39 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.0 to group 40 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:07.1 to group 41 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.0 to group 42 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:80:08.1 to group 43 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.0 to group 44 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:81:00.1 to group 44 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:82:00.0 to group 45 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:83:00.0 to group 45 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:84:00.0 to group 46 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.0 to group 47 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:85:00.2 to group 48 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.0 to group 49 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.1 to group 50 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:86:00.2 to group 51 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.0 to group 52 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:01.1 to group 53 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:02.0 to group 54 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:03.0 to group 55 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:04.0 to group 56 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.0 to group 57 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:07.1 to group 58 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.0 to group 59 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c0:08.1 to group 60 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c1:00.0 to group 61 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.0 to group 62 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c2:00.2 to group 63 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.0 to group 64 Nov 04 15:38:49 fir-md1-s1 kernel: iommu: Adding device 0000:c3:00.1 to group 65 Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Nov 04 15:38:49 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Nov 04 15:38:49 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:80:00.2 cap 0x40 Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Nov 04 15:38:49 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Found IOMMU at 0000:c0:00.2 cap 0x40 Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Extended features (0xf77ef22294ada): Nov 04 15:38:49 fir-md1-s1 kernel: PPR NX GT IA GA PC GA_vAPIC Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Interrupt remapping enabled Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: virtual APIC enabled Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:00:00.2: irq 26 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:40:00.2: irq 27 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:80:00.2: irq 28 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c0:00.2: irq 29 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: AMD-Vi: Lazy IO/TLB flushing enabled Nov 04 15:38:49 fir-md1-s1 kernel: perf: AMD NB counters detected Nov 04 15:38:49 fir-md1-s1 kernel: perf: AMD LLC counters detected Nov 04 15:38:49 fir-md1-s1 kernel: sha1_ssse3: Using SHA-NI optimized SHA-1 implementation Nov 04 15:38:49 fir-md1-s1 kernel: sha256_ssse3: Using SHA-256-NI optimized SHA-256 implementation Nov 04 15:38:49 fir-md1-s1 kernel: futex hash table entries: 32768 (order: 9, 2097152 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: Initialise system trusted keyring Nov 04 15:38:49 fir-md1-s1 kernel: audit: initializing netlink socket (disabled) Nov 04 15:38:49 fir-md1-s1 kernel: type=2000 audit(1572910722.196:1): initialized Nov 04 15:38:49 fir-md1-s1 kernel: HugeTLB registered 1 GB page size, pre-allocated 0 pages Nov 04 15:38:49 fir-md1-s1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Nov 04 15:38:49 fir-md1-s1 kernel: zpool: loaded Nov 04 15:38:49 fir-md1-s1 kernel: zbud: loaded Nov 04 15:38:49 fir-md1-s1 kernel: VFS: Disk quotas dquot_6.6.0 Nov 04 15:38:49 fir-md1-s1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Nov 04 15:38:49 fir-md1-s1 kernel: msgmni has been set to 32768 Nov 04 15:38:49 fir-md1-s1 kernel: Key type big_key registered Nov 04 15:38:49 fir-md1-s1 kernel: SELinux: Registering netfilter hooks Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 38 Nov 04 15:38:49 fir-md1-s1 kernel: Key type asymmetric registered Nov 04 15:38:49 fir-md1-s1 kernel: Asymmetric key parser 'x509' registered Nov 04 15:38:49 fir-md1-s1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) Nov 04 15:38:49 fir-md1-s1 kernel: io scheduler noop registered Nov 04 15:38:49 fir-md1-s1 kernel: io scheduler deadline registered (default) Nov 04 15:38:49 fir-md1-s1 kernel: io scheduler cfq registered Nov 04 15:38:49 fir-md1-s1 kernel: io scheduler mq-deadline registered Nov 04 15:38:49 fir-md1-s1 kernel: io scheduler kyber registered Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:03.1: irq 30 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:07.1: irq 31 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:08.1: irq 33 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:40:07.1: irq 34 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:40:08.1: irq 36 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:01.1: irq 37 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:01.2: irq 38 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:03.1: irq 39 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:07.1: irq 41 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:08.1: irq 43 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:01.1: irq 44 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:07.1: irq 46 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:08.1: irq 48 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:01:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:07.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:02:00.3: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:00:07.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:00:08.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:03:00.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:00:08.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:40:07.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:41:00.3: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:40:07.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:40:08.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:42:00.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:40:08.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:01.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:81:00.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:80:01.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:01.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:82:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:83:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:80:01.2:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:03.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:84:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:80:03.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:07.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:85:00.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:80:07.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:80:08.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:86:00.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:80:08.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:01.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c1:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:c0:01.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:07.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c2:00.2: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:c0:07.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pcieport 0000:c0:08.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.0: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pci 0000:c3:00.1: Signaling PME through PCIe PME interrupt Nov 04 15:38:49 fir-md1-s1 kernel: pcie_pme 0000:c0:08.1:pcie001: service driver pcie_pme loaded Nov 04 15:38:49 fir-md1-s1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Nov 04 15:38:49 fir-md1-s1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Nov 04 15:38:49 fir-md1-s1 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Nov 04 15:38:49 fir-md1-s1 kernel: efifb: probing for efifb Nov 04 15:38:49 fir-md1-s1 kernel: efifb: framebuffer at 0xab000000, mapped to 0xffffb85e19800000, using 3072k, total 3072k Nov 04 15:38:49 fir-md1-s1 kernel: efifb: mode is 1024x768x32, linelength=4096, pages=1 Nov 04 15:38:49 fir-md1-s1 kernel: efifb: scrolling: redraw Nov 04 15:38:49 fir-md1-s1 kernel: efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 Nov 04 15:38:49 fir-md1-s1 kernel: Console: switching to colour frame buffer device 128x48 Nov 04 15:38:49 fir-md1-s1 kernel: fb0: EFI VGA frame buffer device Nov 04 15:38:49 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Power Button [PWRB] Nov 04 15:38:49 fir-md1-s1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 Nov 04 15:38:49 fir-md1-s1 kernel: ACPI: Power Button [PWRF] Nov 04 15:38:49 fir-md1-s1 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. Nov 04 15:38:49 fir-md1-s1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Nov 04 15:38:49 fir-md1-s1 kernel: 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Nov 04 15:38:49 fir-md1-s1 kernel: 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Nov 04 15:38:49 fir-md1-s1 kernel: Non-volatile memory driver v1.3 Nov 04 15:38:49 fir-md1-s1 kernel: Linux agpgart interface v0.103 Nov 04 15:38:49 fir-md1-s1 kernel: crash memory driver: version 1.1 Nov 04 15:38:49 fir-md1-s1 kernel: rdac: device handler registered Nov 04 15:38:49 fir-md1-s1 kernel: hp_sw: device handler registered Nov 04 15:38:49 fir-md1-s1 kernel: emc: device handler registered Nov 04 15:38:49 fir-md1-s1 kernel: alua: device handler registered Nov 04 15:38:49 fir-md1-s1 kernel: libphy: Fixed MDIO Bus: probed Nov 04 15:38:49 fir-md1-s1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Nov 04 15:38:49 fir-md1-s1 kernel: ehci-pci: EHCI PCI platform driver Nov 04 15:38:49 fir-md1-s1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Nov 04 15:38:49 fir-md1-s1 kernel: ohci-pci: OHCI PCI platform driver Nov 04 15:38:49 fir-md1-s1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 1 Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 50 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 51 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 52 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 53 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 54 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 55 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 56 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: irq 57 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb1: Product: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: usb usb1: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 xhci-hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb usb1: SerialNumber: 0000:02:00.3 Nov 04 15:38:49 fir-md1-s1 kernel: hub 1-0:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 1-0:1.0: 2 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 2 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: Product: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 xhci-hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb usb2: SerialNumber: 0000:02:00.3 Nov 04 15:38:49 fir-md1-s1 kernel: hub 2-0:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 2-0:1.0: 2 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 3 Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 59 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 60 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 61 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 62 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 63 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 64 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 65 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: irq 66 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb3: Product: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: usb usb3: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 xhci-hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb usb3: SerialNumber: 0000:41:00.3 Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-0:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-0:1.0: 2 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 4 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: Product: xHCI Host Controller Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 xhci-hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb usb4: SerialNumber: 0000:41:00.3 Nov 04 15:38:49 fir-md1-s1 kernel: hub 4-0:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 4-0:1.0: 2 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: usbcore: registered new interface driver usbserial_generic Nov 04 15:38:49 fir-md1-s1 kernel: usbserial: USB Serial support registered for generic Nov 04 15:38:49 fir-md1-s1 kernel: i8042: PNP: No PS/2 controller found. Probing ports directly. Nov 04 15:38:49 fir-md1-s1 kernel: tsc: Refined TSC clocksource calibration: 1996.248 MHz Nov 04 15:38:49 fir-md1-s1 kernel: i8042: No controller found Nov 04 15:38:49 fir-md1-s1 kernel: usb 1-1: new high-speed USB device number 2 using xhci_hcd Nov 04 15:38:49 fir-md1-s1 kernel: Switched to clocksource tsc Nov 04 15:38:49 fir-md1-s1 kernel: mousedev: PS/2 mouse device common for all mice Nov 04 15:38:49 fir-md1-s1 kernel: rtc_cmos 00:01: RTC can wake from S4 Nov 04 15:38:49 fir-md1-s1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Nov 04 15:38:49 fir-md1-s1 kernel: rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs Nov 04 15:38:49 fir-md1-s1 kernel: cpuidle: using governor menu Nov 04 15:38:49 fir-md1-s1 kernel: EFI Variables Facility v0.08 2004-May-17 Nov 04 15:38:49 fir-md1-s1 kernel: hidraw: raw HID events driver (C) Jiri Kosina Nov 04 15:38:49 fir-md1-s1 kernel: usbcore: registered new interface driver usbhid Nov 04 15:38:49 fir-md1-s1 kernel: usbhid: USB HID core driver Nov 04 15:38:49 fir-md1-s1 kernel: drop_monitor: Initializing network drop monitor service Nov 04 15:38:49 fir-md1-s1 kernel: TCP: cubic registered Nov 04 15:38:49 fir-md1-s1 kernel: Initializing XFRM netlink socket Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 10 Nov 04 15:38:49 fir-md1-s1 kernel: NET: Registered protocol family 17 Nov 04 15:38:49 fir-md1-s1 kernel: mpls_gso: MPLS GSO support Nov 04 15:38:49 fir-md1-s1 kernel: mce: Using 23 MCE banks Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU0: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU1: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU2: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU3: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU4: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU5: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU6: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU7: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU8: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU9: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU10: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU11: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU12: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU13: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU14: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU15: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU16: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU17: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU18: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU19: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU20: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU21: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU22: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU23: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU24: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU25: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU26: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU27: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU28: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU29: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU30: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU31: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU32: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU33: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU34: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU35: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU36: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU37: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU38: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU39: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU40: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU41: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU42: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU43: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU44: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU45: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU46: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: CPU47: patch_level=0x08001250 Nov 04 15:38:49 fir-md1-s1 kernel: microcode: Microcode Update Driver: v2.01 , Peter Oruba Nov 04 15:38:49 fir-md1-s1 kernel: PM: Hibernation image not present or could not be loaded. Nov 04 15:38:49 fir-md1-s1 kernel: Loading compiled-in X.509 certificates Nov 04 15:38:49 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' Nov 04 15:38:49 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' Nov 04 15:38:49 fir-md1-s1 kernel: Loaded X.509 cert 'CentOS Linux kernel signing key: 9e53aba22e464fccc5bb7396174083706426f6e2' Nov 04 15:38:49 fir-md1-s1 kernel: registered taskstats version 1 Nov 04 15:38:49 fir-md1-s1 kernel: Key type trusted registered Nov 04 15:38:49 fir-md1-s1 kernel: Key type encrypted registered Nov 04 15:38:49 fir-md1-s1 kernel: IMA: No TPM chip found, activating TPM-bypass! (rc=-19) Nov 04 15:38:49 fir-md1-s1 kernel: Magic number: 11:353:655 Nov 04 15:38:49 fir-md1-s1 kernel: memory memory1861: hash matches Nov 04 15:38:49 fir-md1-s1 kernel: rtc_cmos 00:01: setting system clock to 2019-11-04 23:38:48 UTC (1572910728) Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1: new high-speed USB device number 2 using xhci_hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb 1-1: New USB device found, idVendor=0424, idProduct=2744 Nov 04 15:38:49 fir-md1-s1 kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Nov 04 15:38:49 fir-md1-s1 kernel: usb 1-1: Product: USB2734 Nov 04 15:38:49 fir-md1-s1 kernel: usb 1-1: Manufacturer: Microchip Tech Nov 04 15:38:49 fir-md1-s1 kernel: hub 1-1:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 1-1:1.0: 4 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1: New USB device found, idVendor=1604, idProduct=10c0 Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-1:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-1:1.0: 4 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd Nov 04 15:38:49 fir-md1-s1 kernel: usb 2-1: New USB device found, idVendor=0424, idProduct=5744 Nov 04 15:38:49 fir-md1-s1 kernel: usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0 Nov 04 15:38:49 fir-md1-s1 kernel: usb 2-1: Product: USB5734 Nov 04 15:38:49 fir-md1-s1 kernel: usb 2-1: Manufacturer: Microchip Tech Nov 04 15:38:49 fir-md1-s1 kernel: hub 2-1:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 2-1:1.0: 4 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: usb: port power management may be unreliable Nov 04 15:38:49 fir-md1-s1 kernel: Freeing unused kernel memory: 1876k freed Nov 04 15:38:49 fir-md1-s1 kernel: Write protecting the kernel read-only data: 12288k Nov 04 15:38:49 fir-md1-s1 kernel: Freeing unused kernel memory: 504k freed Nov 04 15:38:49 fir-md1-s1 kernel: Freeing unused kernel memory: 596k freed Nov 04 15:38:49 fir-md1-s1 systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Nov 04 15:38:49 fir-md1-s1 systemd[1]: Detected architecture x86-64. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Running in initial RAM disk. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Set hostname to . Nov 04 15:38:49 fir-md1-s1 systemd[1]: Reached target Swap. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Created slice Root Slice. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Created slice System Slice. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Reached target Slices. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Listening on udev Kernel Socket. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Listening on Journal Socket. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Starting Setup Virtual Console... Nov 04 15:38:49 fir-md1-s1 systemd[1]: Listening on udev Control Socket. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Starting Create list of required static device nodes for the current kernel... Nov 04 15:38:49 fir-md1-s1 systemd[1]: Reached target Local File Systems. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Reached target Timers. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Starting Journal Service... Nov 04 15:38:49 fir-md1-s1 systemd[1]: Starting dracut cmdline hook... Nov 04 15:38:49 fir-md1-s1 systemd[1]: Reached target Sockets. Nov 04 15:38:49 fir-md1-s1 systemd[1]: Starting Apply Kernel Variables... Nov 04 15:38:49 fir-md1-s1 systemd[1]: Started Journal Service. Nov 04 15:38:49 fir-md1-s1 kernel: pps_core: LinuxPPS API ver. 1 registered Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.1: new high-speed USB device number 3 using xhci_hcd Nov 04 15:38:49 fir-md1-s1 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti Nov 04 15:38:49 fir-md1-s1 kernel: PTP clock support registered Nov 04 15:38:49 fir-md1-s1 kernel: megasas: 07.705.02.00-rh1 Nov 04 15:38:49 fir-md1-s1 kernel: mlx_compat: loading out-of-tree module taints kernel. Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW now in Ready state Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: 64 bit DMA mask and 32 bit consistent mask Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 68 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 69 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 70 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 71 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 72 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 73 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 74 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 75 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 76 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 77 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 78 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 79 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 80 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 81 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 82 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 83 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 84 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 85 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 86 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 87 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 88 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 89 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 90 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 91 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 92 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 93 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 94 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 95 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 96 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 97 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 98 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 99 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 100 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 101 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 102 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 103 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 104 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 105 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 106 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 107 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 108 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 109 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 110 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 111 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 112 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 113 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 114 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: irq 115 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware supports msix : (96) Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: current msix/online cpus : (48/48) Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: RDPQ mode : (disabled) Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Configured max firmware commands: 927 Nov 04 15:38:49 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: FW supports sync cache : No Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.1: New USB device found, idVendor=1604, idProduct=10c0 Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-1.1:1.0: USB hub found Nov 04 15:38:49 fir-md1-s1 kernel: hub 3-1.1:1.0: 4 ports detected Nov 04 15:38:49 fir-md1-s1 kernel: mlx_compat: module verification failed: signature and/or required key missing - tainting kernel Nov 04 15:38:49 fir-md1-s1 kernel: Compat-mlnx-ofed backport release: 1c4bf42 Nov 04 15:38:49 fir-md1-s1 kernel: Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 1c4bf42 Nov 04 15:38:49 fir-md1-s1 kernel: compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.4: new high-speed USB device number 4 using xhci_hcd Nov 04 15:38:49 fir-md1-s1 kernel: tg3.c:v3.137 (May 11, 2014) Nov 04 15:38:49 fir-md1-s1 kernel: libata version 3.00 loaded. Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas version 31.00.00.00 loaded Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7d Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (263565236 kB) Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.4: New USB device found, idVendor=1604, idProduct=10c0 Nov 04 15:38:49 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: firmware version: 20.26.1040 Nov 04 15:38:49 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16 GT/s x16 link) Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4a:7e Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Nov 04 15:38:49 fir-md1-s1 kernel: tg3 0000:81:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] Nov 04 15:38:49 fir-md1-s1 kernel: usb 3-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 120 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 121 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 122 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 123 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 124 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 125 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 126 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 127 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 128 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 129 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 130 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 131 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 132 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 133 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 134 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 135 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 136 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 137 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 138 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 139 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 140 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 141 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 142 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 143 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 144 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 145 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 146 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 147 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 148 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 149 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 150 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 151 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 152 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 153 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 154 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 155 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 156 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 157 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 158 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 159 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 160 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 161 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 162 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 163 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 164 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 165 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 166 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: irq 167 for MSI/MSI-X Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 120 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 121 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 122 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 123 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 124 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 125 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 126 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 127 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 128 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 129 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 130 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 131 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 132 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 133 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 134 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 135 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 136 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 137 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 138 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 139 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 140 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 141 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 142 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 143 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 144 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 145 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 146 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 147 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 148 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 149 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 150 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 151 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 152 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 153 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 154 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 155 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 156 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 157 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 158 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 159 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 160 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 161 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 162 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 163 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 164 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 165 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 166 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 167 Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas_cm0: iomem(0x00000000ac000000), mapped(0xffffb85e1a200000), size(1048576) Nov 04 15:38:49 fir-md1-s1 kernel: mpt3sas_cm0: ioport(0x0000000000008000), size(256) Nov 04 15:38:50 fir-md1-s1 kernel: hub 3-1.4:1.0: USB hub found Nov 04 15:38:50 fir-md1-s1 kernel: hub 3-1.4:1.0: 4 ports detected Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: version 3.0 Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 169 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 170 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 171 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 172 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 173 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 174 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 175 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 176 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 177 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 178 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 179 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 180 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 181 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 182 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 183 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: irq 184 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode Nov 04 15:38:50 fir-md1-s1 kernel: ahci 0000:86:00.2: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part Nov 04 15:38:50 fir-md1-s1 kernel: scsi host2: ahci Nov 04 15:38:50 fir-md1-s1 kernel: ata1: SATA max UDMA/133 abar m4096@0xc0a02000 port 0xc0a02100 irq 169 Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: IOC Number : 0 Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Init cmd return status SUCCESS for SCSI host 0 Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware type : Legacy(64 VD) firmware Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: controller type : iMR(0MB) Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Online Controller Reset(OCR) : Enabled Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Secure JBOD support : No Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: NVMe passthru support : No Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: INIT adapter done Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: unevenspan support : yes Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: firmware crash dump : no Nov 04 15:38:50 fir-md1-s1 kernel: megaraid_sas 0000:c1:00.0: jbod sync map : no Nov 04 15:38:50 fir-md1-s1 kernel: scsi host0: Avago SAS based MegaRAID driver Nov 04 15:38:50 fir-md1-s1 kernel: scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.30 PQ: 0 ANSI: 5 Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 185 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 186 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 187 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 188 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 189 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 190 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 191 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 192 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 193 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 194 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 195 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 196 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 197 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 198 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 199 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 200 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 201 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 202 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 203 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 204 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 205 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 206 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 207 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 208 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 209 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 210 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 211 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 212 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 213 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 214 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 215 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 216 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 217 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 218 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 219 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 220 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 221 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 222 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 223 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 224 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 225 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 226 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 227 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 228 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 229 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 230 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 231 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 232 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: irq 233 for MSI/MSI-X Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: mlx5_pcie_event:303:(pid 6): PCIe slot advertised sufficient power (27W). Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: mlx5_fw_tracer_start:776:(pid 297): FWTracer: Ownership granted and active Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: Allocated physical memory: size(38831 kB) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: Current Controller Queue Depth(7564), Max Controller Queue Depth(7680) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128) Nov 04 15:38:50 fir-md1-s1 kernel: ata1: SATA link down (SStatus 0 SControl 300) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: FW Package Version(12.00.00.00) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: SAS3616: FWVersion(12.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: Protocol=(Initiator,Target,NVMe), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ) Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas 0000:84:00.0: Enabled Extended Tags as Controller Supports Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 Nov 04 15:38:50 fir-md1-s1 kernel: scsi host1: Fusion MPT SAS Host Nov 04 15:38:50 fir-md1-s1 kernel: mpt3sas_cm0: sending port enable !! Nov 04 15:38:50 fir-md1-s1 kernel: mlx5_ib: Mellanox Connect-IB Infiniband driver v4.7-1.0.0 Nov 04 15:38:50 fir-md1-s1 kernel: random: crng init done Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: hba_port entry: ffffa12e34a24a80, port: 255 is added to hba_port list Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00db90c00), phys(21) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0011), sas_address(0x300705b00db90c00), phy(16) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0011), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0011), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: Enclosure LSI VirtualSES 03 PQ: 0 ANSI: 7 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: set ignore_delay_remove for handle(0x0011) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: SES: handle(0x0011), sas_addr(0x300705b00db90c00), phy(16), device_name(0x300705b00db90c00) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: enclosure logical id(0x300605b00d110c00), slot(16) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: enclosure level(0x0000), connector name( C3 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: serial_number(300605B00D110C00) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:0:0: qdepth(1), tagged(0), simple(0), ordered(0), scsi_level(8), cmd_que(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: log_info(0x31200206): originator(PL), code(0x20), sub_code(0x0206) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(1) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984db2fa920), phy(8) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: enclosure logical id(0x300605b00d110c00), slot(5) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: enclosure level(0x0000), connector name( C1 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: serial_number(021815000354 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: enclosure logical id(0x300605b00d110c00), slot(5) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: enclosure level(0x0000), connector name( C1 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: serial_number(021815000354 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:1: Mode parameters changed Nov 04 15:38:52 fir-md1-s1 kernel: sd 0:2:0:0: [sda] 233308160 512-byte logical blocks: (119 GB/111 GiB) Nov 04 15:38:52 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Write Protect is off Nov 04 15:38:52 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 Nov 04 15:38:52 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: enclosure logical id(0x300605b00d110c00), slot(5) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: enclosure level(0x0000), connector name( C1 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: serial_number(021815000354 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:2: Mode parameters changed Nov 04 15:38:52 fir-md1-s1 kernel: sda: sda1 sda2 sda3 Nov 04 15:38:52 fir-md1-s1 kernel: sd 0:2:0:0: [sda] Attached SCSI disk Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: SSP: handle(0x0017), sas_addr(0x500a0984db2fa920), phy(8), device_name(0x500a0984db2fa920) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: enclosure logical id(0x300605b00d110c00), slot(5) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: enclosure level(0x0000), connector name( C1 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: serial_number(021815000354 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(1) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa1fa20), phy(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: enclosure logical id(0x300605b00d110c00), slot(13) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: enclosure level(0x0000), connector name( C3 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: serial_number(021825001369 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: enclosure logical id(0x300605b00d110c00), slot(13) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: enclosure level(0x0000), connector name( C3 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: serial_number(021825001369 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:1: Mode parameters changed Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: SSP: handle(0x0018), sas_addr(0x500a0984dfa1fa20), phy(0), device_name(0x500a0984dfa1fa20) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: enclosure logical id(0x300605b00d110c00), slot(13) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: enclosure level(0x0000), connector name( C3 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: serial_number(021825001369 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:2:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa20c14), phy(4) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(1) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa20c14), phy(4) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) Nov 04 15:38:52 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:3:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:3:0: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:3:0: enclosure logical id(0x300605b00d110c00), slot(9) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:3:0: enclosure level(0x0000), connector name( C2 ) Nov 04 15:38:52 fir-md1-s1 kernel: scsi 1:0:3:0: serial_number(021825001558 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: enclosure logical id(0x300605b00d110c00), slot(9) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: enclosure level(0x0000), connector name( C2 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: serial_number(021825001558 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: SSP: handle(0x0019), sas_addr(0x500a0984dfa20c14), phy(4), device_name(0x500a0984dfa20c14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: enclosure logical id(0x300605b00d110c00), slot(9) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: enclosure level(0x0000), connector name( C2 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: serial_number(021825001558 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:3:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984da0f9b14), phy(12) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(1) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984da0f9b14), phy(12) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) Nov 04 15:38:53 fir-md1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: enclosure logical id(0x300605b00d110c00), slot(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: enclosure level(0x0000), connector name( C0 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: serial_number(021812047179 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: enclosure logical id(0x300605b00d110c00), slot(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: enclosure level(0x0000), connector name( C0 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: serial_number(021812047179 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: enclosure logical id(0x300605b00d110c00), slot(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: enclosure level(0x0000), connector name( C0 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: serial_number(021812047179 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: SSP: handle(0x001a), sas_addr(0x500a0984da0f9b14), phy(12), device_name(0x500a0984da0f9b14) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: enclosure logical id(0x300605b00d110c00), slot(1) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: enclosure level(0x0000), connector name( C0 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: serial_number(021812047179 ) Nov 04 15:38:53 fir-md1-s1 kernel: scsi 1:0:4:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Nov 04 15:38:58 fir-md1-s1 kernel: mpt3sas_cm0: port enable: SUCCESS Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (owned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] 926167040 512-byte logical blocks: (474 GB/441 GiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] 4096-byte physical blocks Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (owned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:2:0: rdac: LUN 0 (IOSHIP) (owned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:0: [sde] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:2:1: rdac: LUN 1 (IOSHIP) (unowned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:3:0: rdac: LUN 0 (IOSHIP) (unowned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:3:1: rdac: LUN 1 (IOSHIP) (owned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:4:0: rdac: LUN 0 (IOSHIP) (unowned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] 926167040 512-byte logical blocks: (474 GB/441 GiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] 4096-byte physical blocks Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:4:1: rdac: LUN 1 (IOSHIP) (owned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:1: [sdj] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:0: [sde] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: scsi 1:0:4:2: rdac: LUN 2 (IOSHIP) (unowned) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:1: [sdj] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:1: [sdj] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:2: [sdk] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:2: [sdd] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:2: [sdk] Write Protect is off Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:2: [sdk] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:2: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:1: [sdh] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:2:1: [sdf] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:3:0: [sdg] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:0: [sdi] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:2: [sdk] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:4:1: [sdj] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:1: [sdc] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Mode Sense: 83 00 10 08 Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA Nov 04 15:38:58 fir-md1-s1 kernel: sd 1:0:1:0: [sdb] Attached SCSI disk Nov 04 15:38:58 fir-md1-s1 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) Nov 04 15:38:58 fir-md1-s1 systemd-journald[358]: Received SIGTERM from PID 1 (systemd). Nov 04 15:38:58 fir-md1-s1 kernel: SELinux: Disabled at runtime. Nov 04 15:38:58 fir-md1-s1 kernel: SELinux: Unregistering netfilter hooks Nov 04 15:38:58 fir-md1-s1 kernel: type=1404 audit(1572910738.741:2): selinux=0 auid=4294967295 ses=4294967295 Nov 04 15:38:58 fir-md1-s1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Nov 04 15:38:58 fir-md1-s1 systemd[1]: Inserted module 'ip_tables' Nov 04 15:38:58 fir-md1-s1 kernel: EXT4-fs (sda2): re-mounted. Opts: (null) Nov 04 15:38:58 fir-md1-s1 kernel: ACPI Error: No handler for Region [SYSI] (ffffa10ee9e98a68) [IPMI] (20130517/evregion-162) Nov 04 15:38:58 fir-md1-s1 kernel: ipmi message handler version 39.2 Nov 04 15:38:58 fir-md1-s1 kernel: ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) Nov 04 15:38:59 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffffa10ee9e795a0), AE_NOT_EXIST (20130517/psparse-536) Nov 04 15:38:59 fir-md1-s1 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffffa10ee9e79500), AE_NOT_EXIST (20130517/psparse-536) Nov 04 15:38:59 fir-md1-s1 kernel: ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: 3 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 235 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: irq 236 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input2 Nov 04 15:38:59 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 Nov 04 15:38:59 fir-md1-s1 kernel: piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 3 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 4 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 0 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 1 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: Queue 2 gets LSB 6 Nov 04 15:38:59 fir-md1-s1 kernel: cryptd: max_cpu_qlen set to 1000 Nov 04 15:38:59 fir-md1-s1 kernel: ipmi device interface Nov 04 15:38:59 fir-md1-s1 kernel: sd 0:2:0:0: Attached scsi generic sg0 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:0:0: Attached scsi generic sg1 type 13 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:0: Attached scsi generic sg2 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:1: Attached scsi generic sg3 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:2: Attached scsi generic sg4 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:1:31: Attached scsi generic sg5 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:2:0: Attached scsi generic sg6 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:2:1: Attached scsi generic sg7 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:2:31: Attached scsi generic sg8 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:3:0: Attached scsi generic sg9 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:02:00.2: enabled Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:3:1: Attached scsi generic sg10 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:3:31: Attached scsi generic sg11 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:0: Attached scsi generic sg12 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: 5 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:1: Attached scsi generic sg13 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: irq 238 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 0 gets LSB 1 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 1 gets LSB 2 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 2 gets LSB 3 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 3 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: Queue 4 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:2: Attached scsi generic sg14 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:4:31: Attached scsi generic sg15 type 0 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:03:00.1: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: 3 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 240 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: irq 241 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 3 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 4 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 0 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 1 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: Queue 2 gets LSB 6 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:41:00.2: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: 5 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: irq 243 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 0 gets LSB 1 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 1 gets LSB 2 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 2 gets LSB 3 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 3 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: Queue 4 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:42:00.1: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: 3 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 245 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: irq 246 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 3 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 4 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 0 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 1 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: Queue 2 gets LSB 6 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:85:00.2: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: 5 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: irq 248 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 0 gets LSB 1 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 1 gets LSB 2 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 2 gets LSB 3 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 3 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: Queue 4 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:86:00.1: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: 3 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 250 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: irq 251 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 3 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 4 can access 4 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 0 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 1 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: Queue 2 gets LSB 6 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c2:00.2: enabled Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: 5 command queues available Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: irq 253 for MSI/MSI-X Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 can access 7 LSB regions Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 0 gets LSB 1 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 1 gets LSB 2 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 2 gets LSB 3 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 3 gets LSB 4 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: Queue 4 gets LSB 5 Nov 04 15:38:59 fir-md1-s1 kernel: ccp 0000:c3:00.1: enabled Nov 04 15:38:59 fir-md1-s1 kernel: IPMI System Interface driver Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: ipmi_platform: probing via ACPI Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: [io 0x0ca8] regsize 1 spacing 4 irq 10 Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si: Adding ACPI-specified kcs state machine Nov 04 15:38:59 fir-md1-s1 kernel: device-mapper: uevent: version 1.0.3 Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 Nov 04 15:38:59 fir-md1-s1 kernel: device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: Using irq 10 Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) Nov 04 15:38:59 fir-md1-s1 kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:0: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:1: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:1:2: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:1:31: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:2:0: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:2:1: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:2:31: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:3:0: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:3:1: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:3:31: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:0: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:1: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: sd 1:0:4:2: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: scsi 1:0:4:31: Embedded Enclosure Device Nov 04 15:38:59 fir-md1-s1 kernel: ses 1:0:0:0: Attached Enclosure device Nov 04 15:38:59 fir-md1-s1 kernel: dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) Nov 04 15:38:59 fir-md1-s1 kernel: AVX2 version of gcm_enc/dec engaged. Nov 04 15:38:59 fir-md1-s1 kernel: AES CTR mode by8 optimization enabled Nov 04 15:38:59 fir-md1-s1 kernel: alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) Nov 04 15:38:59 fir-md1-s1 kernel: alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) Nov 04 15:38:59 fir-md1-s1 kernel: kvm: Nested Paging enabled Nov 04 15:38:59 fir-md1-s1 kernel: MCE: In-kernel MCE decoding enabled. Nov 04 15:38:59 fir-md1-s1 kernel: AMD64 EDAC driver v3.4.0 Nov 04 15:38:59 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Nov 04 15:38:59 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 0). Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC0: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:18.3 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 1). Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC1: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:19.3 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 2). Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC2: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1a.3 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: DRAM ECC enabled. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: F17h detected (node 3). Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC0 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC: UMC1 chip selects: Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 0: 0MB 1: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 2: 16383MB 3: 16383MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 4: 0MB 5: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MC: 6: 0MB 7: 0MB Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: using x8 syndromes. Nov 04 15:39:00 fir-md1-s1 kernel: EDAC amd64: MCT channel count: 2 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC MC3: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1b.3 Nov 04 15:39:00 fir-md1-s1 kernel: EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.0' (POLLED) Nov 04 15:39:29 fir-md1-s1 kernel: device-mapper: multipath round-robin: version 1.2.0 loaded Nov 04 15:39:32 fir-md1-s1 kernel: Adding 4194300k swap on /dev/sda3. Priority:-2 extents:1 across:4194300k FS Nov 04 15:39:32 fir-md1-s1 kernel: FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. Nov 04 15:39:32 fir-md1-s1 kernel: type=1305 audit(1572910772.804:3): audit_pid=11212 old=0 auid=4294967295 ses=4294967295 res=1 Nov 04 15:39:32 fir-md1-s1 kernel: RPC: Registered named UNIX socket transport module. Nov 04 15:39:32 fir-md1-s1 kernel: RPC: Registered udp transport module. Nov 04 15:39:32 fir-md1-s1 kernel: RPC: Registered tcp transport module. Nov 04 15:39:32 fir-md1-s1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Nov 04 15:39:33 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: slow_pci_heuristic:5575:(pid 11498): Max link speed = 100000, PCI BW = 126016 Nov 04 15:39:33 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) Nov 04 15:39:33 fir-md1-s1 kernel: mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) Nov 04 15:39:33 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 254 for MSI/MSI-X Nov 04 15:39:33 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 255 for MSI/MSI-X Nov 04 15:39:33 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 256 for MSI/MSI-X Nov 04 15:39:33 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 257 for MSI/MSI-X Nov 04 15:39:33 fir-md1-s1 kernel: tg3 0000:81:00.0: irq 258 for MSI/MSI-X Nov 04 15:39:34 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready Nov 04 15:39:37 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Link is up at 1000 Mbps, full duplex Nov 04 15:39:37 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: Flow control is on for TX and on for RX Nov 04 15:39:37 fir-md1-s1 kernel: tg3 0000:81:00.0 em1: EEE is enabled Nov 04 15:39:37 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready Nov 04 15:39:38 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready Nov 04 15:39:38 fir-md1-s1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Nov 04 15:39:42 fir-md1-s1 kernel: FS-Cache: Loaded Nov 04 15:39:42 fir-md1-s1 kernel: FS-Cache: Netfs 'nfs' registered for caching Nov 04 15:39:42 fir-md1-s1 kernel: Key type dns_resolver registered Nov 04 15:39:42 fir-md1-s1 kernel: NFS: Registering the id_resolver key type Nov 04 15:39:42 fir-md1-s1 kernel: Key type id_resolver registered Nov 04 15:39:42 fir-md1-s1 kernel: Key type id_legacy registered Nov 04 15:39:51 fir-md1-s1 kernel: LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 Nov 04 15:39:51 fir-md1-s1 kernel: alg: No test for adler32 (adler32-zlib) Nov 04 15:39:52 fir-md1-s1 kernel: Lustre: Lustre: Build Version: 2.12.3_2_gb033996 Nov 04 15:39:52 fir-md1-s1 kernel: LNet: 18927:0:(config.c:1627:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down Nov 04 15:39:52 fir-md1-s1 kernel: LNet: Using FastReg for registration Nov 04 15:39:52 fir-md1-s1 kernel: LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] Nov 04 15:40:27 fir-md1-s1 kernel: LNetError: 19765:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 Nov 04 15:40:30 fir-md1-s1 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Nov 04 15:41:12 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 15:41:15 fir-md1-s1 kernel: LDISKFS-fs (dm-1): recovery complete Nov 04 15:41:15 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 15:41:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.110.61@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:16 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.106.15@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:16 fir-md1-s1 kernel: LustreError: Skipped 39 previous similar messages Nov 04 15:41:17 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.102.60@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:17 fir-md1-s1 kernel: LustreError: Skipped 31 previous similar messages Nov 04 15:41:19 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.104.61@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:19 fir-md1-s1 kernel: LustreError: Skipped 58 previous similar messages Nov 04 15:41:23 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.106.14@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:23 fir-md1-s1 kernel: LustreError: Skipped 143 previous similar messages Nov 04 15:41:24 fir-md1-s1 kernel: LustreError: 21585:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.104.41@o2ib4 arrived at 1572910884 with bad export cookie 1371060822959543134 Nov 04 15:41:31 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.115.6@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:31 fir-md1-s1 kernel: LustreError: Skipped 261 previous similar messages Nov 04 15:41:33 fir-md1-s1 kernel: LustreError: 21537:0:(mgc_request.c:249:do_config_log_add()) MGC10.0.10.51@o2ib7: failed processing log, type 1: rc = -5 Nov 04 15:41:36 fir-md1-s1 kernel: LustreError: 21579:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.110.47@o2ib4 arrived at 1572910896 with bad export cookie 1371060835260497905 Nov 04 15:41:42 fir-md1-s1 kernel: LNetError: 19765:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.211@o2ib7 added to recovery queue. Health = 900 Nov 04 15:41:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.101.68@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:41:47 fir-md1-s1 kernel: LustreError: Skipped 362 previous similar messages Nov 04 15:42:04 fir-md1-s1 kernel: LustreError: 21537:0:(mgc_request.c:249:do_config_log_add()) MGC10.0.10.51@o2ib7: failed processing log, type 4: rc = -110 Nov 04 15:42:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.107.72@o2ib4 (not set up) Nov 04 15:42:20 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.9.110.22@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:42:20 fir-md1-s1 kernel: LustreError: Skipped 133 previous similar messages Nov 04 15:43:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery not enabled, recovery window 300-900 Nov 04 15:43:10 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 15:43:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 5:00, or until 1256 clients reconnect Nov 04 15:43:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.105.52@o2ib4) Nov 04 15:43:11 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Nov 04 15:43:12 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.115.6@o2ib4) Nov 04 15:43:12 fir-md1-s1 kernel: Lustre: Skipped 26 previous similar messages Nov 04 15:43:13 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.26.34@o2ib6) Nov 04 15:43:13 fir-md1-s1 kernel: Lustre: Skipped 32 previous similar messages Nov 04 15:43:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.109.1@o2ib4) Nov 04 15:43:15 fir-md1-s1 kernel: Lustre: Skipped 96 previous similar messages Nov 04 15:43:15 fir-md1-s1 kernel: LustreError: 22068:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11e36031b00 x1649241182920544/t0(0) o601->fir-MDT0000-lwp-OST0056_UUID@10.0.10.115@o2ib7:620/0 lens 336/0 e 0 to 0 dl 1572911750 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:15 fir-md1-s1 kernel: LustreError: 22068:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 2 previous similar messages Nov 04 15:43:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 3efe8bf8-66ec-7ea4-2df5-b70424ee9926 (at 10.9.108.26@o2ib4), waiting for 1256 known clients (195 recovered, 7 in progress, and 0 evicted) to recover in 4:55 Nov 04 15:43:17 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 16e3b1d1-a45e-70b5-4903-5b92ac5832f6 (at 10.8.31.9@o2ib6), waiting for 1256 known clients (266 recovered, 9 in progress, and 0 evicted) to recover in 4:53 Nov 04 15:43:18 fir-md1-s1 kernel: LustreError: 22068:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11e29bf0900 x1649241182952768/t0(0) o601->fir-MDT0000-lwp-OST005c_UUID@10.0.10.115@o2ib7:623/0 lens 336/0 e 0 to 0 dl 1572911753 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:18 fir-md1-s1 kernel: LustreError: 22068:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 12 previous similar messages Nov 04 15:43:19 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.24.6@o2ib6) Nov 04 15:43:19 fir-md1-s1 kernel: Lustre: Skipped 190 previous similar messages Nov 04 15:43:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client ec6ea41b-d9f4-cfae-8341-af1fd0a7d21a (at 10.9.117.10@o2ib4), waiting for 1256 known clients (402 recovered, 8 in progress, and 0 evicted) to recover in 4:50 Nov 04 15:43:20 fir-md1-s1 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Nov 04 15:43:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client ba7da825-00cf-9ed7-710f-c9fe6f8d412f (at 10.9.103.68@o2ib4), waiting for 1256 known clients (516 recovered, 12 in progress, and 0 evicted) to recover in 4:48 Nov 04 15:43:22 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 15:43:23 fir-md1-s1 kernel: LustreError: 22068:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11e3f579f80 x1649241182953472/t0(0) o601->fir-MDT0000-lwp-OST005c_UUID@10.0.10.115@o2ib7:628/0 lens 336/0 e 0 to 0 dl 1572911758 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:25 fir-md1-s1 kernel: LustreError: 21608:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10e1c4f1200 x1649241031700768/t0(0) o601->fir-MDT0000-lwp-OST000b_UUID@10.0.10.102@o2ib7:630/0 lens 336/0 e 0 to 0 dl 1572911760 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:25 fir-md1-s1 kernel: LustreError: 21608:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 56 previous similar messages Nov 04 15:43:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.0.10.106@o2ib7) Nov 04 15:43:27 fir-md1-s1 kernel: Lustre: Skipped 429 previous similar messages Nov 04 15:43:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client c9059d0b-84af-6560-d5e6-5d64867155f8 (at 10.9.108.27@o2ib4), waiting for 1256 known clients (740 recovered, 28 in progress, and 0 evicted) to recover in 4:42 Nov 04 15:43:27 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Nov 04 15:43:29 fir-md1-s1 kernel: LustreError: 22165:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa13e260d4050 x1649241036436768/t0(0) o601->fir-MDT0000-lwp-OST0016_UUID@10.0.10.103@o2ib7:634/0 lens 336/0 e 0 to 0 dl 1572911764 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:29 fir-md1-s1 kernel: LustreError: 22165:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 46 previous similar messages Nov 04 15:43:37 fir-md1-s1 kernel: LustreError: 22134:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa12e017be050 x1649241034295920/t0(0) o601->fir-MDT0000-lwp-OST0002_UUID@10.0.10.101@o2ib7:642/0 lens 336/0 e 0 to 0 dl 1572911772 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:37 fir-md1-s1 kernel: LustreError: 22134:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 61 previous similar messages Nov 04 15:43:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 3efe8bf8-66ec-7ea4-2df5-b70424ee9926 (at 10.9.108.26@o2ib4), waiting for 1256 known clients (1156 recovered, 34 in progress, and 0 evicted) to recover in 4:29 Nov 04 15:43:40 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Nov 04 15:43:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.103.34@o2ib4) Nov 04 15:43:43 fir-md1-s1 kernel: Lustre: Skipped 489 previous similar messages Nov 04 15:43:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:43:45 fir-md1-s1 kernel: LustreError: Skipped 3 previous similar messages Nov 04 15:43:53 fir-md1-s1 kernel: LustreError: 22069:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11e2bd7bf00 x1649241033371168/t0(0) o601->fir-MDT0000-lwp-OST000f_UUID@10.0.10.104@o2ib7:658/0 lens 336/0 e 0 to 0 dl 1572911788 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:43:53 fir-md1-s1 kernel: LustreError: 22069:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 41 previous similar messages Nov 04 15:43:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 6464c68a-677b-2ae8-74d8-45f2f01c0362 (at 10.9.117.15@o2ib4), waiting for 1256 known clients (1215 recovered, 36 in progress, and 0 evicted) to recover in 4:13 Nov 04 15:43:56 fir-md1-s1 kernel: Lustre: Skipped 15 previous similar messages Nov 04 15:44:02 fir-md1-s1 kernel: LDISKFS-fs (dm-0): recovery complete Nov 04 15:44:02 fir-md1-s1 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Nov 04 15:44:15 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.8.9.2@o2ib6) Nov 04 15:44:15 fir-md1-s1 kernel: Lustre: Skipped 315 previous similar messages Nov 04 15:44:25 fir-md1-s1 kernel: LustreError: 22139:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10e340b6780 x1649241031768976/t0(0) o601->fir-MDT0000-lwp-OST0009_UUID@10.0.10.102@o2ib7:690/0 lens 336/0 e 0 to 0 dl 1572911820 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:44:25 fir-md1-s1 kernel: LustreError: 22139:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 145 previous similar messages Nov 04 15:44:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 3efe8bf8-66ec-7ea4-2df5-b70424ee9926 (at 10.9.108.26@o2ib4), waiting for 1256 known clients (1218 recovered, 36 in progress, and 0 evicted) to recover in 3:39 Nov 04 15:44:30 fir-md1-s1 kernel: Lustre: Skipped 21 previous similar messages Nov 04 15:45:20 fir-md1-s1 kernel: Lustre: MGS: Connection restored to 81c0e68e-8d72-8ac3-64ca-cbef7f483d12 (at 10.9.101.20@o2ib4) Nov 04 15:45:20 fir-md1-s1 kernel: Lustre: Skipped 983 previous similar messages Nov 04 15:45:29 fir-md1-s1 kernel: LustreError: 22148:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa12dd7c49b00 x1649241034364928/t0(0) o601->fir-MDT0000-lwp-OST0002_UUID@10.0.10.101@o2ib7:754/0 lens 336/0 e 0 to 0 dl 1572911884 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:45:29 fir-md1-s1 kernel: LustreError: 22148:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 359 previous similar messages Nov 04 15:45:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 65f17af2-0f81-d0c0-07ec-f60881376e81 (at 10.8.15.8@o2ib6), waiting for 1256 known clients (1218 recovered, 36 in progress, and 0 evicted) to recover in 2:34 Nov 04 15:45:36 fir-md1-s1 kernel: Lustre: Skipped 56 previous similar messages Nov 04 15:46:15 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:46:15 fir-md1-s1 kernel: LustreError: Skipped 5 previous similar messages Nov 04 15:47:38 fir-md1-s1 kernel: LustreError: 22151:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10e340b5e80 x1649241031780032/t0(0) o601->fir-MDT0000-lwp-OST0003_UUID@10.0.10.102@o2ib7:128/0 lens 336/0 e 0 to 0 dl 1572912013 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 15:47:38 fir-md1-s1 kernel: LustreError: 22151:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 734 previous similar messages Nov 04 15:47:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client aabcf78d-5204-1b94-9fe6-5190b000256d (at 10.9.101.24@o2ib4), waiting for 1256 known clients (1218 recovered, 36 in progress, and 0 evicted) to recover in 0:24 Nov 04 15:47:46 fir-md1-s1 kernel: Lustre: Skipped 117 previous similar messages Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: recovery is timed out, evict stale exports Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 2 stale clients Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: 21998:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 16a6e06f-fa65-03f9-cf57-33b94edf5776 (at 10.9.101.31@o2ib4) reconnected, waiting for 1256 clients in recovery for 9:59 Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.31@o2ib4) Nov 04 15:48:10 fir-md1-s1 kernel: Lustre: Skipped 49 previous similar messages Nov 04 15:48:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 5:00, of 1256 clients 1254 recovered and 2 were evicted. Nov 04 15:50:43 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.9.117.28@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffffa10e11a18fc0/0x675682d7aa317ea lrc: 3/0,0 mode: PW/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 823 type: IBT flags: 0x60200400000020 nid: 10.9.117.28@o2ib4 remote: 0x6517eb693bd99f26 expref: 51 pid: 22754 timeout: 720 lvb_type: 0 Nov 04 15:50:43 fir-md1-s1 kernel: LustreError: 22763:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa13e26299000 ns: mdt-fir-MDT0000_UUID lock: ffffa13dcbc9bf00/0x675682d7aa3183e lrc: 3/0,0 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 821 type: IBT flags: 0x54a01400000020 nid: 10.9.117.28@o2ib4 remote: 0x6517eb693bd99ec4 expref: 49 pid: 22763 timeout: 0 lvb_type: 0 Nov 04 15:50:44 fir-md1-s1 kernel: INFO: task mdt03_000:21605 blocked for more than 120 seconds. Nov 04 15:50:44 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:44 fir-md1-s1 kernel: mdt03_000 D ffffa12e2f8a4100 0 21605 2 0x00000080 Nov 04 15:50:44 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:44 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:44 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? osd_object_alloc+0x360/0x360 [osd_ldiskfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:44 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ldlm_lock_enqueue+0x3ff/0xa20 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:44 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:44 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:44 fir-md1-s1 kernel: INFO: task mdt03_004:21659 blocked for more than 120 seconds. Nov 04 15:50:44 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:44 fir-md1-s1 kernel: mdt03_004 D ffffa12e158e0000 0 21659 2 0x00000080 Nov 04 15:50:44 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:44 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:44 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? osd_object_alloc+0x360/0x360 [osd_ldiskfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:44 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ldlm_lock_enqueue+0x3ff/0xa20 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:44 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:44 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? queued_spin_lock_slowpath+0xb/0xf Nov 04 15:50:44 fir-md1-s1 kernel: [] ? _raw_spin_lock+0x20/0x30 Nov 04 15:50:44 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:44 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:44 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: INFO: task mdt00_012:22587 blocked for more than 120 seconds. Nov 04 15:50:45 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:45 fir-md1-s1 kernel: mdt00_012 D ffffa10e39b6c100 0 22587 2 0x00000080 Nov 04 15:50:45 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:45 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:45 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:45 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:45 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x48a/0x780 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ldlm_lock_enqueue+0x3ff/0xa20 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: INFO: task mdt03_014:22595 blocked for more than 120 seconds. Nov 04 15:50:45 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:45 fir-md1-s1 kernel: mdt03_014 D ffffa12e35f8a080 0 22595 2 0x00000080 Nov 04 15:50:45 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:45 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:45 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? osd_object_alloc+0x360/0x360 [osd_ldiskfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:45 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ___slab_alloc+0x209/0x4f0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: INFO: task mdt03_016:22601 blocked for more than 120 seconds. Nov 04 15:50:45 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:45 fir-md1-s1 kernel: mdt03_016 D ffffa12ddfc50000 0 22601 2 0x00000080 Nov 04 15:50:45 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:45 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:45 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:45 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:45 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? list_del+0xd/0x30 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? get_partial_node.isra.48+0xfe/0x230 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:45 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:45 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:45 fir-md1-s1 kernel: INFO: task mdt03_021:22616 blocked for more than 120 seconds. Nov 04 15:50:45 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:45 fir-md1-s1 kernel: mdt03_021 D ffffa12ddfcfe180 0 22616 2 0x00000080 Nov 04 15:50:45 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:45 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:45 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:45 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:45 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:45 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ___slab_alloc+0x209/0x4f0 Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:45 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? kmalloc_order_trace+0x2e/0xa0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: INFO: task mdt02_014:22623 blocked for more than 120 seconds. Nov 04 15:50:46 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:46 fir-md1-s1 kernel: mdt02_014 D ffffa12ddfc1d140 0 22623 2 0x00000080 Nov 04 15:50:46 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:46 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:46 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:46 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:46 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ldlm_cli_cancel_local+0x29e/0x3f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ldlm_lock_enqueue+0x3ff/0xa20 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? kmalloc_order_trace+0x2e/0xa0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: INFO: task mdt01_022:22642 blocked for more than 120 seconds. Nov 04 15:50:46 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:46 fir-md1-s1 kernel: mdt01_022 D ffffa12ddfdd8000 0 22642 2 0x00000080 Nov 04 15:50:46 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:46 fir-md1-s1 kernel: [] ? zone_statistics+0x88/0xa0 Nov 04 15:50:46 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:46 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:46 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:46 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ldlm_cli_cancel_local+0x29e/0x3f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? kmalloc_order_trace+0x2e/0xa0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:46 fir-md1-s1 kernel: INFO: task mdt03_030:22657 blocked for more than 120 seconds. Nov 04 15:50:46 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:46 fir-md1-s1 kernel: mdt03_030 D ffffa12ddfd3a080 0 22657 2 0x00000080 Nov 04 15:50:46 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:46 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:46 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:46 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:46 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? ___slab_alloc+0x209/0x4f0 Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:46 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:47 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:47 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:47 fir-md1-s1 kernel: INFO: task mdt03_033:22665 blocked for more than 120 seconds. Nov 04 15:50:47 fir-md1-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 04 15:50:47 fir-md1-s1 kernel: mdt03_033 D ffffa12ddfd7d140 0 22665 2 0x00000080 Nov 04 15:50:47 fir-md1-s1 kernel: Call Trace: Nov 04 15:50:47 fir-md1-s1 kernel: [] schedule+0x29/0x70 Nov 04 15:50:47 fir-md1-s1 kernel: [] rwsem_down_write_failed+0x225/0x3a0 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? osd_object_alloc+0x360/0x360 [osd_ldiskfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 15:50:47 fir-md1-s1 kernel: [] down_write+0x2d/0x3d Nov 04 15:50:47 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ___slab_alloc+0x209/0x4f0 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ldlm_pool_recalc+0x12e/0x1f0 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lod_prepare_avoidance+0x440/0x780 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lod_alloc_comp_entries+0xce/0x520 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 15:50:47 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 15:50:47 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? mdt_intent_open+0x3a0/0x3a0 [mdt] Nov 04 15:50:47 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] Nov 04 15:50:47 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? __wake_up+0x44/0x50 Nov 04 15:50:47 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] ? __schedule+0x42a/0x860 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Nov 04 15:50:47 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:50:47 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 15:50:47 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Nov 04 15:52:16 fir-md1-s1 kernel: LNetError: 19748:0:(o2iblnd_cb.c:3350:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Nov 04 15:52:16 fir-md1-s1 kernel: LNetError: 19748:0:(o2iblnd_cb.c:3425:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 6, oc: 0, rc: 8 Nov 04 15:52:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to f934e85e-83d5-f0b4-d160-68208a74e69c (at 10.9.112.2@o2ib4) Nov 04 15:52:28 fir-md1-s1 kernel: Lustre: Skipped 12 previous similar messages Nov 04 15:52:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 151a14fd-eede-c30d-ad14-4e1815cd58c9 (at 10.9.105.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13e27fc7c00, cur 1572911574 expire 1572911424 last 1572911347 Nov 04 15:53:11 fir-md1-s1 kernel: LustreError: 22772:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572911290, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa13dcbc9c380/0x675682d7aa31972 lrc: 3/0,1 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 814 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22772 timeout: 0 lvb_type: 0 Nov 04 15:53:11 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572911590.22718 Nov 04 15:53:11 fir-md1-s1 kernel: LustreError: 22772:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 136 previous similar messages Nov 04 15:53:14 fir-md1-s1 kernel: LustreError: 22918:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572911293, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa10e0c436c00/0x675682d7aa3569d lrc: 3/0,1 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 820 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22918 timeout: 0 lvb_type: 0 Nov 04 15:53:14 fir-md1-s1 kernel: LustreError: 22706:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572911293, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa10e0b788000/0x675682d7aa35688 lrc: 3/0,1 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 820 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22706 timeout: 0 lvb_type: 0 Nov 04 15:53:14 fir-md1-s1 kernel: LustreError: 22706:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 160 previous similar messages Nov 04 15:53:14 fir-md1-s1 kernel: LustreError: 22918:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 40 previous similar messages Nov 04 15:54:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 26b80e34-2fe6-2525-1dab-43a1c160bf00 (at 10.0.10.3@o2ib7) in 220 seconds. I think it's dead, and I am evicting it. exp ffffa13e33824c00, cur 1572911650 expire 1572911500 last 1572911430 Nov 04 15:54:10 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 15:54:17 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client a7dd94b8-06b8-2fc8-b59f-8255142818f4 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6ad7800, cur 1572911657 expire 1572911507 last 1572911430 Nov 04 15:55:43 fir-md1-s1 kernel: LustreError: 22764:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572911443, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa10e1537bcc0/0x675682d7aa460c0 lrc: 3/0,1 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 814 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 22764 timeout: 0 lvb_type: 0 Nov 04 15:55:43 fir-md1-s1 kernel: LustreError: 22764:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message Nov 04 15:58:05 fir-md1-s1 kernel: Lustre: 23074:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa12dd7c4de80 x1649307047843984/t0(0) o101->9484b05f-bbba-6023-e820-67c71f8b4c9f@10.9.117.38@o2ib4:5/0 lens 480/0 e 2 to 0 dl 1572911890 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 04 15:58:05 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa10e3524da00 x1649288843096576/t0(0) o101->20fc9a22-3c78-afea-56ad-c9c9b697fb1b@10.9.117.42@o2ib4:5/0 lens 480/568 e 2 to 0 dl 1572911890 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 15:58:05 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 267 previous similar messages Nov 04 15:58:06 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa10e33a95e80 x1648438961259760/t0(0) o101->1a4ab277-fe05-2e26-b5d2-9132422c71ff@10.8.27.10@o2ib6:6/0 lens 4480/0 e 0 to 0 dl 1572911891 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 04 15:58:06 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 286 previous similar messages Nov 04 15:58:08 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa12e2f96ad00 x1649289061587696/t0(0) o101->06d17a9f-df4a-c46f-3ae9-bef8fc59e075@10.8.27.1@o2ib6:8/0 lens 4496/0 e 1 to 0 dl 1572911893 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 15:58:08 fir-md1-s1 kernel: Lustre: 22006:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 33 previous similar messages Nov 04 15:58:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 8cea164c-9518-4a0e-6f8c-c6ee346f8a71 (at 10.9.109.58@o2ib4) reconnecting Nov 04 15:58:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 8fc47135-56f5-75f0-aff9-e705b31d0c67 (at 10.9.117.7@o2ib4) reconnecting Nov 04 15:58:11 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 15:58:11 fir-md1-s1 kernel: Lustre: 22718:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffffa10e0e001f80 x1649288700608784/t0(0) o101->d246fb7a-efd2-a1e6-a2a3-ad672329506a@10.9.117.25@o2ib4:5/0 lens 480/536 e 0 to 0 dl 1572911890 ref 1 fl Complete:/0/0 rc 0/0 Nov 04 15:58:11 fir-md1-s1 kernel: LustreError: 22777:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.117.38@o2ib4: deadline 600:1s ago req@ffffa10e3524f080 x1649307047816320/t0(0) o101->9484b05f-bbba-6023-e820-67c71f8b4c9f@10.9.117.38@o2ib4:5/0 lens 480/0 e 0 to 0 dl 1572911890 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Nov 04 15:58:11 fir-md1-s1 kernel: LustreError: 22788:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa13e26299000 ns: mdt-fir-MDT0000_UUID lock: ffffa10e11a1af40/0x675682d7aa31a3d lrc: 3/0,0 mode: PW/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 794 type: IBT flags: 0x50200400000020 nid: 10.9.117.28@o2ib4 remote: 0x6517eb693bd99f49 expref: 38 pid: 22788 timeout: 0 lvb_type: 0 Nov 04 15:58:11 fir-md1-s1 kernel: LustreError: 22788:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 1 previous similar message Nov 04 15:58:11 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 15:58:11 fir-md1-s1 kernel: LustreError: Skipped 4 previous similar messages Nov 04 15:58:11 fir-md1-s1 kernel: LNet: 19748:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 1 seconds Nov 04 15:58:11 fir-md1-s1 kernel: Lustre: 22718:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 223 previous similar messages Nov 04 15:58:16 fir-md1-s1 kernel: LNet: 19748:0:(o2iblnd_cb.c:3396:kiblnd_check_conns()) Timed out tx for 10.0.10.3@o2ib7: 0 seconds Nov 04 15:58:29 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 39fed5ea-37c6-8f78-8229-955b2fa3b26f (at 10.9.113.11@o2ib4) reconnecting Nov 04 15:58:29 fir-md1-s1 kernel: Lustre: Skipped 40 previous similar messages Nov 04 15:58:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 65f17af2-0f81-d0c0-07ec-f60881376e81 (at 10.8.15.8@o2ib6) reconnecting Nov 04 15:59:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 8577d3de-6c7b-c5ca-b709-d67dcbca9946 (at 10.9.101.20@o2ib4) reconnecting Nov 04 15:59:21 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client f71a05cb-0281-cfe8-7d7a-9b10e944e06b (at 10.8.9.1@o2ib6) reconnecting Nov 04 15:59:21 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:00:40 fir-md1-s1 kernel: Lustre: 22989:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa12dde2dad00 x1649307082729104/t0(0) o101->69e6f715-e563-d488-ec1a-99417e975455@10.9.117.34@o2ib4:160/0 lens 480/568 e 0 to 0 dl 1572912045 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:00:40 fir-md1-s1 kernel: Lustre: 22989:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Nov 04 16:00:41 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.9.117.39@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffffa10e0fb18b40/0x675682d7aa320b8 lrc: 3/0,0 mode: PW/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 739 type: IBT flags: 0x60200400000020 nid: 10.9.117.39@o2ib4 remote: 0x6ac7aeec8dbc050b expref: 48 pid: 22864 timeout: 1318 lvb_type: 0 Nov 04 16:00:41 fir-md1-s1 kernel: LustreError: 22867:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa13e27fc0800 ns: mdt-fir-MDT0000_UUID lock: ffffa10e11a1cc80/0x675682d7aa320d4 lrc: 1/0,0 mode: --/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 737 type: IBT flags: 0x54a01400000020 nid: 10.9.117.39@o2ib4 remote: 0x6ac7aeec8dbc04e8 expref: 46 pid: 22867 timeout: 0 lvb_type: 0 Nov 04 16:00:41 fir-md1-s1 kernel: LustreError: 22867:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 8 previous similar messages Nov 04 16:00:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 69e6f715-e563-d488-ec1a-99417e975455 (at 10.9.117.34@o2ib4) reconnecting Nov 04 16:00:47 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Nov 04 16:01:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 5029860d-b620-6b82-de1b-7de16aac10db (at 10.9.101.24@o2ib4) Nov 04 16:01:07 fir-md1-s1 kernel: Lustre: Skipped 88 previous similar messages Nov 04 16:01:50 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 9fa94fe7-00f3-1fd6-8dd1-d572d676dfb3 (at 10.9.110.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc76b4000, cur 1572912110 expire 1572911960 last 1572911883 Nov 04 16:03:07 fir-md1-s1 kernel: LustreError: 23645:0:(ldlm_resource.c:1147:ldlm_resource_complain()) fir-MDT0000-lwp-MDT0000: namespace resource [0x200000006:0x2020000:0x0].0x0 (ffffa10e29a93080) refcount nonzero (1) after lock cleanup; forcing cleanup. Nov 04 16:03:07 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Nov 04 16:03:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.8.24.35@o2ib6 (stopping) Nov 04 16:03:07 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22872:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa13e27e92c00 ns: mdt-fir-MDT0000_UUID lock: ffffa13dcc875100/0x675682d7aa3211a lrc: 3/0,0 mode: PW/PW res: [0x200032083:0x1:0x0].0x0 bits 0x40/0x0 rrc: 741 type: IBT flags: 0x50200400000020 nid: 10.9.117.31@o2ib4 remote: 0x22321efe95d2a128 expref: 47 pid: 22872 timeout: 0 lvb_type: 0 Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22872:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 1 previous similar message Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22776:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.48@o2ib4 arrived at 1572912188 with bad export cookie 465392681041451745 Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22776:0:(ldlm_lock.c:2710:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffffa11e2af5d580/0x675682d7aa2f35b lrc: 3/0,0 mode: PW/PW res: [0x200031931:0xabb:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x40000000000000 nid: 10.9.101.48@o2ib4 remote: 0xafba66f160b8e0d4 expref: 38 pid: 21998 timeout: 0 lvb_type: 0 Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 21709:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0002-osc-MDT0000: cannot cleanup orphans: rc = -108 Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 21709:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 87 previous similar messages Nov 04 16:03:08 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.117.36@o2ib4 (stopping) Nov 04 16:03:08 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22865:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.107.3@o2ib4: deadline 100:598s ago req@ffffa13d7f37f080 x1649316141776528/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572911590 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:03:08 fir-md1-s1 kernel: Lustre: 22955:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:590s); client may timeout. req@ffffa12d4cf5e780 x1649316196305344/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572911598 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:03:08 fir-md1-s1 kernel: Lustre: 22955:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 11 previous similar messages Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 22865:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 176 previous similar messages Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 23645:0:(ldlm_resource.c:1147:ldlm_resource_complain()) mdt-fir-MDT0000_UUID: namespace resource [0x20003205d:0x5f28:0x0].0x0 (ffffa10e0b7a32c0) refcount nonzero (2) after lock cleanup; forcing cleanup. Nov 04 16:03:08 fir-md1-s1 kernel: LustreError: 23645:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 5 previous similar messages Nov 04 16:03:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.107.48@o2ib4 (stopping) Nov 04 16:03:09 fir-md1-s1 kernel: Lustre: Skipped 436 previous similar messages Nov 04 16:03:10 fir-md1-s1 kernel: LustreError: 23715:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-108 Nov 04 16:03:10 fir-md1-s1 kernel: LustreError: 23715:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 5 previous similar messages Nov 04 16:03:11 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.105.32@o2ib4 (stopping) Nov 04 16:03:11 fir-md1-s1 kernel: Lustre: Skipped 90 previous similar messages Nov 04 16:03:11 fir-md1-s1 kernel: LustreError: 22825:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.23@o2ib4 arrived at 1572912191 with bad export cookie 465392681041452389 Nov 04 16:03:11 fir-md1-s1 kernel: LustreError: 22825:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Nov 04 16:03:14 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Nov 04 16:03:17 fir-md1-s1 kernel: sched: RT throttling activated Nov 04 16:03:24 fir-md1-s1 kernel: LustreError: 21579:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.105@o2ib7 arrived at 1572912204 with bad export cookie 465392681042057763 Nov 04 16:03:24 fir-md1-s1 kernel: LustreError: 22776:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.105@o2ib7 arrived at 1572912204 with bad export cookie 465392681042057798 Nov 04 16:03:24 fir-md1-s1 kernel: LustreError: 21579:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 12 previous similar messages Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22825:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.103@o2ib7 arrived at 1572912235 with bad export cookie 465392681042196804 Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22617:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.103@o2ib7 arrived at 1572912235 with bad export cookie 465392681042196825 Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22971:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.103@o2ib7 arrived at 1572912235 with bad export cookie 465392681042196783 Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22967:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.103@o2ib7 arrived at 1572912235 with bad export cookie 465392681042196776 Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22617:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22971:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22967:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Nov 04 16:03:55 fir-md1-s1 kernel: LustreError: 22825:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 11 previous similar messages Nov 04 16:06:54 fir-md1-s1 kernel: LustreError: 21582:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.104@o2ib7 arrived at 1572912414 with bad export cookie 465392681042537634 Nov 04 16:06:54 fir-md1-s1 kernel: LustreError: 21581:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.104@o2ib7 arrived at 1572912414 with bad export cookie 465392681042537641 Nov 04 16:06:54 fir-md1-s1 kernel: LustreError: 21581:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 1 previous similar message Nov 04 16:06:54 fir-md1-s1 kernel: LustreError: 21582:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 12 previous similar messages Nov 04 16:07:28 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 16:07:28 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 16:07:29 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.103.63@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:07:29 fir-md1-s1 kernel: LustreError: Skipped 20 previous similar messages Nov 04 16:08:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 04 16:08:00 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 16:08:00 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Nov 04 16:08:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1284 clients reconnect Nov 04 16:08:11 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912480/real 1572912480] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912491 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Nov 04 16:08:22 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912491/real 1572912491] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912502 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:08:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 6c3fdbb8-f579-b7a4-c417-b5982815b7ed (at 10.9.108.36@o2ib4), waiting for 1284 known clients (578 recovered, 44 in progress, and 0 evicted) to recover in 2:01 Nov 04 16:08:30 fir-md1-s1 kernel: Lustre: Skipped 24 previous similar messages Nov 04 16:08:30 fir-md1-s1 kernel: LustreError: 24235:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa12d543cf500 x1649241187420048/t0(0) o601->fir-MDT0000-lwp-OST004b_UUID@10.0.10.114@o2ib7:625/0 lens 336/0 e 0 to 0 dl 1572913265 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:08:30 fir-md1-s1 kernel: LustreError: 24235:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 95 previous similar messages Nov 04 16:08:33 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912502/real 1572912502] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912513 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:08:44 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912513/real 1572912513] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912524 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:08:55 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912524/real 1572912524] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912535 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:09:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client a9a34ed4-bbde-ec18-c686-03db6cd11dd1 (at 10.9.108.25@o2ib4), waiting for 1284 known clients (581 recovered, 41 in progress, and 0 evicted) to recover in 1:25 Nov 04 16:09:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:09:06 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912535/real 1572912535] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912546 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:09:08 fir-md1-s1 kernel: LustreError: 24337:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa0ff693b9b00 x1649241033927744/t0(0) o601->fir-MDT0000-lwp-OST0022_UUID@10.0.10.105@o2ib7:663/0 lens 336/0 e 0 to 0 dl 1572913303 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:09:08 fir-md1-s1 kernel: LustreError: 24337:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 9 previous similar messages Nov 04 16:09:28 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912557/real 1572912557] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912568 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:09:28 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message Nov 04 16:10:01 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572912590/real 1572912590] req@ffffa13e33496300 x1649316506755584/t0(0) o104->MGS@10.9.101.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572912601 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:10:01 fir-md1-s1 kernel: Lustre: 22252:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Nov 04 16:10:10 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 6c3fdbb8-f579-b7a4-c417-b5982815b7ed (at 10.9.108.36@o2ib4), waiting for 1284 known clients (1238 recovered, 43 in progress, and 0 evicted) to recover in 0:21 Nov 04 16:10:10 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Nov 04 16:10:15 fir-md1-s1 kernel: LustreError: 24340:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10e13bd1200 x1649241033934240/t0(0) o601->fir-MDT0000-lwp-OST0022_UUID@10.0.10.105@o2ib7:730/0 lens 336/0 e 0 to 0 dl 1572913370 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:10:15 fir-md1-s1 kernel: LustreError: 24340:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 19 previous similar messages Nov 04 16:10:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: recovery is timed out, evict stale exports Nov 04 16:10:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 3 stale clients Nov 04 16:10:31 fir-md1-s1 kernel: Lustre: 24197:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Nov 04 16:10:31 fir-md1-s1 kernel: Lustre: 24197:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 518 previous similar messages Nov 04 16:10:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 2:30, of 1284 clients 1281 recovered and 3 were evicted. Nov 04 16:10:34 fir-md1-s1 kernel: LustreError: 22252:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.101.22@o2ib4) failed to reply to blocking AST (req@ffffa13e33496300 x1649316506755584 status 0 rc -110), evict it ns: MGS lock: ffffa13dc77bca40/0x675682d7aa21c29 lrc: 3/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 1289 type: PLN flags: 0x40000400000020 nid: 10.9.101.22@o2ib4 remote: 0xa0a228131ec02bd3 expref: 8 pid: 22244 timeout: 0 lvb_type: 0 Nov 04 16:10:34 fir-md1-s1 kernel: LustreError: 138-a: MGS: A client on nid 10.9.101.22@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Nov 04 16:10:34 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1572912634s: evicting client at 10.9.101.22@o2ib4 ns: MGS lock: ffffa13dc77bca40/0x675682d7aa21c29 lrc: 3/0,0 mode: CR/CR res: [0x726966:0x2:0x0].0x0 rrc: 1289 type: PLN flags: 0x40000400000020 nid: 10.9.101.22@o2ib4 remote: 0xa0a228131ec02bd3 expref: 9 pid: 22244 timeout: 0 lvb_type: 0 Nov 04 16:14:39 fir-md1-s1 kernel: LNet: Service thread pid 24329 was inactive for 200.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:14:39 fir-md1-s1 kernel: Pid: 24329, comm: mdt_rdpg01_008 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:14:39 fir-md1-s1 kernel: Call Trace: Nov 04 16:14:39 fir-md1-s1 kernel: [] mdt_lsom_update+0x41/0x440 [mdt] Nov 04 16:14:39 fir-md1-s1 kernel: [] mdt_mfd_close+0x47f/0x860 [mdt] Nov 04 16:14:39 fir-md1-s1 kernel: [] mdt_close_internal+0x121/0x220 [mdt] Nov 04 16:14:39 fir-md1-s1 kernel: [] mdt_close+0x220/0x780 [mdt] Nov 04 16:14:39 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:14:39 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:14:39 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:14:39 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:14:39 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:14:39 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:14:39 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572912879.24329 Nov 04 16:17:37 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:17:37 fir-md1-s1 kernel: LustreError: Skipped 700 previous similar messages Nov 04 16:20:26 fir-md1-s1 kernel: Lustre: 23904:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa13e33cc3600 x1648687785975776/t0(0) o101->84fe4322-0bf9-6fc7-4c46-ef148bf26b79@10.9.104.20@o2ib4:591/0 lens 744/1600 e 0 to 0 dl 1572913231 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:20:26 fir-md1-s1 kernel: Lustre: 23904:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Nov 04 16:20:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 468f7b3c-1170-9b0c-3534-a8d3e646f567 (at 10.9.104.17@o2ib4) reconnecting Nov 04 16:20:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.27.1@o2ib6) Nov 04 16:20:32 fir-md1-s1 kernel: Lustre: Skipped 1388 previous similar messages Nov 04 16:20:32 fir-md1-s1 kernel: Lustre: Skipped 10 previous similar messages Nov 04 16:21:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client a3b508eb-f449-4b0a-3bd3-d12f6d97173e (at 10.8.22.9@o2ib6) reconnecting Nov 04 16:21:05 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:21:14 fir-md1-s1 kernel: Lustre: 24328:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa11da3a4e780 x1649043955725904/t0(0) o35->32ab1a50-74ca-44b4-212a-e2a408110c8d@10.9.101.13@o2ib4:639/0 lens 392/456 e 23 to 0 dl 1572913279 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:21:14 fir-md1-s1 kernel: Lustre: 24328:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 33 previous similar messages Nov 04 16:21:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 32ab1a50-74ca-44b4-212a-e2a408110c8d (at 10.9.101.13@o2ib4) reconnecting Nov 04 16:22:02 fir-md1-s1 kernel: LNet: Service thread pid 24631 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:22:02 fir-md1-s1 kernel: Pid: 24631, comm: mdt02_053 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:22:02 fir-md1-s1 kernel: Call Trace: Nov 04 16:22:02 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:22:02 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:22:02 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:22:02 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:22:02 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:22:02 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:22:02 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:22:02 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:22:02 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:22:02 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:22:02 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:22:02 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:22:02 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:22:02 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:22:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572913322.24631 Nov 04 16:22:24 fir-md1-s1 kernel: LNet: Service thread pid 24499 was inactive for 200.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:22:24 fir-md1-s1 kernel: Pid: 24499, comm: mdt00_010 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:22:24 fir-md1-s1 kernel: Call Trace: Nov 04 16:22:24 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:22:24 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:22:24 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:22:24 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:22:24 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:22:24 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:22:24 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:22:24 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:22:24 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:22:24 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:22:24 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:22:24 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:22:24 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:22:24 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:22:24 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572913344.24499 Nov 04 16:23:01 fir-md1-s1 kernel: Lustre: 24657:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa13dccb77080 x1648382941218048/t0(0) o101->4983aa27-7e1b-a0cf-2f90-20677288041d@10.9.102.46@o2ib4:746/0 lens 376/1600 e 0 to 0 dl 1572913386 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:23:01 fir-md1-s1 kernel: Lustre: 24657:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Nov 04 16:23:05 fir-md1-s1 kernel: Lustre: 24551:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa12ddb2d5580 x1648687325170672/t0(0) o101->6caeb0b0-1879-6bd2-8b60-f282f622a3f7@10.9.109.68@o2ib4:750/0 lens 376/1600 e 0 to 0 dl 1572913390 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:23:05 fir-md1-s1 kernel: Lustre: 24551:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 48 previous similar messages Nov 04 16:23:07 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 71e4c1d0-52ac-42b5-d3ea-39353cdf1435 (at 10.9.101.17@o2ib4) reconnecting Nov 04 16:23:12 fir-md1-s1 kernel: LNet: Service thread pid 24878 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:23:12 fir-md1-s1 kernel: Pid: 24878, comm: mdt01_090 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:23:12 fir-md1-s1 kernel: Call Trace: Nov 04 16:23:12 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:23:12 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:23:12 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:23:12 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:23:12 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:23:12 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:23:12 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:23:12 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:23:12 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:23:12 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:23:12 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:23:12 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:23:12 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:23:12 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:23:12 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572913392.24878 Nov 04 16:23:13 fir-md1-s1 kernel: Lustre: 24551:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa11e35f9da00 x1649044111477056/t0(0) o101->9e398d75-1972-c5ef-77d9-1177762bd09b@10.9.101.48@o2ib4:3/0 lens 376/1600 e 0 to 0 dl 1572913398 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:23:13 fir-md1-s1 kernel: Lustre: 24551:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 28 previous similar messages Nov 04 16:23:29 fir-md1-s1 kernel: Lustre: 24545:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa13ccef30480 x1648687731719376/t0(0) o101->5ab78d57-f2a2-6bde-6a71-52927141dfd7@10.9.104.46@o2ib4:19/0 lens 376/1600 e 0 to 0 dl 1572913414 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:23:29 fir-md1-s1 kernel: Lustre: 24545:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 23 previous similar messages Nov 04 16:23:43 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 3218724a-75fe-97b2-be47-c34f7c2b702a (at 10.9.107.15@o2ib4) reconnecting Nov 04 16:23:43 fir-md1-s1 kernel: Lustre: Skipped 81 previous similar messages Nov 04 16:23:47 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client e1b5aaa6-ff92-544e-4dcd-1920ce36637f (at 10.9.110.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6d4d400, cur 1572913427 expire 1572913277 last 1572913200 Nov 04 16:23:47 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 16:24:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client c20b1107-8bbf-11f6-f0e1-96c356a905b8 (at 10.9.110.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa12e14d27400, cur 1572913441 expire 1572913291 last 1572913214 Nov 04 16:24:01 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:24:01 fir-md1-s1 kernel: Lustre: 24547:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa13e3a238000 x1649043943647056/t0(0) o101->1ee626f9-d97c-7d51-b2c4-7b4904f3a2e4@10.9.101.70@o2ib4:51/0 lens 376/1600 e 0 to 0 dl 1572913446 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:24:01 fir-md1-s1 kernel: Lustre: 24547:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 32 previous similar messages Nov 04 16:24:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 06034f62-d0d7-87cf-21ad-0803a69fdd37 (at 10.9.102.68@o2ib4) reconnecting Nov 04 16:24:48 fir-md1-s1 kernel: Lustre: Skipped 23 previous similar messages Nov 04 16:25:16 fir-md1-s1 kernel: Lustre: 24581:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa10df4e79680 x1648694170440000/t0(0) o101->aadb781f-4f10-8fda-1583-af0fb4145c60@10.9.104.54@o2ib4:126/0 lens 376/1600 e 0 to 0 dl 1572913521 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:25:16 fir-md1-s1 kernel: Lustre: 24581:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages Nov 04 16:27:06 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client b8b7ffb5-0542-8024-8094-0defd0f81dd2 (at 10.8.27.8@o2ib6) reconnecting Nov 04 16:27:06 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Nov 04 16:27:18 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572913627/real 1572913627] req@ffffa13ce1adda00 x1649316507371040/t0(0) o104->fir-MDT0000@10.9.108.48@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572913638 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Nov 04 16:27:18 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 04 16:27:29 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572913638/real 1572913638] req@ffffa13ce1adda00 x1649316507371040/t0(0) o104->fir-MDT0000@10.9.108.48@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572913649 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:27:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:27:39 fir-md1-s1 kernel: LustreError: Skipped 23 previous similar messages Nov 04 16:27:51 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572913660/real 1572913660] req@ffffa13ce1adda00 x1649316507371040/t0(0) o104->fir-MDT0000@10.9.108.48@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572913671 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:27:51 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message Nov 04 16:28:25 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572913693/real 1572913693] req@ffffa13ce1adda00 x1649316507371040/t0(0) o104->fir-MDT0000@10.9.108.48@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1572913704 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Nov 04 16:28:25 fir-md1-s1 kernel: Lustre: 24538:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Nov 04 16:28:38 fir-md1-s1 kernel: Lustre: 24684:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffffa13dcfdd8900 x1649043481537824/t0(0) o101->5aa7d6de-875b-7f1e-fa2e-01fb0b68841a@10.9.108.1@o2ib4:327/0 lens 376/1600 e 6 to 0 dl 1572913722 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:28:38 fir-md1-s1 kernel: Lustre: 24684:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 25574:0:(ldlm_resource.c:1147:ldlm_resource_complain()) fir-MDT0000-lwp-MDT0000: namespace resource [0x200000006:0x20000:0x0].0x0 (ffffa12e3159ab40) refcount nonzero (1) after lock cleanup; forcing cleanup. Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 25574:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 38 previous similar messages Nov 04 16:28:39 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Nov 04 16:28:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.0.10.104@o2ib7 (stopping) Nov 04 16:28:39 fir-md1-s1 kernel: Lustre: Skipped 120 previous similar messages Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 24538:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.9.108.48@o2ib4) failed to reply to blocking AST (req@ffffa13ce1adda00 x1649316507371040 status 0 rc -5), evict it ns: mdt-fir-MDT0000_UUID lock: ffffa10e0e9445c0/0x675682d7ac46b5b lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x6:0x0].0x0 bits 0x13/0x0 rrc: 33 type: IBT flags: 0x60200400000020 nid: 10.9.108.48@o2ib4 remote: 0x56452c4277aaeb41 expref: 3 pid: 24263 timeout: 3142 lvb_type: 0 Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0000: A client on nid 10.9.108.48@o2ib4 was evicted due to a lock blocking callback time out: rc -5 Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 24545:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa13dc68f9400 ns: mdt-fir-MDT0000_UUID lock: ffffa13e2c544a40/0x675682d805ad920 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x6:0x0].0x0 bits 0x13/0x0 rrc: 30 type: IBT flags: 0x50200000000000 nid: 10.9.0.62@o2ib4 remote: 0xfc124b171fcc4cf7 expref: 1452 pid: 24545 timeout: 0 lvb_type: 0 Nov 04 16:28:39 fir-md1-s1 kernel: LustreError: 24545:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 338 previous similar messages Nov 04 16:28:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.0.62@o2ib4 (stopping) Nov 04 16:28:39 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Nov 04 16:28:40 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client debb87df-efd4-b53b-2ef3-20fc1c5eaaec (at 10.9.108.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc82b4400, cur 1572913720 expire 1572913570 last 1572913493 Nov 04 16:28:40 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 16:28:40 fir-md1-s1 kernel: LustreError: 23986:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0004-osc-MDT0000: cannot cleanup orphans: rc = -108 Nov 04 16:28:40 fir-md1-s1 kernel: LustreError: 23986:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 100 previous similar messages Nov 04 16:28:40 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.113.12@o2ib4 (stopping) Nov 04 16:28:40 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Nov 04 16:28:42 fir-md1-s1 kernel: LustreError: 25574:0:(ldlm_resource.c:1147:ldlm_resource_complain()) mdt-fir-MDT0000_UUID: namespace resource [0x200031931:0xacf:0x0].0x0 (ffffa12d8716b200) refcount nonzero (2) after lock cleanup; forcing cleanup. Nov 04 16:28:42 fir-md1-s1 kernel: LustreError: 25574:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 5 previous similar messages Nov 04 16:28:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.109.36@o2ib4 (stopping) Nov 04 16:28:42 fir-md1-s1 kernel: Lustre: Skipped 124 previous similar messages Nov 04 16:28:46 fir-md1-s1 kernel: LustreError: 22618:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.110@o2ib7 arrived at 1572913726 with bad export cookie 465392681044775646 Nov 04 16:28:46 fir-md1-s1 kernel: LustreError: 22618:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 17 previous similar messages Nov 04 16:28:46 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.116.12@o2ib4 (stopping) Nov 04 16:28:46 fir-md1-s1 kernel: Lustre: Skipped 220 previous similar messages Nov 04 16:28:51 fir-md1-s1 kernel: Lustre: 24493:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:345s); client may timeout. req@ffffa12d89ebda00 x1648382547994928/t0(0) o101->0a47e23b-d919-b6ef-44f1-3d23188ad0f5@10.9.103.72@o2ib4:746/0 lens 376/1568 e 0 to 0 dl 1572913386 ref 1 fl Complete:/0/0 rc -19/-19 Nov 04 16:28:51 fir-md1-s1 kernel: LNet: Service thread pid 24329 completed after 1052.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Nov 04 16:28:51 fir-md1-s1 kernel: Lustre: 24493:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 46 previous similar messages Nov 04 16:28:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.109.42@o2ib4 (stopping) Nov 04 16:28:54 fir-md1-s1 kernel: Lustre: Skipped 474 previous similar messages Nov 04 16:28:59 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Nov 04 16:29:41 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 16:29:41 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 16:29:54 fir-md1-s1 kernel: LustreError: 22617:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.116@o2ib7 arrived at 1572913794 with bad export cookie 465392681045028731 Nov 04 16:29:54 fir-md1-s1 kernel: LustreError: 22617:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 2 previous similar messages Nov 04 16:30:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 04 16:30:09 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 16:30:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Nov 04 16:30:14 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1283 clients reconnect Nov 04 16:30:32 fir-md1-s1 kernel: LustreError: 26155:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11dc9385050 x1649241183545520/t0(0) o601->fir-MDT0000-lwp-OST005e_UUID@10.0.10.115@o2ib7:437/0 lens 336/0 e 0 to 0 dl 1572914587 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:30:32 fir-md1-s1 kernel: LustreError: 26155:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 5 previous similar messages Nov 04 16:30:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to 10.0.10.51@o2ib7 (at 0@lo) Nov 04 16:30:34 fir-md1-s1 kernel: Lustre: Skipped 1439 previous similar messages Nov 04 16:31:08 fir-md1-s1 kernel: LustreError: 26156:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa1136c629b00 x1649241183555328/t0(0) o601->fir-MDT0000-lwp-OST005e_UUID@10.0.10.115@o2ib7:473/0 lens 336/0 e 0 to 0 dl 1572914623 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:31:08 fir-md1-s1 kernel: LustreError: 26156:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 4 previous similar messages Nov 04 16:31:40 fir-md1-s1 kernel: LustreError: 26156:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa11cfeb77500 x1649241034614752/t0(0) o601->fir-MDT0000-lwp-OST000f_UUID@10.0.10.104@o2ib7:505/0 lens 336/0 e 0 to 0 dl 1572914655 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 16:31:40 fir-md1-s1 kernel: LustreError: 26156:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 5 previous similar messages Nov 04 16:32:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: recovery is timed out, evict stale exports Nov 04 16:32:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 1 stale clients Nov 04 16:32:44 fir-md1-s1 kernel: Lustre: 26090:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Nov 04 16:32:44 fir-md1-s1 kernel: Lustre: 26090:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 47 previous similar messages Nov 04 16:32:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 2:30, of 1283 clients 1282 recovered and 1 was evicted. Nov 04 16:34:01 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client d3c01e0c-bb2c-42c3-386d-4b0690bec299 (at 10.9.110.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa1103d57e400, cur 1572914041 expire 1572913891 last 1572913814 Nov 04 16:36:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 82d481b1-41f7-4915-d55d-14263c8ee3bb (at 10.9.110.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa11e395a7c00, cur 1572914191 expire 1572914041 last 1572913964 Nov 04 16:36:31 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Nov 04 16:37:41 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:37:41 fir-md1-s1 kernel: LustreError: Skipped 259 previous similar messages Nov 04 16:37:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 51390574-c509-f8c2-383b-446baae03d6d (at 10.9.0.61@o2ib4) reconnecting Nov 04 16:37:44 fir-md1-s1 kernel: Lustre: Skipped 4 previous similar messages Nov 04 16:41:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.57@o2ib4) Nov 04 16:41:32 fir-md1-s1 kernel: Lustre: Skipped 80 previous similar messages Nov 04 16:42:39 fir-md1-s1 kernel: Lustre: 26706:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa12e39695a00 x1648687786050320/t0(0) o101->84fe4322-0bf9-6fc7-4c46-ef148bf26b79@10.9.104.20@o2ib4:414/0 lens 376/1600 e 0 to 0 dl 1572914564 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:44:20 fir-md1-s1 kernel: LNet: Service thread pid 26984 was inactive for 200.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:44:20 fir-md1-s1 kernel: Pid: 26984, comm: mdt03_087 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:44:20 fir-md1-s1 kernel: Call Trace: Nov 04 16:44:20 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:44:20 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:44:20 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:44:20 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:44:20 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:44:20 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:44:20 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:44:20 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:44:20 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:44:20 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:44:20 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:44:20 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:44:20 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:44:20 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:44:20 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572914660.26984 Nov 04 16:47:15 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client fc986e43-3f52-ee44-b77e-4e68f09360fb (at 10.9.108.43@o2ib4) reconnecting Nov 04 16:47:15 fir-md1-s1 kernel: Lustre: Skipped 146 previous similar messages Nov 04 16:47:43 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:47:43 fir-md1-s1 kernel: LustreError: Skipped 23 previous similar messages Nov 04 16:51:38 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 3c74ec16-d1d7-c441-cbd5-5fdb296db236 (at 10.9.101.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10c2deb6000, cur 1572915098 expire 1572914948 last 1572914871 Nov 04 16:51:38 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Nov 04 16:51:56 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.106.10@o2ib4) Nov 04 16:51:56 fir-md1-s1 kernel: Lustre: Skipped 164 previous similar messages Nov 04 16:52:07 fir-md1-s1 kernel: Lustre: 26964:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/-151), not sending early reply req@ffffa10c2e276300 x1649043556711856/t0(0) o101->16353a1c-88a9-e4a3-4477-42e748ac557a@10.9.108.15@o2ib4:226/0 lens 376/1600 e 0 to 0 dl 1572915131 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 16:52:07 fir-md1-s1 kernel: Lustre: 26964:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 291 previous similar messages Nov 04 16:52:48 fir-md1-s1 kernel: LNet: Service thread pid 26694 was inactive for 1203.95s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:52:48 fir-md1-s1 kernel: Pid: 26694, comm: mdt02_066 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:52:48 fir-md1-s1 kernel: Call Trace: Nov 04 16:52:48 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:52:48 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:52:48 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:52:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915168.26694 Nov 04 16:52:48 fir-md1-s1 kernel: Pid: 26123, comm: mdt01_006 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:52:48 fir-md1-s1 kernel: Call Trace: Nov 04 16:52:48 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:52:48 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:52:48 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:52:48 fir-md1-s1 kernel: Pid: 26349, comm: mdt01_009 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:52:48 fir-md1-s1 kernel: Call Trace: Nov 04 16:52:48 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:52:48 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:52:48 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:52:48 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:52:48 fir-md1-s1 kernel: Pid: 26355, comm: mdt01_010 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:52:48 fir-md1-s1 kernel: Call Trace: Nov 04 16:52:48 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:52:48 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:52:49 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:52:49 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:52:49 fir-md1-s1 kernel: Pid: 26359, comm: mdt01_011 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:52:49 fir-md1-s1 kernel: Call Trace: Nov 04 16:52:49 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:52:49 fir-md1-s1 kernel: [] lod_alloc_qos.constprop.18+0x205/0x1840 [lod] Nov 04 16:52:49 fir-md1-s1 kernel: [] lod_qos_prep_create+0x12d7/0x1890 [lod] Nov 04 16:52:49 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:52:49 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:52:49 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:52:49 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:52:49 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:52:49 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:52:49 fir-md1-s1 kernel: LNet: Service thread pid 26363 was inactive for 1204.77s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:53:08 fir-md1-s1 kernel: LNet: Service thread pid 26698 was inactive for 1202.43s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:53:08 fir-md1-s1 kernel: LNet: Skipped 313 previous similar messages Nov 04 16:53:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915188.26698 Nov 04 16:53:17 fir-md1-s1 kernel: LNet: Service thread pid 26732 was inactive for 1200.60s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:53:17 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915197.26732 Nov 04 16:53:29 fir-md1-s1 kernel: LNet: Service thread pid 26735 was inactive for 1203.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:53:29 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915209.26735 Nov 04 16:53:58 fir-md1-s1 kernel: LNet: Service thread pid 26695 was inactive for 1202.59s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:53:58 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915238.26695 Nov 04 16:54:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915242.26716 Nov 04 16:54:06 fir-md1-s1 kernel: LNet: Service thread pid 26731 was inactive for 1201.61s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:54:06 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Nov 04 16:54:06 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915246.26731 Nov 04 16:54:14 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915254.26701 Nov 04 16:54:18 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915258.26709 Nov 04 16:54:39 fir-md1-s1 kernel: LNet: Service thread pid 26725 was inactive for 1202.14s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:54:39 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Nov 04 16:54:39 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915279.26725 Nov 04 16:54:43 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915283.26793 Nov 04 16:54:51 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915291.26690 Nov 04 16:55:07 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915307.26679 Nov 04 16:55:29 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client a582ba50-0e57-95fe-3f36-d04c3438c28c (at 10.9.105.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6ab1800, cur 1572915329 expire 1572915179 last 1572915102 Nov 04 16:55:29 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 16:55:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client f3e3148e-c3d9-1e95-ad7b-f31e31645f11 (at 10.9.105.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0fe66912400, cur 1572915334 expire 1572915184 last 1572915107 Nov 04 16:55:40 fir-md1-s1 kernel: LNet: Service thread pid 26714 was inactive for 1200.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:55:40 fir-md1-s1 kernel: LNet: Skipped 6 previous similar messages Nov 04 16:55:40 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915340.26714 Nov 04 16:55:56 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915356.26843 Nov 04 16:56:29 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915389.26827 Nov 04 16:56:33 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915393.26539 Nov 04 16:56:37 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915397.26810 Nov 04 16:56:54 fir-md1-s1 kernel: LNet: Service thread pid 26722 was inactive for 1204.44s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:56:54 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Nov 04 16:56:54 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915414.26722 Nov 04 16:57:45 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 16:57:45 fir-md1-s1 kernel: LustreError: Skipped 23 previous similar messages Nov 04 16:57:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d61c01fd-7673-605a-fed5-dbd34b64cc55 (at 10.8.23.8@o2ib6) reconnecting Nov 04 16:57:47 fir-md1-s1 kernel: Lustre: Skipped 27 previous similar messages Nov 04 16:58:08 fir-md1-s1 kernel: LNet: Service thread pid 26527 was inactive for 1204.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:58:08 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Nov 04 16:58:08 fir-md1-s1 kernel: Pid: 26527, comm: mdt02_037 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:58:08 fir-md1-s1 kernel: Call Trace: Nov 04 16:58:08 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:58:08 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:58:08 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:58:08 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:58:08 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:58:08 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:58:08 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:58:08 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:58:08 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:58:08 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:58:08 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:58:08 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:58:08 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:58:08 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:58:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915488.26527 Nov 04 16:58:57 fir-md1-s1 kernel: LNet: Service thread pid 26570 was inactive for 1200.92s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:58:57 fir-md1-s1 kernel: Pid: 26570, comm: mdt00_058 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:58:57 fir-md1-s1 kernel: Call Trace: Nov 04 16:58:57 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:58:57 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:58:57 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:58:57 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:58:57 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:58:57 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:58:57 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:58:57 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:58:57 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:58:57 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:58:57 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:58:57 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:58:57 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:58:57 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:58:57 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915537.26570 Nov 04 16:59:38 fir-md1-s1 kernel: LNet: Service thread pid 26603 was inactive for 1201.61s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 16:59:38 fir-md1-s1 kernel: Pid: 26603, comm: mdt03_062 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:59:38 fir-md1-s1 kernel: Call Trace: Nov 04 16:59:38 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:59:38 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:59:38 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:59:38 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915578.26603 Nov 04 16:59:38 fir-md1-s1 kernel: Pid: 26571, comm: mdt00_059 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:59:38 fir-md1-s1 kernel: Call Trace: Nov 04 16:59:38 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:59:38 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:59:38 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:59:38 fir-md1-s1 kernel: Pid: 26713, comm: mdt01_095 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 16:59:38 fir-md1-s1 kernel: Call Trace: Nov 04 16:59:38 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 16:59:38 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 16:59:38 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 16:59:38 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 16:59:42 fir-md1-s1 kernel: LNet: Service thread pid 26724 was inactive for 1203.67s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 16:59:42 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915582.26724 Nov 04 16:59:59 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d38a46d7-9411-c5e7-6315-e43adbb2e668 (at 10.9.103.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10df5653c00, cur 1572915599 expire 1572915449 last 1572915372 Nov 04 17:02:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.107.13@o2ib4) Nov 04 17:02:47 fir-md1-s1 kernel: Lustre: Skipped 171 previous similar messages Nov 04 17:03:42 fir-md1-s1 kernel: Lustre: 27555:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1226), not sending early reply req@ffffa10e399ab600 x1649043938740784/t0(0) o101->6cbfcfb3-623c-b6c9-9ca5-3952dd8c5086@10.9.101.57@o2ib4:167/0 lens 376/1600 e 0 to 0 dl 1572915827 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 17:03:42 fir-md1-s1 kernel: Lustre: 27555:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages Nov 04 17:04:07 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b2f4a237-5d30-b5b7-4243-751ca5e7f731 (at 10.9.106.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0fe65bff000, cur 1572915847 expire 1572915697 last 1572915620 Nov 04 17:04:07 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:05:22 fir-md1-s1 kernel: LNet: Service thread pid 26885 was inactive for 1201.88s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:05:22 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Nov 04 17:05:22 fir-md1-s1 kernel: Pid: 26885, comm: mdt02_079 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:05:22 fir-md1-s1 kernel: Call Trace: Nov 04 17:05:22 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:05:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:05:22 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:05:22 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572915922.26885 Nov 04 17:05:22 fir-md1-s1 kernel: Pid: 26727, comm: mdt01_102 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:05:22 fir-md1-s1 kernel: Call Trace: Nov 04 17:05:22 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:05:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:05:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:05:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:05:22 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:05:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client d84bfd80-9170-84b3-0a17-3f86b551fab7 (at 10.9.107.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10e3e8db400, cur 1572915953 expire 1572915803 last 1572915726 Nov 04 17:05:53 fir-md1-s1 kernel: Lustre: Skipped 9 previous similar messages Nov 04 17:06:21 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Nov 4 17:06:21 2019 Nov 04 17:06:44 fir-md1-s1 kernel: LNet: Service thread pid 26919 was inactive for 1200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:06:44 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Nov 04 17:06:44 fir-md1-s1 kernel: Pid: 26919, comm: mdt03_083 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:06:44 fir-md1-s1 kernel: Call Trace: Nov 04 17:06:44 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:06:44 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:06:44 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:06:44 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:06:44 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:06:44 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:06:44 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:06:44 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:06:44 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:06:44 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:06:44 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:06:44 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:06:44 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:06:44 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:06:44 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916004.26919 Nov 04 17:06:52 fir-md1-s1 kernel: Pid: 26929, comm: mdt03_085 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:06:52 fir-md1-s1 kernel: Call Trace: Nov 04 17:06:52 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:06:52 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:06:52 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:06:52 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:06:52 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:06:52 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:06:52 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:06:52 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:06:52 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:06:52 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:06:52 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:06:52 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:06:52 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:06:52 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:06:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916012.26929 Nov 04 17:07:08 fir-md1-s1 kernel: Pid: 26159, comm: mdt_rdpg03_003 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:07:08 fir-md1-s1 kernel: Call Trace: Nov 04 17:07:08 fir-md1-s1 kernel: [] mdt_lsom_update+0x41/0x440 [mdt] Nov 04 17:07:08 fir-md1-s1 kernel: [] mdt_mfd_close+0x47f/0x860 [mdt] Nov 04 17:07:08 fir-md1-s1 kernel: [] mdt_close_internal+0x121/0x220 [mdt] Nov 04 17:07:08 fir-md1-s1 kernel: [] mdt_close+0x220/0x780 [mdt] Nov 04 17:07:08 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:07:08 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:07:08 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:07:08 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:07:08 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:07:08 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:07:08 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916028.26159 Nov 04 17:07:47 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 17:07:47 fir-md1-s1 kernel: LustreError: Skipped 14 previous similar messages Nov 04 17:07:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d61c01fd-7673-605a-fed5-dbd34b64cc55 (at 10.8.23.8@o2ib6) reconnecting Nov 04 17:07:48 fir-md1-s1 kernel: Lustre: Skipped 160 previous similar messages Nov 04 17:08:06 fir-md1-s1 kernel: LNet: Service thread pid 27099 was inactive for 1203.83s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 17:08:06 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916086.27099 Nov 04 17:12:32 fir-md1-s1 kernel: LNet: Service thread pid 26964 was inactive for 1200.67s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:12:32 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Nov 04 17:12:32 fir-md1-s1 kernel: Pid: 26964, comm: mdt03_086 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:12:32 fir-md1-s1 kernel: Call Trace: Nov 04 17:12:32 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:12:32 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:12:32 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:12:32 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:12:32 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:12:32 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:12:32 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:12:32 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:12:32 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:12:32 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:12:32 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:12:32 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:12:32 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:12:32 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:12:32 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916352.26964 Nov 04 17:12:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.107.16@o2ib4) Nov 04 17:12:48 fir-md1-s1 kernel: Lustre: Skipped 158 previous similar messages Nov 04 17:12:48 fir-md1-s1 kernel: Pid: 26715, comm: mdt01_097 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:12:48 fir-md1-s1 kernel: Call Trace: Nov 04 17:12:48 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:12:48 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:12:48 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:12:48 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:12:48 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:12:48 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:12:48 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:12:48 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:12:48 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:12:48 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:12:48 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:12:48 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:12:48 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:12:48 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:12:48 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916368.26715 Nov 04 17:12:52 fir-md1-s1 kernel: Pid: 26720, comm: mdt02_071 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:12:52 fir-md1-s1 kernel: Call Trace: Nov 04 17:12:52 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:12:52 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:12:52 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:12:52 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:12:52 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:12:52 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:12:52 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:12:52 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:12:52 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:12:52 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:12:52 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:12:52 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:12:52 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:12:52 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:12:52 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916372.26720 Nov 04 17:13:00 fir-md1-s1 kernel: Pid: 26696, comm: mdt01_088 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:13:00 fir-md1-s1 kernel: Call Trace: Nov 04 17:13:00 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:13:00 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:13:00 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:13:00 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:13:00 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:13:00 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:13:00 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:13:00 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:13:00 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:13:00 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:13:00 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:13:00 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:13:01 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:13:01 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:13:01 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:13:01 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:13:01 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:13:01 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:13:01 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:13:01 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916381.26696 Nov 04 17:13:43 fir-md1-s1 kernel: Lustre: 25804:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa136d52b6c00 x1649043947881536/t0(0) o35->6cbfcfb3-623c-b6c9-9ca5-3952dd8c5086@10.9.101.57@o2ib4:13/0 lens 392/456 e 24 to 0 dl 1572916428 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 17:13:43 fir-md1-s1 kernel: Lustre: 25804:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 30 previous similar messages Nov 04 17:14:31 fir-md1-s1 kernel: LNet: Service thread pid 26699 was inactive for 1203.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:14:31 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Nov 04 17:14:31 fir-md1-s1 kernel: Pid: 26699, comm: mdt00_086 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:14:31 fir-md1-s1 kernel: Call Trace: Nov 04 17:14:31 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:14:31 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:14:31 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:14:31 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:14:31 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:14:31 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:14:31 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:14:31 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:14:31 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:14:31 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:14:31 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:14:31 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:14:31 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:14:31 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:14:31 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916471.26699 Nov 04 17:15:28 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916528.27085 Nov 04 17:16:09 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916569.26826 Nov 04 17:17:49 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 17:17:49 fir-md1-s1 kernel: LustreError: Skipped 11 previous similar messages Nov 04 17:17:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d61c01fd-7673-605a-fed5-dbd34b64cc55 (at 10.8.23.8@o2ib6) reconnecting Nov 04 17:17:50 fir-md1-s1 kernel: Lustre: Skipped 249 previous similar messages Nov 04 17:19:42 fir-md1-s1 kernel: LNet: Service thread pid 26726 was inactive for 1202.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:19:42 fir-md1-s1 kernel: Pid: 26726, comm: mdt01_101 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:19:42 fir-md1-s1 kernel: Call Trace: Nov 04 17:19:42 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:19:42 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:19:42 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:19:42 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916782.26726 Nov 04 17:19:42 fir-md1-s1 kernel: Pid: 26838, comm: mdt01_104 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:19:42 fir-md1-s1 kernel: Call Trace: Nov 04 17:19:42 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:19:42 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:19:42 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:19:42 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:19:42 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:19:46 fir-md1-s1 kernel: Pid: 27328, comm: mdt01_110 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:19:46 fir-md1-s1 kernel: Call Trace: Nov 04 17:19:46 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:19:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:19:46 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:19:46 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916786.27328 Nov 04 17:19:46 fir-md1-s1 kernel: Pid: 27047, comm: mdt00_096 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:19:46 fir-md1-s1 kernel: Call Trace: Nov 04 17:19:46 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:19:46 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:19:46 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:19:46 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:19:46 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:20:23 fir-md1-s1 kernel: Pid: 27121, comm: mdt02_081 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:20:23 fir-md1-s1 kernel: Call Trace: Nov 04 17:20:23 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:20:23 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:20:23 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:20:23 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:20:23 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:20:23 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:20:23 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:20:23 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:20:23 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:20:23 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:20:23 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:20:23 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:20:23 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:20:23 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:20:23 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916823.27121 Nov 04 17:21:00 fir-md1-s1 kernel: LNet: Service thread pid 27344 was inactive for 1204.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 17:21:00 fir-md1-s1 kernel: LNet: Skipped 2 previous similar messages Nov 04 17:21:00 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916860.27344 Nov 04 17:21:04 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572916864.27046 Nov 04 17:21:56 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b11749f4-a422-eca3-ec6d-da7360c54c14 (at 10.9.117.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6dee000, cur 1572916916 expire 1572916766 last 1572916689 Nov 04 17:21:56 fir-md1-s1 kernel: Lustre: Skipped 25 previous similar messages Nov 04 17:22:05 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 09d5976f-36eb-8525-5fb7-c49d065b6a22 (at 10.9.117.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10e3911e000, cur 1572916925 expire 1572916775 last 1572916698 Nov 04 17:22:49 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.27.3@o2ib6) Nov 04 17:22:49 fir-md1-s1 kernel: Lustre: Skipped 144 previous similar messages Nov 04 17:23:47 fir-md1-s1 kernel: Lustre: 26983:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa11679f5ba80 x1648785721737456/t0(0) o41->ee45735a-3c72-071c-fe40-2e82d3a751bd@10.8.7.12@o2ib6:617/0 lens 440/0 e 0 to 0 dl 1572917032 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 17:23:47 fir-md1-s1 kernel: Lustre: 26983:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 142 previous similar messages Nov 04 17:27:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client c8532145-7e7c-8d40-dfc1-9ccf63a3262d (at 10.9.104.16@o2ib4) reconnecting Nov 04 17:27:50 fir-md1-s1 kernel: Lustre: Skipped 230 previous similar messages Nov 04 17:27:51 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 17:27:51 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Nov 04 17:32:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.104.20@o2ib4) Nov 04 17:32:50 fir-md1-s1 kernel: Lustre: Skipped 289 previous similar messages Nov 04 17:33:58 fir-md1-s1 kernel: Lustre: 27326:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa10b8ead1b00 x1648662153574800/t0(0) o41->51d039f0-180b-c2f2-39da-443d9476c206@10.8.7.16@o2ib6:473/0 lens 440/0 e 1 to 0 dl 1572917643 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 17:33:58 fir-md1-s1 kernel: Lustre: 27326:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 190 previous similar messages Nov 04 17:37:52 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client d61c01fd-7673-605a-fed5-dbd34b64cc55 (at 10.8.23.8@o2ib6) reconnecting Nov 04 17:37:52 fir-md1-s1 kernel: Lustre: Skipped 255 previous similar messages Nov 04 17:39:21 fir-md1-s1 kernel: LNet: Service thread pid 26706 was inactive for 1202.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 17:39:21 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Nov 04 17:39:21 fir-md1-s1 kernel: Pid: 26706, comm: mdt02_068 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 17:39:21 fir-md1-s1 kernel: Call Trace: Nov 04 17:39:21 fir-md1-s1 kernel: [] call_rwsem_down_write_failed+0x17/0x30 Nov 04 17:39:22 fir-md1-s1 kernel: [] lod_qos_statfs_update+0x97/0x2b0 [lod] Nov 04 17:39:22 fir-md1-s1 kernel: [] lod_qos_prep_create+0x16a/0x1890 [lod] Nov 04 17:39:22 fir-md1-s1 kernel: [] lod_declare_instantiate_components+0x9a/0x1d0 [lod] Nov 04 17:39:22 fir-md1-s1 kernel: [] lod_declare_layout_change+0xb65/0x10f0 [lod] Nov 04 17:39:22 fir-md1-s1 kernel: [] mdd_declare_layout_change+0x62/0x120 [mdd] Nov 04 17:39:22 fir-md1-s1 kernel: [] mdd_layout_change+0x882/0x1000 [mdd] Nov 04 17:39:22 fir-md1-s1 kernel: [] mdt_layout_change+0x337/0x430 [mdt] Nov 04 17:39:22 fir-md1-s1 kernel: [] mdt_intent_layout+0x7ee/0xcc0 [mdt] Nov 04 17:39:22 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 17:39:22 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 17:39:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 17:39:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 17:39:22 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 17:39:22 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572917962.26706 Nov 04 17:39:41 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 18f3424c-7cce-31b6-0c78-e6e7e21ffd78 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa0fe60c48800, cur 1572917981 expire 1572917831 last 1572917754 Nov 04 17:39:57 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Nov 4 17:39:57 2019 Nov 04 17:41:18 fir-md1-s1 kernel: Lustre: 28246:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572917477/real 1572917477] req@ffffa12a46bc7980 x1649316508029840/t0(0) o39->fir-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 1 to 1 dl 1572918078 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Nov 04 17:41:18 fir-md1-s1 kernel: Lustre: 28246:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message Nov 04 17:41:18 fir-md1-s1 kernel: LustreError: 28246:0:(ldlm_resource.c:1147:ldlm_resource_complain()) fir-MDT0000-lwp-MDT0000: namespace resource [0x200000006:0x20000:0x0].0x0 (ffffa13e39825080) refcount nonzero (1) after lock cleanup; forcing cleanup. Nov 04 17:41:18 fir-md1-s1 kernel: LustreError: 28246:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 314 previous similar messages Nov 04 17:41:18 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Nov 04 17:41:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.0.10.113@o2ib7 (stopping) Nov 04 17:41:18 fir-md1-s1 kernel: Lustre: Skipped 215 previous similar messages Nov 04 17:41:20 fir-md1-s1 kernel: LustreError: 25871:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) fir-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 Nov 04 17:41:20 fir-md1-s1 kernel: LustreError: 25871:0:(osp_precreate.c:940:osp_precreate_cleanup_orphans()) Skipped 92 previous similar messages Nov 04 17:41:20 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.108.15@o2ib4 arrived at 1572918080 with bad export cookie 465392681142855124 Nov 04 17:41:20 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lock.c:2710:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffffa11fa1b021c0/0x675682d81573cd2 lrc: 3/0,0 mode: CR/CR res: [0x20002a621:0x5475:0x0].0x0 bits 0x9/0x0 rrc: 11 type: IBT flags: 0x40200000000000 nid: 10.9.108.15@o2ib4 remote: 0x82d08336bce2f27b expref: 83 pid: 26838 timeout: 0 lvb_type: 0 Nov 04 17:41:20 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lock.c:2710:ldlm_lock_dump_handle()) Skipped 1 previous similar message Nov 04 17:41:20 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.102.25@o2ib4 (stopping) Nov 04 17:41:20 fir-md1-s1 kernel: Lustre: Skipped 50 previous similar messages Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 28246:0:(ldlm_resource.c:1147:ldlm_resource_complain()) mdt-fir-MDT0000_UUID: namespace resource [0x200031931:0xacf:0x0].0x0 (ffffa10dffe3acc0) refcount nonzero (2) after lock cleanup; forcing cleanup. Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 21578:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.112.3@o2ib4 arrived at 1572918081 with bad export cookie 465392681142856986 Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 21578:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 21 previous similar messages Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 28246:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 5 previous similar messages Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 28519:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-108 Nov 04 17:41:21 fir-md1-s1 kernel: LustreError: 28519:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 5 previous similar messages Nov 04 17:41:22 fir-md1-s1 kernel: LustreError: 28359:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.103.42@o2ib4 arrived at 1572918082 with bad export cookie 465392681142856706 Nov 04 17:41:24 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.0.10.110@o2ib7 (stopping) Nov 04 17:41:24 fir-md1-s1 kernel: Lustre: Skipped 103 previous similar messages Nov 04 17:41:25 fir-md1-s1 kernel: LustreError: 21584:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.0.61@o2ib4 arrived at 1572918085 with bad export cookie 465392681142862782 Nov 04 17:41:25 fir-md1-s1 kernel: LustreError: 21584:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 19 previous similar messages Nov 04 17:41:30 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.107.27@o2ib4 arrived at 1572918090 with bad export cookie 465392681142855691 Nov 04 17:41:30 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 9 previous similar messages Nov 04 17:41:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.117.5@o2ib4 (stopping) Nov 04 17:41:32 fir-md1-s1 kernel: Lustre: Skipped 188 previous similar messages Nov 04 17:41:39 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 17:41:39 fir-md1-s1 kernel: LustreError: Skipped 9 previous similar messages Nov 04 17:41:46 fir-md1-s1 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-fir-MDT0000_UUID namespace with 119 resources in use, (rc=-110) Nov 04 17:41:47 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.9.107.15@o2ib4 arrived at 1572918107 with bad export cookie 465392681142856335 Nov 04 17:41:47 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 4 previous similar messages Nov 04 17:41:48 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.107.18@o2ib4 (stopping) Nov 04 17:41:48 fir-md1-s1 kernel: Lustre: Skipped 374 previous similar messages Nov 04 17:42:04 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.8.27.12@o2ib6 arrived at 1572918124 with bad export cookie 465392681142860339 Nov 04 17:42:04 fir-md1-s1 kernel: LustreError: 25275:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 2 previous similar messages Nov 04 17:42:11 fir-md1-s1 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-fir-MDT0000_UUID namespace with 119 resources in use, (rc=-110) Nov 04 17:42:21 fir-md1-s1 kernel: LustreError: 28542:0:(qsd_reint.c:56:qsd_reint_completion()) fir-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x20000:0x0], rc:-108 Nov 04 17:42:21 fir-md1-s1 kernel: LustreError: 28542:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 2 previous similar messages Nov 04 17:42:22 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.114.4@o2ib4 (stopping) Nov 04 17:42:22 fir-md1-s1 kernel: Lustre: Skipped 141 previous similar messages Nov 04 17:42:36 fir-md1-s1 kernel: LustreError: 0-0: Forced cleanup waiting for mdt-fir-MDT0000_UUID namespace with 119 resources in use, (rc=-110) Nov 04 17:42:42 fir-md1-s1 kernel: LustreError: 22584:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.104@o2ib7 arrived at 1572918162 with bad export cookie 465392681143385290 Nov 04 17:42:42 fir-md1-s1 kernel: LustreError: 22584:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 28 previous similar messages Nov 04 17:42:44 fir-md1-s1 kernel: LNet: Service thread pid 26116 completed after 4200.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Nov 04 17:42:44 fir-md1-s1 kernel: Lustre: 26359:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:1748s); client may timeout. req@ffffa119edef6c00 x1647637336665008/t0(0) o101->ba15a10b-a95d-133b-f873-39f741c8accb@10.9.115.11@o2ib4:1/0 lens 328/0 e 0 to 0 dl 1572916416 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:42:44 fir-md1-s1 kernel: LustreError: 26379:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.11.28@o2ib6: deadline 100:2373s ago req@ffffa12e16da1b00 x1642777541211248/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572915791 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:42:44 fir-md1-s1 kernel: LNet: Skipped 370 previous similar messages Nov 04 17:42:52 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Nov 04 17:43:01 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client ec08c9d1-10bf-b6d8-119e-14ccf73a2f2d (at 10.9.114.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc7637000, cur 1572918181 expire 1572918031 last 1572917954 Nov 04 17:43:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:43:41 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Nov 4 17:43:41 2019 Nov 04 17:43:58 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 17:43:59 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 17:44:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.101.35@o2ib4 (not set up) Nov 04 17:44:28 fir-md1-s1 kernel: Lustre: Skipped 503 previous similar messages Nov 04 17:44:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 04 17:44:30 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 17:44:30 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Nov 04 17:44:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1290 clients reconnect Nov 04 17:44:31 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.8.30.14@o2ib6) Nov 04 17:44:31 fir-md1-s1 kernel: Lustre: Skipped 273 previous similar messages Nov 04 17:44:39 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 46c68581-0a7a-c26b-fe11-9d23d4c06392 (at 10.8.26.4@o2ib6), waiting for 1290 known clients (943 recovered, 30 in progress, and 0 evicted) to recover in 2:22 Nov 04 17:44:39 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 17:44:39 fir-md1-s1 kernel: LustreError: 28726:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10df5618900 x1649241186490528/t0(0) o601->fir-MDT0000-lwp-OST0050_UUID@10.0.10.113@o2ib7:354/0 lens 336/0 e 0 to 0 dl 1572919034 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:44:39 fir-md1-s1 kernel: LustreError: 28726:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 36 previous similar messages Nov 04 17:44:49 fir-md1-s1 kernel: LustreError: 28726:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10e1d64c800 x1649241186527008/t0(0) o601->fir-MDT0000-lwp-OST0048_UUID@10.0.10.113@o2ib7:364/0 lens 336/0 e 0 to 0 dl 1572919044 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:44:49 fir-md1-s1 kernel: LustreError: 29092:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10daf68c800 x1649241186526992/t0(0) o601->fir-MDT0000-lwp-OST0048_UUID@10.0.10.113@o2ib7:364/0 lens 336/0 e 0 to 0 dl 1572919044 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:44:49 fir-md1-s1 kernel: LustreError: 29092:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 5 previous similar messages Nov 04 17:45:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 46c68581-0a7a-c26b-fe11-9d23d4c06392 (at 10.8.26.4@o2ib6), waiting for 1290 known clients (1256 recovered, 32 in progress, and 0 evicted) to recover in 1:56 Nov 04 17:45:21 fir-md1-s1 kernel: LustreError: 28726:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa0ff6904d850 x1649241186548368/t0(0) o601->fir-MDT0000-lwp-OST0048_UUID@10.0.10.113@o2ib7:396/0 lens 336/0 e 0 to 0 dl 1572919076 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:45:21 fir-md1-s1 kernel: LustreError: 28726:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 17 previous similar messages Nov 04 17:45:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 65a331c7-c8b1-e0cb-86fe-525801257b2e (at 10.9.110.20@o2ib4), waiting for 1290 known clients (1256 recovered, 32 in progress, and 0 evicted) to recover in 1:18 Nov 04 17:45:42 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:46:26 fir-md1-s1 kernel: LustreError: 29082:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffffa10b4d74b600 x1649241186550432/t0(0) o601->fir-MDT0000-lwp-OST0048_UUID@10.0.10.113@o2ib7:461/0 lens 336/0 e 0 to 0 dl 1572919141 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 17:46:26 fir-md1-s1 kernel: LustreError: 29082:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 33 previous similar messages Nov 04 17:46:57 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client 65a331c7-c8b1-e0cb-86fe-525801257b2e (at 10.9.110.20@o2ib4), waiting for 1290 known clients (1256 recovered, 32 in progress, and 0 evicted) to recover in 0:03 Nov 04 17:46:57 fir-md1-s1 kernel: Lustre: Skipped 5 previous similar messages Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: recovery is timed out, evict stale exports Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 2 stale clients Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: 29007:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: 29007:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 111 previous similar messages Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6caeb0b0-1879-6bd2-8b60-f282f622a3f7 (at 10.9.109.68@o2ib4) reconnected, waiting for 1290 clients in recovery for 12:29 Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:47:01 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 2:30, of 1290 clients 1288 recovered and 2 were evicted. Nov 04 17:49:41 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.117.8@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffffa13aef3b5e80/0x675682d84f5c20d lrc: 3/0,0 mode: PW/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 933 type: IBT flags: 0x60200400000020 nid: 10.9.117.8@o2ib4 remote: 0x64baaf751f185649 expref: 225 pid: 29452 timeout: 7858 lvb_type: 0 Nov 04 17:49:41 fir-md1-s1 kernel: LustreError: 29574:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa10e2ada2800 ns: mdt-fir-MDT0000_UUID lock: ffffa11377bb9200/0x675682d84f5c276 lrc: 3/0,0 mode: PW/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 930 type: IBT flags: 0x50200400000020 nid: 10.9.117.8@o2ib4 remote: 0x64baaf751f185650 expref: 37 pid: 29574 timeout: 0 lvb_type: 0 Nov 04 17:49:41 fir-md1-s1 kernel: LustreError: 29574:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 11 previous similar messages Nov 04 17:52:12 fir-md1-s1 kernel: LustreError: 29058:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572918431, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa10be96721c0/0x675682d84f5c2a0 lrc: 3/0,1 mode: --/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 931 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 29058 timeout: 0 lvb_type: 0 Nov 04 17:52:12 fir-md1-s1 kernel: LustreError: 29058:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 131 previous similar messages Nov 04 17:53:34 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 2bccfc61-f792-b731-f815-5bc0d2c6d2df (at 10.9.105.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6a0ec00, cur 1572918814 expire 1572918664 last 1572918587 Nov 04 17:53:34 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:54:41 fir-md1-s1 kernel: LustreError: 29578:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572918581, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa124bfe39200/0x675682d84f6f407 lrc: 3/0,1 mode: --/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 931 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 29578 timeout: 0 lvb_type: 0 Nov 04 17:54:41 fir-md1-s1 kernel: LustreError: 29578:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 274 previous similar messages Nov 04 17:54:50 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client daabe3a2-111a-4e57-4ded-d15240d2bf1d (at 10.9.108.52@o2ib4) in 200 seconds. I think it's dead, and I am evicting it. exp ffffa13dc7f2cc00, cur 1572918890 expire 1572918740 last 1572918690 Nov 04 17:54:50 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 17:56:06 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client af21bd45-2b94-3d1a-7622-280e90af179f (at 10.9.105.4@o2ib4) in 178 seconds. I think it's dead, and I am evicting it. exp ffffa13dc92a1c00, cur 1572918966 expire 1572918816 last 1572918788 Nov 04 17:56:18 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 12db1734-6c11-a510-5911-1f8778c71046 (at 10.9.105.2@o2ib4) in 391 seconds. I think it's dead, and I am evicting it. exp ffffa10e2a613800, cur 1572918978 expire 1572918828 last 1572918587 Nov 04 17:59:41 fir-md1-s1 kernel: Lustre: 29366:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa13c63abba80 x1649307040066640/t0(0) o101->1f71d405-70b6-e77e-25f5-b126748c2783@10.9.117.31@o2ib4:506/0 lens 480/568 e 0 to 0 dl 1572919186 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 17:59:41 fir-md1-s1 kernel: Lustre: 29366:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 163 previous similar messages Nov 04 17:59:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 13fa6193-3244-ade1-fbaa-b38d691852b1 (at 10.9.117.4@o2ib4) reconnecting Nov 04 17:59:47 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.117.7@o2ib4) Nov 04 17:59:47 fir-md1-s1 kernel: Lustre: Skipped 1383 previous similar messages Nov 04 17:59:47 fir-md1-s1 kernel: Lustre: Skipped 70 previous similar messages Nov 04 18:00:58 fir-md1-s1 kernel: Lustre: 29366:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa136d6341f80 x1648516377119024/t0(0) o101->71a16c65-9b4b-bfb1-4d3b-deb3735dc117@10.9.102.52@o2ib4:583/0 lens 576/0 e 0 to 0 dl 1572919263 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 18:00:58 fir-md1-s1 kernel: Lustre: 29366:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1087 previous similar messages Nov 04 18:01:04 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 71a16c65-9b4b-bfb1-4d3b-deb3735dc117 (at 10.9.102.52@o2ib4) reconnecting Nov 04 18:01:04 fir-md1-s1 kernel: Lustre: Skipped 260 previous similar messages Nov 04 18:03:29 fir-md1-s1 kernel: Lustre: 29457:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa13cafa94800 x1648687748931584/t0(0) o101->35dbeca4-2e43-aa3d-e6f6-7a8cb186e580@10.9.104.53@o2ib4:734/0 lens 576/0 e 0 to 0 dl 1572919414 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 18:03:29 fir-md1-s1 kernel: Lustre: 29457:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 192 previous similar messages Nov 04 18:03:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 8eef2689-96ff-a8ab-e086-841469d52e32 (at 10.9.104.61@o2ib4) reconnecting Nov 04 18:03:34 fir-md1-s1 kernel: Lustre: Skipped 154 previous similar messages Nov 04 18:07:13 fir-md1-s1 kernel: LNet: Service thread pid 29280 was inactive for 1201.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 18:07:13 fir-md1-s1 kernel: Pid: 29280, comm: mdt03_009 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:07:13 fir-md1-s1 kernel: Call Trace: Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:07:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:07:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:07:13 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572919633.29280 Nov 04 18:07:13 fir-md1-s1 kernel: Pid: 30030, comm: mdt03_111 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:07:13 fir-md1-s1 kernel: Call Trace: Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:07:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:07:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:07:13 fir-md1-s1 kernel: Pid: 30029, comm: mdt03_110 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:07:13 fir-md1-s1 kernel: Call Trace: Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:07:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:07:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:07:13 fir-md1-s1 kernel: Pid: 30028, comm: mdt03_109 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:07:13 fir-md1-s1 kernel: Call Trace: Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:07:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:07:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:07:13 fir-md1-s1 kernel: Pid: 30025, comm: mdt03_108 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:07:13 fir-md1-s1 kernel: Call Trace: Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:07:13 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:07:13 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:07:13 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:07:13 fir-md1-s1 kernel: LNet: Service thread pid 30023 was inactive for 1201.67s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 18:07:13 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Nov 04 18:07:43 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client c3d0bd3d-e306-300f-e88a-162836a71677 (at 10.9.116.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10e0b535000, cur 1572919663 expire 1572919513 last 1572919436 Nov 04 18:07:43 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Nov 04 18:08:35 fir-md1-s1 kernel: Lustre: 29606:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa1126e796780 x1648470275486224/t0(0) o101->24fab89a-6f6a-550a-7225-4734c7f7b849@10.8.27.12@o2ib6:285/0 lens 376/0 e 1 to 0 dl 1572919720 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 04 18:08:35 fir-md1-s1 kernel: Lustre: 29606:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 462 previous similar messages Nov 04 18:08:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 6fed2ba3-b270-1e43-8fe7-c9eed479e113 (at 10.9.109.2@o2ib4) reconnecting Nov 04 18:08:53 fir-md1-s1 kernel: Lustre: Skipped 257 previous similar messages Nov 04 18:09:21 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 9381878c-6883-58e4-b959-1f0bab8da019 (at 10.9.101.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc6ce2800, cur 1572919761 expire 1572919611 last 1572919534 Nov 04 18:09:21 fir-md1-s1 kernel: Lustre: Skipped 3 previous similar messages Nov 04 18:09:32 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 6cbfcfb3-623c-b6c9-9ca5-3952dd8c5086 (at 10.9.101.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10e1f2db000, cur 1572919772 expire 1572919622 last 1572919545 Nov 04 18:09:32 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 18:09:44 fir-md1-s1 kernel: LNet: Service thread pid 29578 was inactive for 1203.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 18:09:44 fir-md1-s1 kernel: LNet: Skipped 432 previous similar messages Nov 04 18:09:44 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572919784.29578 Nov 04 18:09:50 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.55@o2ib4) Nov 04 18:09:50 fir-md1-s1 kernel: Lustre: Skipped 689 previous similar messages Nov 04 18:17:41 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client b196544e-055c-1a78-0473-f617fb8c2206 (at 10.9.101.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc76dd800, cur 1572920261 expire 1572920111 last 1572920034 Nov 04 18:17:41 fir-md1-s1 kernel: Lustre: Skipped 1 previous similar message Nov 04 18:17:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 2e8bd693-fc04-4642-fb37-2c8cc911484b (at 10.9.101.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa10e2ad4dc00, cur 1572920274 expire 1572920124 last 1572920047 Nov 04 18:18:36 fir-md1-s1 kernel: Lustre: 29691:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffffa12a3bbcb180 x1649323175862320/t0(0) o101->2ad8ff13-d978-9373-7245-882c6479cc4c@10.9.110.63@o2ib4:131/0 lens 576/0 e 0 to 0 dl 1572920321 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 04 18:18:36 fir-md1-s1 kernel: Lustre: 29691:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1472 previous similar messages Nov 04 18:18:53 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client bc665767-df8b-9748-9d41-68bff1ff621a (at 10.9.113.13@o2ib4) reconnecting Nov 04 18:18:53 fir-md1-s1 kernel: Lustre: Skipped 642 previous similar messages Nov 04 18:19:51 fir-md1-s1 kernel: Lustre: fir-MDT0000: Connection restored to (at 10.9.101.55@o2ib4) Nov 04 18:19:51 fir-md1-s1 kernel: Lustre: Skipped 662 previous similar messages Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: 30683:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572920389/real 1572920389] req@ffffa112717fa400 x1649316508966512/t0(0) o39->fir-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1572920395 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Nov 04 18:19:55 fir-md1-s1 kernel: LNet: Service thread pid 29058 completed after 1963.24s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: 29058:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:1208s); client may timeout. req@ffffa10e1aba1b00 x1649289077056880/t0(0) o101->b8b7ffb5-0542-8024-8094-0defd0f81dd2@10.8.27.8@o2ib6:507/0 lens 1784/0 e 0 to 0 dl 1572919187 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: 29058:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 668 previous similar messages Nov 04 18:19:55 fir-md1-s1 kernel: LustreError: 29058:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.117.8@o2ib4: deadline 100:1617s ago req@ffffa10a8430f980 x1649307094590048/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572918778 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:19:55 fir-md1-s1 kernel: LustreError: 29058:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 215 previous similar messages Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.116.2@o2ib4 (stopping) Nov 04 18:19:55 fir-md1-s1 kernel: Lustre: Skipped 2 previous similar messages Nov 04 18:19:56 fir-md1-s1 kernel: LustreError: 29580:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa10e398f2000 ns: mdt-fir-MDT0000_UUID lock: ffffa13e3f3257c0/0x675682d84f5c317 lrc: 3/0,0 mode: PW/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 914 type: IBT flags: 0x50200400000020 nid: 10.9.117.4@o2ib4 remote: 0xbbd87c3fe0c3a22a expref: 284 pid: 29580 timeout: 0 lvb_type: 0 Nov 04 18:19:56 fir-md1-s1 kernel: LNet: Service thread pid 29580 completed after 1964.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Nov 04 18:19:56 fir-md1-s1 kernel: Lustre: 29637:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:1209s); client may timeout. req@ffffa13dcc40ec00 x1647602275858064/t0(0) o101->c7da60e2-1da2-1aa0-71e3-83d7cefee0c5@10.9.112.3@o2ib4:507/0 lens 1768/0 e 0 to 0 dl 1572919187 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:19:56 fir-md1-s1 kernel: Lustre: 29637:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 697 previous similar messages Nov 04 18:19:56 fir-md1-s1 kernel: LustreError: 29637:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.110.20@o2ib4: deadline 100:1753s ago req@ffffa13c78641b00 x1649324407653488/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572918643 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:19:56 fir-md1-s1 kernel: LustreError: 29637:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 24 previous similar messages Nov 04 18:19:56 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 18:19:56 fir-md1-s1 kernel: LustreError: Skipped 67 previous similar messages Nov 04 18:19:56 fir-md1-s1 kernel: LNet: Skipped 11 previous similar messages Nov 04 18:20:00 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Nov 04 18:20:24 fir-md1-s1 kernel: LustreError: 21580:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.107@o2ib7 arrived at 1572920424 with bad export cookie 465392681212081778 Nov 04 18:20:24 fir-md1-s1 kernel: LustreError: 21580:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 36 previous similar messages Nov 04 18:21:03 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 18:21:03 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 18:21:13 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.101.57@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 18:21:13 fir-md1-s1 kernel: LustreError: Skipped 219 previous similar messages Nov 04 18:21:27 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.9.107.52@o2ib4 (not set up) Nov 04 18:21:27 fir-md1-s1 kernel: Lustre: Skipped 586 previous similar messages Nov 04 18:21:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 04 18:21:28 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 18:21:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Nov 04 18:21:28 fir-md1-s1 kernel: Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1279 clients reconnect Nov 04 18:21:34 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client a040630b-6b1d-e359-6ec4-01dbc14e42d3 (at 10.9.117.8@o2ib4), waiting for 1279 known clients (338 recovered, 23 in progress, and 0 evicted) to recover in 2:23 Nov 04 18:21:43 fir-md1-s1 kernel: Lustre: 31131:0:(ldlm_lib.c:1765:extend_recovery_timer()) fir-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1 Nov 04 18:21:43 fir-md1-s1 kernel: Lustre: 31131:0:(ldlm_lib.c:1765:extend_recovery_timer()) Skipped 285 previous similar messages Nov 04 18:21:44 fir-md1-s1 kernel: Lustre: fir-MDT0000: Recovery over after 0:16, of 1279 clients 1279 recovered and 0 were evicted. Nov 04 18:22:55 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Nov 4 18:22:54 2019 Nov 04 18:23:25 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.9.117.11@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffffa10c2969e300/0x675682d85038bf7 lrc: 3/0,0 mode: PW/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 906 type: IBT flags: 0x60200400000020 nid: 10.9.117.11@o2ib4 remote: 0xba5999bc9a8d07d4 expref: 17 pid: 31409 timeout: 9882 lvb_type: 0 Nov 04 18:24:05 fir-md1-s1 kernel: LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.8.11.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Nov 04 18:24:05 fir-md1-s1 kernel: LustreError: Skipped 302 previous similar messages Nov 04 18:24:54 fir-md1-s1 kernel: Lustre: 31859:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1572920688/real 1572920688] req@ffffa11e29439b00 x1649316509092880/t0(0) o39->fir-MDT0000-lwp-MDT0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1572920694 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Nov 04 18:24:54 fir-md1-s1 kernel: Lustre: Failing over fir-MDT0000 Nov 04 18:24:54 fir-md1-s1 kernel: Lustre: fir-MDT0000: Not available for connect from 10.0.10.116@o2ib7 (stopping) Nov 04 18:24:54 fir-md1-s1 kernel: Lustre: Skipped 51 previous similar messages Nov 04 18:24:54 fir-md1-s1 kernel: LustreError: 31401:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa10e3903b400 ns: mdt-fir-MDT0000_UUID lock: ffffa11f2f7ab3c0/0x675682d850391d1 lrc: 3/0,0 mode: PW/PW res: [0x200032088:0x94:0x0].0x0 bits 0x40/0x0 rrc: 926 type: IBT flags: 0x50200400000020 nid: 10.9.117.40@o2ib4 remote: 0xeb83584a2efbe45e expref: 49 pid: 31401 timeout: 0 lvb_type: 0 Nov 04 18:24:54 fir-md1-s1 kernel: LustreError: 31401:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 438 previous similar messages Nov 04 18:24:55 fir-md1-s1 kernel: LustreError: 31249:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.101.57@o2ib4: deadline 100:72s ago req@ffffa12ce5cfc800 x1649325833719424/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572920623 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:24:55 fir-md1-s1 kernel: LustreError: 31249:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 78 previous similar messages Nov 04 18:24:55 fir-md1-s1 kernel: Lustre: 31249:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:72s); client may timeout. req@ffffa12ce5cfc800 x1649325833719424/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1572920623 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Nov 04 18:24:55 fir-md1-s1 kernel: Lustre: 31249:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2105 previous similar messages Nov 04 18:24:56 fir-md1-s1 kernel: Lustre: server umount fir-MDT0000 complete Nov 04 18:25:19 fir-md1-s1 kernel: LustreError: 22971:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.103@o2ib7 arrived at 1572920719 with bad export cookie 465392681217252853 Nov 04 18:25:19 fir-md1-s1 kernel: LustreError: 22971:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 21 previous similar messages Nov 04 18:25:20 fir-md1-s1 kernel: LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 Nov 04 18:25:20 fir-md1-s1 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Nov 04 18:25:25 fir-md1-s1 kernel: Lustre: MGS: haven't heard from client 19e4440e-5621-b53b-62eb-ef1b1a0fc59d (at 10.9.108.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa13dc7eefc00, cur 1572920725 expire 1572920575 last 1572920498 Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: fir-MDD0000: changelog on Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: in recovery but waiting for the first client to connect Nov 04 18:25:42 fir-md1-s1 kernel: LustreError: 31983:0:(mdt_handler.c:6687:mdt_iocontrol()) fir-MDT0000: Aborting recovery for device Nov 04 18:25:42 fir-md1-s1 kernel: LustreError: 31983:0:(ldlm_lib.c:2605:target_stop_recovery_thread()) fir-MDT0000: Aborting recovery Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: 32287:0:(ldlm_lib.c:2056:target_recovery_overseer()) recovery is aborted, evict exports in recovery Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: disconnecting 1283 stale clients Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: fir-MDT0000: Denying connection for new client f8d5264b-1de7-5abd-fef8-60297df9f169 (at 10.8.22.20@o2ib6), waiting for 1284 known clients (1 recovered, 0 in progress, and 0 evicted) already passed deadline 166:59 Nov 04 18:25:42 fir-md1-s1 kernel: Lustre: Skipped 6 previous similar messages Nov 04 18:29:10 fir-md1-s1 kernel: Lustre: DEBUG MARKER: Mon Nov 4 18:29:10 2019 Nov 04 18:29:25 fir-md1-s1 kernel: LustreError: 21591:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.117.13@o2ib4 ns: mdt-fir-MDT0000_UUID lock: ffffa10e0bbb7500/0x675682d854097ee lrc: 3/0,0 mode: PW/PW res: [0x200034eb7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 913 type: IBT flags: 0x60200400000020 nid: 10.9.117.13@o2ib4 remote: 0x241142a8c8b47dde expref: 212 pid: 32835 timeout: 10242 lvb_type: 0 Nov 04 18:29:25 fir-md1-s1 kernel: LustreError: 32404:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) ### lock on destroyed export ffffa1273281a000 ns: mdt-fir-MDT0000_UUID lock: ffffa12d92719d40/0x675682d85409818 lrc: 1/0,0 mode: --/PW res: [0x200034eb7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 910 type: IBT flags: 0x54a01400000020 nid: 10.9.117.13@o2ib4 remote: 0x241142a8c8b47de5 expref: 83 pid: 32404 timeout: 0 lvb_type: 0 Nov 04 18:29:25 fir-md1-s1 kernel: LustreError: 32404:0:(ldlm_lockd.c:1348:ldlm_handle_enqueue0()) Skipped 435 previous similar messages Nov 04 18:29:36 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client 0e67c056-6635-c70f-79c1-ba277ef85897 (at 10.9.107.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffffa139f3fb0400, cur 1572920976 expire 1572920826 last 1572920749 Nov 04 18:30:16 fir-md1-s1 kernel: LNet: Service thread pid 32592 was inactive for 200.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 18:30:16 fir-md1-s1 kernel: LNet: Skipped 4 previous similar messages Nov 04 18:30:16 fir-md1-s1 kernel: Pid: 32592, comm: mdt01_036 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:30:16 fir-md1-s1 kernel: Call Trace: Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:30:16 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:30:16 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:30:16 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572921016.32592 Nov 04 18:30:16 fir-md1-s1 kernel: Pid: 32603, comm: mdt01_039 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:30:16 fir-md1-s1 kernel: Call Trace: Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:30:16 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:30:16 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:30:16 fir-md1-s1 kernel: Pid: 32383, comm: mdt01_017 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:30:16 fir-md1-s1 kernel: Call Trace: Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:30:16 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:30:16 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:30:16 fir-md1-s1 kernel: Pid: 32392, comm: mdt02_022 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:30:16 fir-md1-s1 kernel: Call Trace: Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Nov 04 18:30:16 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:30:17 fir-md1-s1 kernel: LNet: Service thread pid 32408 was inactive for 201.10s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 04 18:30:17 fir-md1-s1 kernel: LNet: Skipped 3 previous similar messages Nov 04 18:30:17 fir-md1-s1 kernel: Pid: 32408, comm: mdt01_024 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 Nov 04 18:30:17 fir-md1-s1 kernel: Call Trace: Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_completion_ast+0x430/0x860 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_local_lock+0x50b/0xb20 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_lock_internal+0x70/0x360 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_object_lock+0x20/0x30 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_brw_enqueue+0x44b/0x760 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_intent_brw+0x1f/0x30 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] mdt_intent_policy+0x435/0xd80 [mdt] Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] ptlrpc_main+0xb2c/0x1460 [ptlrpc] Nov 04 18:30:17 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Nov 04 18:30:17 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Nov 04 18:30:17 fir-md1-s1 kernel: [] 0xffffffffffffffff Nov 04 18:30:17 fir-md1-s1 kernel: LNet: Service thread pid 32415 was inactive for 201.19s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 18:30:17 fir-md1-s1 kernel: LNet: Skipped 1 previous similar message Nov 04 18:31:56 fir-md1-s1 kernel: LustreError: 32601:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572920815, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa10e0b407bc0/0x675682d854098c0 lrc: 3/0,1 mode: --/PW res: [0x200034eb7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 912 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 32601 timeout: 0 lvb_type: 0 Nov 04 18:31:56 fir-md1-s1 kernel: LustreError: 32601:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 224 previous similar messages Nov 04 18:34:25 fir-md1-s1 kernel: LustreError: 32404:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1572920965, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffffa12da9254ec0/0x675682d8540d5dd lrc: 3/0,1 mode: --/PW res: [0x200034eb7:0x1:0x0].0x0 bits 0x40/0x0 rrc: 913 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 32404 timeout: 0 lvb_type: 0 Nov 04 18:34:25 fir-md1-s1 kernel: LustreError: 32404:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 161 previous similar messages Nov 04 18:35:02 fir-md1-s1 kernel: LNet: Service thread pid 32398 was inactive for 336.82s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 04 18:35:02 fir-md1-s1 kernel: LNet: Skipped 319 previous similar messages Nov 04 18:35:02 fir-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1572921302.32398 Nov 04 18:36:28 fir-md1-s1 kernel: Lustre: MGS: Connection restored to (at 10.9.101.51@o2ib4) Nov 04 18:36:28 fir-md1-s1 kernel: Lustre: Skipped 2682 previous similar messages Nov 04 18:36:50 fir-md1-s1 kernel: Lustre: 33025:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffffa10c267d8000 x1649288723951200/t0(0) o101->1171a9b6-e69f-0d88-ca9a-349489fea18c@10.9.117.1@o2ib4:470/0 lens 480/568 e 24 to 0 dl 1572921415 ref 2 fl Interpret:/0/0 rc 0/0 Nov 04 18:36:50 fir-md1-s1 kernel: Lustre: 33025:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 66 previous similar messages Nov 04 18:36:56 fir-md1-s1 kernel: Lustre: fir-MDT0000: Client 1171a9b6-e69f-0d88-ca9a-349489fea18c (at 10.9.117.1@o2ib4) reconnecting Nov 04 18:36:56 fir-md1-s1 kernel: Lustre: Skipped 28 previous similar messages