[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.27.2.el7_lustre.pl2.x86_64 (sthiell@oak-rbh01) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Thu Nov 7 15:26:16 PST 2019 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=1a559904-7ee3-4f19-99ef-40792627f881 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000008efff] usable [ 0.000000] BIOS-e820: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000000090000-0x000000000009ffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000004f882fff] usable [ 0.000000] BIOS-e820: [mem 0x000000004f883000-0x000000005788bfff] reserved [ 0.000000] BIOS-e820: [mem 0x000000005788c000-0x000000006cacefff] usable [ 0.000000] BIOS-e820: [mem 0x000000006cacf000-0x000000006efcefff] reserved [ 0.000000] BIOS-e820: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed80fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000107f37ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000107f380000-0x000000107fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000001080000000-0x000000207ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000207ff80000-0x000000207fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000002080000000-0x000000307ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000307ff80000-0x000000307fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000003080000000-0x000000407ff7ffff] usable [ 0.000000] BIOS-e820: [mem 0x000000407ff80000-0x000000407fffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] e820: update [mem 0x3707c020-0x370adc5f] usable ==> usable [ 0.000000] e820: update [mem 0x3704a020-0x3707bc5f] usable ==> usable [ 0.000000] e820: update [mem 0x37041020-0x3704905f] usable ==> usable [ 0.000000] e820: update [mem 0x37028020-0x3704065f] usable ==> usable [ 0.000000] extended physical RAM map: [ 0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000008efff] usable [ 0.000000] reserve setup_data: [mem 0x000000000008f000-0x000000000008ffff] ACPI NVS [ 0.000000] reserve setup_data: [mem 0x0000000000090000-0x000000000009ffff] usable [ 0.000000] reserve setup_data: [mem 0x0000000000100000-0x000000003702801f] usable [ 0.000000] reserve setup_data: [mem 0x0000000037028020-0x000000003704065f] usable [ 0.000000] reserve setup_data: [mem 0x0000000037040660-0x000000003704101f] usable [ 0.000000] reserve setup_data: [mem 0x0000000037041020-0x000000003704905f] usable [ 0.000000] reserve setup_data: [mem 0x0000000037049060-0x000000003704a01f] usable [ 0.000000] reserve setup_data: [mem 0x000000003704a020-0x000000003707bc5f] usable [ 0.000000] reserve setup_data: [mem 0x000000003707bc60-0x000000003707c01f] usable [ 0.000000] reserve setup_data: [mem 0x000000003707c020-0x00000000370adc5f] usable [ 0.000000] reserve setup_data: [mem 0x00000000370adc60-0x000000004f882fff] usable [ 0.000000] reserve setup_data: [mem 0x000000004f883000-0x000000005788bfff] reserved [ 0.000000] reserve setup_data: [mem 0x000000005788c000-0x000000006cacefff] usable [ 0.000000] reserve setup_data: [mem 0x000000006cacf000-0x000000006efcefff] reserved [ 0.000000] reserve setup_data: [mem 0x000000006efcf000-0x000000006fdfefff] ACPI NVS [ 0.000000] reserve setup_data: [mem 0x000000006fdff000-0x000000006fffefff] ACPI data [ 0.000000] reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable [ 0.000000] reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fed80000-0x00000000fed80fff] reserved [ 0.000000] reserve setup_data: [mem 0x0000000100000000-0x000000107f37ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000107f380000-0x000000107fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000001080000000-0x000000207ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000207ff80000-0x000000207fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000002080000000-0x000000307ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000307ff80000-0x000000307fffffff] reserved [ 0.000000] reserve setup_data: [mem 0x0000003080000000-0x000000407ff7ffff] usable [ 0.000000] reserve setup_data: [mem 0x000000407ff80000-0x000000407fffffff] reserved [ 0.000000] efi: EFI v2.50 by Dell Inc. [ 0.000000] efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x6eab5000 SMBIOS 3.0=0x6eab3000 [ 0.000000] efi: mem00: type=3, attr=0xf, range=[0x0000000000000000-0x0000000000001000) (0MB) [ 0.000000] efi: mem01: type=2, attr=0xf, range=[0x0000000000001000-0x0000000000002000) (0MB) [ 0.000000] efi: mem02: type=7, attr=0xf, range=[0x0000000000002000-0x0000000000010000) (0MB) [ 0.000000] efi: mem03: type=3, attr=0xf, range=[0x0000000000010000-0x0000000000014000) (0MB) [ 0.000000] efi: mem04: type=7, attr=0xf, range=[0x0000000000014000-0x0000000000063000) (0MB) [ 0.000000] efi: mem05: type=3, attr=0xf, range=[0x0000000000063000-0x000000000008f000) (0MB) [ 0.000000] efi: mem06: type=10, attr=0xf, range=[0x000000000008f000-0x0000000000090000) (0MB) [ 0.000000] efi: mem07: type=3, attr=0xf, range=[0x0000000000090000-0x00000000000a0000) (0MB) [ 0.000000] efi: mem08: type=4, attr=0xf, range=[0x0000000000100000-0x0000000000120000) (0MB) [ 0.000000] efi: mem09: type=7, attr=0xf, range=[0x0000000000120000-0x0000000000c00000) (10MB) [ 0.000000] efi: mem10: type=3, attr=0xf, range=[0x0000000000c00000-0x0000000001000000) (4MB) [ 0.000000] efi: mem11: type=2, attr=0xf, range=[0x0000000001000000-0x000000000267b000) (22MB) [ 0.000000] efi: mem12: type=7, attr=0xf, range=[0x000000000267b000-0x0000000004000000) (25MB) [ 0.000000] efi: mem13: type=4, attr=0xf, range=[0x0000000004000000-0x000000000403b000) (0MB) [ 0.000000] efi: mem14: type=7, attr=0xf, range=[0x000000000403b000-0x0000000037028000) (815MB) [ 0.000000] efi: mem15: type=2, attr=0xf, range=[0x0000000037028000-0x000000004eee6000) (382MB) [ 0.000000] efi: mem16: type=7, attr=0xf, range=[0x000000004eee6000-0x000000004eeea000) (0MB) [ 0.000000] efi: mem17: type=2, attr=0xf, range=[0x000000004eeea000-0x000000004eeec000) (0MB) [ 0.000000] efi: mem18: type=1, attr=0xf, range=[0x000000004eeec000-0x000000004f009000) (1MB) [ 0.000000] efi: mem19: type=2, attr=0xf, range=[0x000000004f009000-0x000000004f128000) (1MB) [ 0.000000] efi: mem20: type=1, attr=0xf, range=[0x000000004f128000-0x000000004f237000) (1MB) [ 0.000000] efi: mem21: type=3, attr=0xf, range=[0x000000004f237000-0x000000004f883000) (6MB) [ 0.000000] efi: mem22: type=0, attr=0xf, range=[0x000000004f883000-0x000000005788c000) (128MB) [ 0.000000] efi: mem23: type=3, attr=0xf, range=[0x000000005788c000-0x000000005796e000) (0MB) [ 0.000000] efi: mem24: type=4, attr=0xf, range=[0x000000005796e000-0x000000005b4cf000) (59MB) [ 0.000000] efi: mem25: type=3, attr=0xf, range=[0x000000005b4cf000-0x000000005b8cf000) (4MB) [ 0.000000] efi: mem26: type=7, attr=0xf, range=[0x000000005b8cf000-0x0000000067b64000) (194MB) [ 0.000000] efi: mem27: type=4, attr=0xf, range=[0x0000000067b64000-0x0000000067b71000) (0MB) [ 0.000000] efi: mem28: type=7, attr=0xf, range=[0x0000000067b71000-0x0000000067b75000) (0MB) [ 0.000000] efi: mem29: type=4, attr=0xf, range=[0x0000000067b75000-0x0000000068189000) (6MB) [ 0.000000] efi: mem30: type=7, attr=0xf, range=[0x0000000068189000-0x000000006818a000) (0MB) [ 0.000000] efi: mem31: type=4, attr=0xf, range=[0x000000006818a000-0x000000006819e000) (0MB) [ 0.000000] efi: mem32: type=7, attr=0xf, range=[0x000000006819e000-0x000000006819f000) (0MB) [ 0.000000] efi: mem33: type=4, attr=0xf, range=[0x000000006819f000-0x00000000681a3000) (0MB) [ 0.000000] efi: mem34: type=7, attr=0xf, range=[0x00000000681a3000-0x00000000681a4000) (0MB) [ 0.000000] efi: mem35: type=4, attr=0xf, range=[0x00000000681a4000-0x00000000681b5000) (0MB) [ 0.000000] efi: mem36: type=7, attr=0xf, range=[0x00000000681b5000-0x00000000681b6000) (0MB) [ 0.000000] efi: mem37: type=4, attr=0xf, range=[0x00000000681b6000-0x00000000681b7000) (0MB) [ 0.000000] efi: mem38: type=7, attr=0xf, range=[0x00000000681b7000-0x00000000681b8000) (0MB) [ 0.000000] efi: mem39: type=4, attr=0xf, range=[0x00000000681b8000-0x00000000681c6000) (0MB) [ 0.000000] efi: mem40: type=7, attr=0xf, range=[0x00000000681c6000-0x00000000681c7000) (0MB) [ 0.000000] efi: mem41: type=4, attr=0xf, range=[0x00000000681c7000-0x00000000681d3000) (0MB) [ 0.000000] efi: mem42: type=7, attr=0xf, range=[0x00000000681d3000-0x00000000681d4000) (0MB) [ 0.000000] efi: mem43: type=4, attr=0xf, range=[0x00000000681d4000-0x00000000681d6000) (0MB) [ 0.000000] efi: mem44: type=7, attr=0xf, range=[0x00000000681d6000-0x00000000681d7000) (0MB) [ 0.000000] efi: mem45: type=4, attr=0xf, range=[0x00000000681d7000-0x00000000681e2000) (0MB) [ 0.000000] efi: mem46: type=7, attr=0xf, range=[0x00000000681e2000-0x00000000681e3000) (0MB) [ 0.000000] efi: mem47: type=4, attr=0xf, range=[0x00000000681e3000-0x00000000681e4000) (0MB) [ 0.000000] efi: mem48: type=7, attr=0xf, range=[0x00000000681e4000-0x00000000681e5000) (0MB) [ 0.000000] efi: mem49: type=4, attr=0xf, range=[0x00000000681e5000-0x00000000681ec000) (0MB) [ 0.000000] efi: mem50: type=7, attr=0xf, range=[0x00000000681ec000-0x00000000681ed000) (0MB) [ 0.000000] efi: mem51: type=4, attr=0xf, range=[0x00000000681ed000-0x00000000681fa000) (0MB) [ 0.000000] efi: mem52: type=7, attr=0xf, range=[0x00000000681fa000-0x00000000681fb000) (0MB) [ 0.000000] efi: mem53: type=4, attr=0xf, range=[0x00000000681fb000-0x0000000068203000) (0MB) [ 0.000000] efi: mem54: type=7, attr=0xf, range=[0x0000000068203000-0x0000000068204000) (0MB) [ 0.000000] efi: mem55: type=4, attr=0xf, range=[0x0000000068204000-0x0000000068207000) (0MB) [ 0.000000] efi: mem56: type=7, attr=0xf, range=[0x0000000068207000-0x0000000068208000) (0MB) [ 0.000000] efi: mem57: type=4, attr=0xf, range=[0x0000000068208000-0x0000000068212000) (0MB) [ 0.000000] efi: mem58: type=7, attr=0xf, range=[0x0000000068212000-0x0000000068213000) (0MB) [ 0.000000] efi: mem59: type=4, attr=0xf, range=[0x0000000068213000-0x0000000068533000) (3MB) [ 0.000000] efi: mem60: type=7, attr=0xf, range=[0x0000000068533000-0x0000000068534000) (0MB) [ 0.000000] efi: mem61: type=4, attr=0xf, range=[0x0000000068534000-0x0000000068550000) (0MB) [ 0.000000] efi: mem62: type=7, attr=0xf, range=[0x0000000068550000-0x0000000068551000) (0MB) [ 0.000000] efi: mem63: type=4, attr=0xf, range=[0x0000000068551000-0x0000000068562000) (0MB) [ 0.000000] efi: mem64: type=7, attr=0xf, range=[0x0000000068562000-0x0000000068564000) (0MB) [ 0.000000] efi: mem65: type=4, attr=0xf, range=[0x0000000068564000-0x0000000068572000) (0MB) [ 0.000000] efi: mem66: type=7, attr=0xf, range=[0x0000000068572000-0x0000000068573000) (0MB) [ 0.000000] efi: mem67: type=4, attr=0xf, range=[0x0000000068573000-0x0000000068598000) (0MB) [ 0.000000] efi: mem68: type=7, attr=0xf, range=[0x0000000068598000-0x0000000068599000) (0MB) [ 0.000000] efi: mem69: type=4, attr=0xf, range=[0x0000000068599000-0x00000000685ad000) (0MB) [ 0.000000] efi: mem70: type=7, attr=0xf, range=[0x00000000685ad000-0x00000000685ae000) (0MB) [ 0.000000] efi: mem71: type=4, attr=0xf, range=[0x00000000685ae000-0x000000006860b000) (0MB) [ 0.000000] efi: mem72: type=7, attr=0xf, range=[0x000000006860b000-0x000000006860c000) (0MB) [ 0.000000] efi: mem73: type=4, attr=0xf, range=[0x000000006860c000-0x0000000068613000) (0MB) [ 0.000000] efi: mem74: type=7, attr=0xf, range=[0x0000000068613000-0x0000000068614000) (0MB) [ 0.000000] efi: mem75: type=4, attr=0xf, range=[0x0000000068614000-0x0000000068618000) (0MB) [ 0.000000] efi: mem76: type=7, attr=0xf, range=[0x0000000068618000-0x0000000068619000) (0MB) [ 0.000000] efi: mem77: type=4, attr=0xf, range=[0x0000000068619000-0x000000006862a000) (0MB) [ 0.000000] efi: mem78: type=7, attr=0xf, range=[0x000000006862a000-0x000000006862b000) (0MB) [ 0.000000] efi: mem79: type=4, attr=0xf, range=[0x000000006862b000-0x0000000068644000) (0MB) [ 0.000000] efi: mem80: type=7, attr=0xf, range=[0x0000000068644000-0x0000000068645000) (0MB) [ 0.000000] efi: mem81: type=4, attr=0xf, range=[0x0000000068645000-0x000000006865f000) (0MB) [ 0.000000] efi: mem82: type=7, attr=0xf, range=[0x000000006865f000-0x0000000068660000) (0MB) [ 0.000000] efi: mem83: type=4, attr=0xf, range=[0x0000000068660000-0x00000000686bb000) (0MB) [ 0.000000] efi: mem84: type=7, attr=0xf, range=[0x00000000686bb000-0x00000000686bc000) (0MB) [ 0.000000] efi: mem85: type=4, attr=0xf, range=[0x00000000686bc000-0x00000000686c0000) (0MB) [ 0.000000] efi: mem86: type=7, attr=0xf, range=[0x00000000686c0000-0x00000000686c1000) (0MB) [ 0.000000] efi: mem87: type=4, attr=0xf, range=[0x00000000686c1000-0x00000000686c3000) (0MB) [ 0.000000] efi: mem88: type=7, attr=0xf, range=[0x00000000686c3000-0x00000000686c4000) (0MB) [ 0.000000] efi: mem89: type=4, attr=0xf, range=[0x00000000686c4000-0x00000000686c9000) (0MB) [ 0.000000] efi: mem90: type=7, attr=0xf, range=[0x00000000686c9000-0x00000000686ca000) (0MB) [ 0.000000] efi: mem91: type=4, attr=0xf, range=[0x00000000686ca000-0x00000000686cd000) (0MB) [ 0.000000] efi: mem92: type=7, attr=0xf, range=[0x00000000686cd000-0x00000000686ce000) (0MB) [ 0.000000] efi: mem93: type=4, attr=0xf, range=[0x00000000686ce000-0x00000000686d6000) (0MB) [ 0.000000] efi: mem94: type=7, attr=0xf, range=[0x00000000686d6000-0x00000000686d7000) (0MB) [ 0.000000] efi: mem95: type=4, attr=0xf, range=[0x00000000686d7000-0x0000000068708000) (0MB) [ 0.000000] efi: mem96: type=7, attr=0xf, range=[0x0000000068708000-0x0000000068709000) (0MB) [ 0.000000] efi: mem97: type=4, attr=0xf, range=[0x0000000068709000-0x000000006b8cf000) (49MB) [ 0.000000] efi: mem98: type=7, attr=0xf, range=[0x000000006b8cf000-0x000000006b8d0000) (0MB) [ 0.000000] efi: mem99: type=3, attr=0xf, range=[0x000000006b8d0000-0x000000006cacf000) (17MB) [ 0.000000] efi: mem100: type=6, attr=0x800000000000000f, range=[0x000000006cacf000-0x000000006cbcf000) (1MB) [ 0.000000] efi: mem101: type=5, attr=0x800000000000000f, range=[0x000000006cbcf000-0x000000006cdcf000) (2MB) [ 0.000000] efi: mem102: type=0, attr=0xf, range=[0x000000006cdcf000-0x000000006efcf000) (34MB) [ 0.000000] efi: mem103: type=10, attr=0xf, range=[0x000000006efcf000-0x000000006fdff000) (14MB) [ 0.000000] efi: mem104: type=9, attr=0xf, range=[0x000000006fdff000-0x000000006ffff000) (2MB) [ 0.000000] efi: mem105: type=4, attr=0xf, range=[0x000000006ffff000-0x0000000070000000) (0MB) [ 0.000000] efi: mem106: type=7, attr=0xf, range=[0x0000000100000000-0x000000107f380000) (63475MB) [ 0.000000] efi: mem107: type=7, attr=0xf, range=[0x0000001080000000-0x000000207ff80000) (65535MB) [ 0.000000] efi: mem108: type=7, attr=0xf, range=[0x0000002080000000-0x000000307ff80000) (65535MB) [ 0.000000] efi: mem109: type=7, attr=0xf, range=[0x0000003080000000-0x000000407ff80000) (65535MB) [ 0.000000] efi: mem110: type=0, attr=0x9, range=[0x0000000070000000-0x0000000080000000) (256MB) [ 0.000000] efi: mem111: type=11, attr=0x800000000000000f, range=[0x0000000080000000-0x0000000090000000) (256MB) [ 0.000000] efi: mem112: type=11, attr=0x800000000000000f, range=[0x00000000fec10000-0x00000000fec11000) (0MB) [ 0.000000] efi: mem113: type=11, attr=0x800000000000000f, range=[0x00000000fed80000-0x00000000fed81000) (0MB) [ 0.000000] efi: mem114: type=0, attr=0x0, range=[0x000000107f380000-0x0000001080000000) (12MB) [ 0.000000] efi: mem115: type=0, attr=0x0, range=[0x000000207ff80000-0x0000002080000000) (0MB) [ 0.000000] efi: mem116: type=0, attr=0x0, range=[0x000000307ff80000-0x0000003080000000) (0MB) [ 0.000000] efi: mem117: type=0, attr=0x0, range=[0x000000407ff80000-0x0000004080000000) (0MB) [ 0.000000] SMBIOS 3.2.0 present. [ 0.000000] DMI: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.10.6 08/15/2019 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x407ff80 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-FFFFF uncachable [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000FF000000 mask FFFFFF000000 write-protect [ 0.000000] 1 base 000000000000 mask FFFF80000000 write-back [ 0.000000] 2 base 000070000000 mask FFFFF0000000 uncachable [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] TOM2: 0000004080000000 aka 264192M [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0x70000 max_arch_pfn = 0x400000000 [ 0.000000] Base memory trampoline at [ffff8acd40099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x17d4e53000, 0x17d4e53fff] PGTABLE [ 0.000000] BRK [0x17d4e54000, 0x17d4e54fff] PGTABLE [ 0.000000] BRK [0x17d4e55000, 0x17d4e55fff] PGTABLE [ 0.000000] BRK [0x17d4e56000, 0x17d4e56fff] PGTABLE [ 0.000000] BRK [0x17d4e57000, 0x17d4e57fff] PGTABLE [ 0.000000] BRK [0x17d4e58000, 0x17d4e58fff] PGTABLE [ 0.000000] BRK [0x17d4e59000, 0x17d4e59fff] PGTABLE [ 0.000000] BRK [0x17d4e5a000, 0x17d4e5afff] PGTABLE [ 0.000000] BRK [0x17d4e5b000, 0x17d4e5bfff] PGTABLE [ 0.000000] BRK [0x17d4e5c000, 0x17d4e5cfff] PGTABLE [ 0.000000] BRK [0x17d4e5d000, 0x17d4e5dfff] PGTABLE [ 0.000000] BRK [0x17d4e5e000, 0x17d4e5efff] PGTABLE [ 0.000000] RAMDISK: [mem 0x370ae000-0x383d1fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 000000006fffe014 00024 (v02 DELL ) [ 0.000000] ACPI: XSDT 000000006fffd0e8 000AC (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: FACP 000000006fff0000 00114 (v06 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: DSDT 000000006ffdc000 1038C (v02 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: FACS 000000006fdd3000 00040 [ 0.000000] ACPI: SSDT 000000006fffc000 000D2 (v02 DELL PE_SC3 00000002 MSFT 04000000) [ 0.000000] ACPI: BERT 000000006fffb000 00030 (v01 DELL BERT 00000001 DELL 00000001) [ 0.000000] ACPI: HEST 000000006fffa000 006DC (v01 DELL HEST 00000001 DELL 00000001) [ 0.000000] ACPI: SSDT 000000006fff9000 00294 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: SRAT 000000006fff8000 00420 (v03 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: MSCT 000000006fff7000 0004E (v01 DELL PE_SC3 00000000 AMD 00000001) [ 0.000000] ACPI: SLIT 000000006fff6000 0003C (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: CRAT 000000006fff3000 02DC0 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: EINJ 000000006fff2000 00150 (v01 DELL PE_SC3 00000001 AMD 00000001) [ 0.000000] ACPI: SLIC 000000006fff1000 00024 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: HPET 000000006ffef000 00038 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: APIC 000000006ffee000 004B2 (v03 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: MCFG 000000006ffed000 0003C (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: SSDT 000000006ffdb000 00629 (v02 DELL xhc_port 00000001 INTL 20170119) [ 0.000000] ACPI: IVRS 000000006ffda000 00210 (v02 DELL PE_SC3 00000001 AMD 00000000) [ 0.000000] ACPI: SSDT 000000006ffd8000 01658 (v01 AMD CPMCMN 00000001 INTL 20170119) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0d -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x10 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x11 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x12 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x13 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x14 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x15 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x18 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x19 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1a -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1b -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1c -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x1d -> Node 1 [ 0.000000] SRAT: PXM 2 -> APIC 0x20 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x21 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x22 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x23 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x24 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x25 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x28 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x29 -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2a -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2b -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2c -> Node 2 [ 0.000000] SRAT: PXM 2 -> APIC 0x2d -> Node 2 [ 0.000000] SRAT: PXM 3 -> APIC 0x30 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x31 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x32 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x33 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x34 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x35 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x38 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x39 -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3a -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3b -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3c -> Node 3 [ 0.000000] SRAT: PXM 3 -> APIC 0x3d -> Node 3 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0x7fffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] SRAT: Node 2 PXM 2 [mem 0x2080000000-0x307fffffff] [ 0.000000] SRAT: Node 3 PXM 3 [mem 0x3080000000-0x407fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=4 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x107f359000-0x107f37ffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x207ff59000-0x207ff7ffff] [ 0.000000] NODE_DATA(2) allocated [mem 0x307ff59000-0x307ff7ffff] [ 0.000000] NODE_DATA(3) allocated [mem 0x407ff58000-0x407ff7efff] [ 0.000000] Reserving 176MB of memory at 704MB for crashkernel (System RAM: 261692MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x407ff7ffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0008efff] [ 0.000000] node 0: [mem 0x00090000-0x0009ffff] [ 0.000000] node 0: [mem 0x00100000-0x4f882fff] [ 0.000000] node 0: [mem 0x5788c000-0x6cacefff] [ 0.000000] node 0: [mem 0x6ffff000-0x6fffffff] [ 0.000000] node 0: [mem 0x100000000-0x107f37ffff] [ 0.000000] node 1: [mem 0x1080000000-0x207ff7ffff] [ 0.000000] node 2: [mem 0x2080000000-0x307ff7ffff] [ 0.000000] node 3: [mem 0x3080000000-0x407ff7ffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x107f37ffff] [ 0.000000] On node 0 totalpages: 16661989 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 1126 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 6380 pages used for memmap [ 0.000000] DMA32 zone: 408263 pages, LIFO batch:31 [ 0.000000] Normal zone: 253902 pages used for memmap [ 0.000000] Normal zone: 16249728 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x1080000000-0x207ff7ffff] [ 0.000000] On node 1 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] Initmem setup node 2 [mem 0x2080000000-0x307ff7ffff] [ 0.000000] On node 2 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] Initmem setup node 3 [mem 0x3080000000-0x407ff7ffff] [ 0.000000] On node 3 totalpages: 16777088 [ 0.000000] Normal zone: 262142 pages used for memmap [ 0.000000] Normal zone: 16777088 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x30] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x18] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x38] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x34] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x0c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x1c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x3c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x31] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x19] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x39] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x33] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x1b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x2b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x3b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x35] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x0d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x1d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x3d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x58] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x59] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x78] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x79] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7a] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7b] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7c] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7d] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7e] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x7f] lapic_id[0x00] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x80] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 128, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x81] address[0xfd880000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 129, version 33, address 0xfd880000, GSI 24-55 [ 0.000000] ACPI: IOAPIC (id[0x82] address[0xe0900000] gsi_base[56]) [ 0.000000] IOAPIC[2]: apic_id 130, version 33, address 0xe0900000, GSI 56-87 [ 0.000000] ACPI: IOAPIC (id[0x83] address[0xc5900000] gsi_base[88]) [ 0.000000] IOAPIC[3]: apic_id 131, version 33, address 0xc5900000, GSI 88-119 [ 0.000000] ACPI: IOAPIC (id[0x84] address[0xaa900000] gsi_base[120]) [ 0.000000] IOAPIC[4]: apic_id 132, version 33, address 0xaa900000, GSI 120-151 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x10228201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 128 CPUs, 80 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0008f000-0x0008ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x37028000-0x37028fff] [ 0.000000] PM: Registered nosave memory: [mem 0x37040000-0x37040fff] [ 0.000000] PM: Registered nosave memory: [mem 0x37041000-0x37041fff] [ 0.000000] PM: Registered nosave memory: [mem 0x37049000-0x37049fff] [ 0.000000] PM: Registered nosave memory: [mem 0x3704a000-0x3704afff] [ 0.000000] PM: Registered nosave memory: [mem 0x3707b000-0x3707bfff] [ 0.000000] PM: Registered nosave memory: [mem 0x3707c000-0x3707cfff] [ 0.000000] PM: Registered nosave memory: [mem 0x370ad000-0x370adfff] [ 0.000000] PM: Registered nosave memory: [mem 0x4f883000-0x5788bfff] [ 0.000000] PM: Registered nosave memory: [mem 0x6cacf000-0x6efcefff] [ 0.000000] PM: Registered nosave memory: [mem 0x6efcf000-0x6fdfefff] [ 0.000000] PM: Registered nosave memory: [mem 0x6fdff000-0x6fffefff] [ 0.000000] PM: Registered nosave memory: [mem 0x70000000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfec0ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec10000-0xfec10fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec11000-0xfed7ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed80000-0xfed80fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed81000-0xffffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x107f380000-0x107fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x207ff80000-0x207fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x307ff80000-0x307fffffff] [ 0.000000] e820: [mem 0x90000000-0xfec0ffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:4 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8add7ee00000 s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 004 008 012 016 020 024 028 [ 0.000000] pcpu-alloc: [0] 032 036 040 044 048 052 056 060 [ 0.000000] pcpu-alloc: [0] 064 068 072 076 080 084 088 092 [ 0.000000] pcpu-alloc: [0] 096 100 104 108 112 116 120 124 [ 0.000000] pcpu-alloc: [1] 001 005 009 013 017 021 025 029 [ 0.000000] pcpu-alloc: [1] 033 037 041 045 049 053 057 061 [ 0.000000] pcpu-alloc: [1] 065 069 073 077 081 085 089 093 [ 0.000000] pcpu-alloc: [1] 097 101 105 109 113 117 121 125 [ 0.000000] pcpu-alloc: [2] 002 006 010 014 018 022 026 030 [ 0.000000] pcpu-alloc: [2] 034 038 042 046 050 054 058 062 [ 0.000000] pcpu-alloc: [2] 066 070 074 078 082 086 090 094 [ 0.000000] pcpu-alloc: [2] 098 102 106 110 114 118 122 126 [ 0.000000] pcpu-alloc: [3] 003 007 011 015 019 023 027 031 [ 0.000000] pcpu-alloc: [3] 035 039 043 047 051 055 059 063 [ 0.000000] pcpu-alloc: [3] 067 071 075 079 083 087 091 095 [ 0.000000] pcpu-alloc: [3] 099 103 107 111 115 119 123 127 [ 0.000000] Built 4 zonelists in Zone order, mobility grouping on. Total pages: 65945355 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl2.x86_64 root=UUID=1a559904-7ee3-4f19-99ef-40792627f881 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 9613556k/270532096k available (7676k kernel code, 2559084k absent, 4654404k reserved, 6045k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=4 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=128. [ 0.000000] NR_IRQS:327936 nr_irqs:3624 0 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 1072693248 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 1996.202 MHz processor [ 0.000056] Calibrating delay loop (skipped), value calculated using timer frequency.. 3992.40 BogoMIPS (lpj=1996202) [ 0.010704] pid_max: default: 131072 minimum: 1024 [ 0.016183] Security Framework initialized [ 0.020302] SELinux: Initializing. [ 0.023864] SELinux: Starting in permissive mode [ 0.023866] Yama: becoming mindful. [ 0.044104] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.100151] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.127889] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.135280] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.144431] Initializing cgroup subsys memory [ 0.148826] Initializing cgroup subsys devices [ 0.153282] Initializing cgroup subsys freezer [ 0.157738] Initializing cgroup subsys net_cls [ 0.162192] Initializing cgroup subsys blkio [ 0.166474] Initializing cgroup subsys perf_event [ 0.171197] Initializing cgroup subsys hugetlb [ 0.175652] Initializing cgroup subsys pids [ 0.179847] Initializing cgroup subsys net_prio [ 0.184473] tseg: 0070000000 [ 0.190087] LVT offset 2 assigned for vector 0xf4 [ 0.194819] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 [ 0.200838] Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768 [ 0.206853] tlb_flushall_shift: 6 [ 0.210202] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.219774] FEATURE SPEC_CTRL Not Present [ 0.223796] FEATURE IBPB_SUPPORT Present [ 0.227733] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.234169] Spectre V2 : Mitigation: Full retpoline [ 0.239730] Freeing SMP alternatives: 28k freed [ 0.246195] ACPI: Core revision 20130517 [ 0.254906] ACPI: All ACPI Tables successfully acquired [ 0.265252] ftrace: allocating 29216 entries in 115 pages [ 0.605889] Switched APIC routing to physical flat. [ 0.612820] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.628825] smpboot: CPU0: AMD EPYC 7401P 24-Core Processor (fam: 17, model: 01, stepping: 02) [ 0.713404] random: fast init done [ 0.741405] APIC calibration not consistent with PM-Timer: 101ms instead of 100ms [ 0.748885] APIC delta adjusted to PM-Timer: 623828 (636296) [ 0.754579] Performance Events: Fam17h core perfctr, AMD PMU driver. [ 0.761014] ... version: 0 [ 0.765025] ... bit width: 48 [ 0.769124] ... generic registers: 6 [ 0.773138] ... value mask: 0000ffffffffffff [ 0.778450] ... max period: 00007fffffffffff [ 0.783764] ... fixed-purpose events: 0 [ 0.787777] ... event mask: 000000000000003f [ 0.796137] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.804222] smpboot: Booting Node 1, Processors #1 OK [ 0.817441] smpboot: Booting Node 2, Processors #2 OK [ 0.830645] smpboot: Booting Node 3, Processors #3 OK [ 0.843835] smpboot: Booting Node 0, Processors #4 OK [ 0.857017] smpboot: Booting Node 1, Processors #5 OK [ 0.870206] smpboot: Booting Node 2, Processors #6 OK [ 0.883397] smpboot: Booting Node 3, Processors #7 OK [ 0.896575] smpboot: Booting Node 0, Processors #8 OK [ 0.909965] smpboot: Booting Node 1, Processors #9 OK [ 0.923162] smpboot: Booting Node 2, Processors #10 OK [ 0.936437] smpboot: Booting Node 3, Processors #11 OK [ 0.949710] smpboot: Booting Node 0, Processors #12 OK [ 0.962979] smpboot: Booting Node 1, Processors #13 OK [ 0.976253] smpboot: Booting Node 2, Processors #14 OK [ 0.989531] smpboot: Booting Node 3, Processors #15 OK [ 1.002805] smpboot: Booting Node 0, Processors #16 OK [ 1.016181] smpboot: Booting Node 1, Processors #17 OK [ 1.029470] smpboot: Booting Node 2, Processors #18 OK [ 1.042749] smpboot: Booting Node 3, Processors #19 OK [ 1.056027] smpboot: Booting Node 0, Processors #20 OK [ 1.069293] smpboot: Booting Node 1, Processors #21 OK [ 1.082572] smpboot: Booting Node 2, Processors #22 OK [ 1.095853] smpboot: Booting Node 3, Processors #23 OK [ 1.109119] smpboot: Booting Node 0, Processors #24 OK [ 1.122860] smpboot: Booting Node 1, Processors #25 OK [ 1.136106] smpboot: Booting Node 2, Processors #26 OK [ 1.149351] smpboot: Booting Node 3, Processors #27 OK [ 1.162596] smpboot: Booting Node 0, Processors #28 OK [ 1.175831] smpboot: Booting Node 1, Processors #29 OK [ 1.189070] smpboot: Booting Node 2, Processors #30 OK [ 1.202314] smpboot: Booting Node 3, Processors #31 OK [ 1.215538] smpboot: Booting Node 0, Processors #32 OK [ 1.228872] smpboot: Booting Node 1, Processors #33 OK [ 1.242111] smpboot: Booting Node 2, Processors #34 OK [ 1.255345] smpboot: Booting Node 3, Processors #35 OK [ 1.268582] smpboot: Booting Node 0, Processors #36 OK [ 1.281810] smpboot: Booting Node 1, Processors #37 OK [ 1.295052] smpboot: Booting Node 2, Processors #38 OK [ 1.308406] smpboot: Booting Node 3, Processors #39 OK [ 1.321643] smpboot: Booting Node 0, Processors #40 OK [ 1.334981] smpboot: Booting Node 1, Processors #41 OK [ 1.348221] smpboot: Booting Node 2, Processors #42 OK [ 1.361465] smpboot: Booting Node 3, Processors #43 OK [ 1.374700] smpboot: Booting Node 0, Processors #44 OK [ 1.388030] smpboot: Booting Node 1, Processors #45 OK [ 1.401263] smpboot: Booting Node 2, Processors #46 OK [ 1.414609] smpboot: Booting Node 3, Processors #47 [ 1.427324] Brought up 48 CPUs [ 1.430587] smpboot: Max logical packages: 3 [ 1.434862] smpboot: Total of 48 processors activated (191635.39 BogoMIPS) [ 1.721868] node 0 initialised, 15462980 pages in 273ms [ 1.731501] node 3 initialised, 15989250 pages in 278ms [ 1.732035] node 2 initialised, 15989367 pages in 279ms [ 1.736827] node 1 initialised, 15984664 pages in 284ms [ 1.747728] devtmpfs: initialized [ 1.773506] EVM: security.selinux [ 1.776829] EVM: security.ima [ 1.779802] EVM: security.capability [ 1.783478] PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes) [ 1.791217] PM: Registering ACPI NVS region [mem 0x6efcf000-0x6fdfefff] (14876672 bytes) [ 1.800883] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.807763] pinctrl core: initialized pinctrl subsystem [ 1.813092] RTC time: 17:20:05, date: 03/18/20 [ 1.817693] NET: Registered protocol family 16 [ 1.822491] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.830061] ACPI: bus type PCI registered [ 1.834075] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.840659] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.849962] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.856752] PCI: Using configuration type 1 for base access [ 1.862338] PCI: Dell System detected, enabling pci=bfsort. [ 1.877086] ACPI: Added _OSI(Module Device) [ 1.881275] ACPI: Added _OSI(Processor Device) [ 1.885719] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.890425] ACPI: Added _OSI(Processor Aggregator Device) [ 1.895825] ACPI: Added _OSI(Linux-Dell-Video) [ 1.901090] ACPI: EC: Look up EC in DSDT [ 1.902071] ACPI: Executed 2 blocks of module-level executable AML code [ 1.914130] ACPI: Interpreter enabled [ 1.917804] ACPI: (supports S0 S5) [ 1.921210] ACPI: Using IOAPIC for interrupt routing [ 1.926389] HEST: Table parsing has been initialized. [ 1.931447] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.940597] ACPI: Enabled 1 GPEs in block 00 to 1F [ 1.952262] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0 [ 1.959174] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0 [ 1.966078] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0 [ 1.972987] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0 [ 1.979893] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0 [ 1.986801] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0 [ 1.993710] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0 [ 2.000616] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0 [ 2.007666] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-3f]) [ 2.013848] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.022066] acpi PNP0A08:00: PCIe AER handled by firmware [ 2.027509] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] [ 2.034455] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.042104] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.050566] PCI host bridge to bus 0000:00 [ 2.054665] pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] [ 2.061451] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] [ 2.068237] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] [ 2.075716] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] [ 2.083195] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] [ 2.090674] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] [ 2.098153] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window] [ 2.105635] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] [ 2.113114] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff window] [ 2.120593] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] [ 2.128073] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] [ 2.135552] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] [ 2.143031] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] [ 2.150511] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] [ 2.157990] pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] [ 2.165469] pci_bus 0000:00: root bus resource [io 0x0d00-0x3fff window] [ 2.172256] pci_bus 0000:00: root bus resource [mem 0xe1000000-0xfebfffff window] [ 2.179736] pci_bus 0000:00: root bus resource [mem 0x10000000000-0x2bf3fffffff window] [ 2.187736] pci_bus 0000:00: root bus resource [bus 00-3f] [ 2.193230] pci 0000:00:00.0: [1022:1450] type 00 class 0x060000 [ 2.193312] pci 0000:00:00.2: [1022:1451] type 00 class 0x080600 [ 2.193401] pci 0000:00:01.0: [1022:1452] type 00 class 0x060000 [ 2.193478] pci 0000:00:02.0: [1022:1452] type 00 class 0x060000 [ 2.193553] pci 0000:00:03.0: [1022:1452] type 00 class 0x060000 [ 2.193615] pci 0000:00:03.1: [1022:1453] type 01 class 0x060400 [ 2.193735] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold [ 2.193834] pci 0000:00:04.0: [1022:1452] type 00 class 0x060000 [ 2.193915] pci 0000:00:07.0: [1022:1452] type 00 class 0x060000 [ 2.193974] pci 0000:00:07.1: [1022:1454] type 01 class 0x060400 [ 2.194742] pci 0000:00:07.1: PME# supported from D0 D3hot D3cold [ 2.194822] pci 0000:00:08.0: [1022:1452] type 00 class 0x060000 [ 2.194883] pci 0000:00:08.1: [1022:1454] type 01 class 0x060400 [ 2.195712] pci 0000:00:08.1: PME# supported from D0 D3hot D3cold [ 2.195827] pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 [ 2.196026] pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 [ 2.196230] pci 0000:00:18.0: [1022:1460] type 00 class 0x060000 [ 2.196280] pci 0000:00:18.1: [1022:1461] type 00 class 0x060000 [ 2.196331] pci 0000:00:18.2: [1022:1462] type 00 class 0x060000 [ 2.196382] pci 0000:00:18.3: [1022:1463] type 00 class 0x060000 [ 2.196433] pci 0000:00:18.4: [1022:1464] type 00 class 0x060000 [ 2.196484] pci 0000:00:18.5: [1022:1465] type 00 class 0x060000 [ 2.196534] pci 0000:00:18.6: [1022:1466] type 00 class 0x060000 [ 2.196584] pci 0000:00:18.7: [1022:1467] type 00 class 0x060000 [ 2.196634] pci 0000:00:19.0: [1022:1460] type 00 class 0x060000 [ 2.196689] pci 0000:00:19.1: [1022:1461] type 00 class 0x060000 [ 2.196744] pci 0000:00:19.2: [1022:1462] type 00 class 0x060000 [ 2.196799] pci 0000:00:19.3: [1022:1463] type 00 class 0x060000 [ 2.196850] pci 0000:00:19.4: [1022:1464] type 00 class 0x060000 [ 2.196903] pci 0000:00:19.5: [1022:1465] type 00 class 0x060000 [ 2.196956] pci 0000:00:19.6: [1022:1466] type 00 class 0x060000 [ 2.197010] pci 0000:00:19.7: [1022:1467] type 00 class 0x060000 [ 2.197062] pci 0000:00:1a.0: [1022:1460] type 00 class 0x060000 [ 2.197116] pci 0000:00:1a.1: [1022:1461] type 00 class 0x060000 [ 2.197169] pci 0000:00:1a.2: [1022:1462] type 00 class 0x060000 [ 2.197223] pci 0000:00:1a.3: [1022:1463] type 00 class 0x060000 [ 2.197274] pci 0000:00:1a.4: [1022:1464] type 00 class 0x060000 [ 2.197327] pci 0000:00:1a.5: [1022:1465] type 00 class 0x060000 [ 2.197381] pci 0000:00:1a.6: [1022:1466] type 00 class 0x060000 [ 2.197435] pci 0000:00:1a.7: [1022:1467] type 00 class 0x060000 [ 2.197490] pci 0000:00:1b.0: [1022:1460] type 00 class 0x060000 [ 2.197545] pci 0000:00:1b.1: [1022:1461] type 00 class 0x060000 [ 2.197599] pci 0000:00:1b.2: [1022:1462] type 00 class 0x060000 [ 2.197655] pci 0000:00:1b.3: [1022:1463] type 00 class 0x060000 [ 2.197710] pci 0000:00:1b.4: [1022:1464] type 00 class 0x060000 [ 2.197763] pci 0000:00:1b.5: [1022:1465] type 00 class 0x060000 [ 2.197817] pci 0000:00:1b.6: [1022:1466] type 00 class 0x060000 [ 2.197870] pci 0000:00:1b.7: [1022:1467] type 00 class 0x060000 [ 2.198748] pci 0000:01:00.0: [15b3:101b] type 00 class 0x020700 [ 2.198892] pci 0000:01:00.0: reg 0x10: [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.199127] pci 0000:01:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.199534] pci 0000:01:00.0: PME# supported from D3cold [ 2.199813] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.204789] pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.204872] pci 0000:02:00.0: [1022:145a] type 00 class 0x130000 [ 2.204971] pci 0000:02:00.2: [1022:1456] type 00 class 0x108000 [ 2.204988] pci 0000:02:00.2: reg 0x18: [mem 0xf7300000-0xf73fffff] [ 2.205000] pci 0000:02:00.2: reg 0x24: [mem 0xf7400000-0xf7401fff] [ 2.205079] pci 0000:02:00.3: [1022:145f] type 00 class 0x0c0330 [ 2.205091] pci 0000:02:00.3: reg 0x10: [mem 0xf7200000-0xf72fffff 64bit] [ 2.205140] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold [ 2.205199] pci 0000:00:07.1: PCI bridge to [bus 02] [ 2.210170] pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] [ 2.210758] pci 0000:03:00.0: [1022:1455] type 00 class 0x130000 [ 2.210867] pci 0000:03:00.1: [1022:1468] type 00 class 0x108000 [ 2.210885] pci 0000:03:00.1: reg 0x18: [mem 0xf7000000-0xf70fffff] [ 2.210899] pci 0000:03:00.1: reg 0x24: [mem 0xf7100000-0xf7101fff] [ 2.210990] pci 0000:00:08.1: PCI bridge to [bus 03] [ 2.215959] pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] [ 2.215975] pci_bus 0000:00: on NUMA node 0 [ 2.216347] ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 40-7f]) [ 2.222526] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.230737] acpi PNP0A08:01: PCIe AER handled by firmware [ 2.236180] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug] [ 2.243128] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.250778] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.259191] PCI host bridge to bus 0000:40 [ 2.263294] pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] [ 2.270081] pci_bus 0000:40: root bus resource [mem 0xc6000000-0xe0ffffff window] [ 2.277559] pci_bus 0000:40: root bus resource [mem 0x2bf40000000-0x47e7fffffff window] [ 2.285559] pci_bus 0000:40: root bus resource [bus 40-7f] [ 2.291050] pci 0000:40:00.0: [1022:1450] type 00 class 0x060000 [ 2.291121] pci 0000:40:00.2: [1022:1451] type 00 class 0x080600 [ 2.291212] pci 0000:40:01.0: [1022:1452] type 00 class 0x060000 [ 2.291285] pci 0000:40:02.0: [1022:1452] type 00 class 0x060000 [ 2.291360] pci 0000:40:03.0: [1022:1452] type 00 class 0x060000 [ 2.291433] pci 0000:40:04.0: [1022:1452] type 00 class 0x060000 [ 2.291513] pci 0000:40:07.0: [1022:1452] type 00 class 0x060000 [ 2.291573] pci 0000:40:07.1: [1022:1454] type 01 class 0x060400 [ 2.292152] pci 0000:40:07.1: PME# supported from D0 D3hot D3cold [ 2.292231] pci 0000:40:08.0: [1022:1452] type 00 class 0x060000 [ 2.292293] pci 0000:40:08.1: [1022:1454] type 01 class 0x060400 [ 2.292405] pci 0000:40:08.1: PME# supported from D0 D3hot D3cold [ 2.293097] pci 0000:41:00.0: [1022:145a] type 00 class 0x130000 [ 2.293204] pci 0000:41:00.2: [1022:1456] type 00 class 0x108000 [ 2.293224] pci 0000:41:00.2: reg 0x18: [mem 0xdb300000-0xdb3fffff] [ 2.293237] pci 0000:41:00.2: reg 0x24: [mem 0xdb400000-0xdb401fff] [ 2.293319] pci 0000:41:00.3: [1022:145f] type 00 class 0x0c0330 [ 2.293333] pci 0000:41:00.3: reg 0x10: [mem 0xdb200000-0xdb2fffff 64bit] [ 2.293387] pci 0000:41:00.3: PME# supported from D0 D3hot D3cold [ 2.293451] pci 0000:40:07.1: PCI bridge to [bus 41] [ 2.298424] pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] [ 2.298522] pci 0000:42:00.0: [1022:1455] type 00 class 0x130000 [ 2.298640] pci 0000:42:00.1: [1022:1468] type 00 class 0x108000 [ 2.298660] pci 0000:42:00.1: reg 0x18: [mem 0xdb000000-0xdb0fffff] [ 2.298674] pci 0000:42:00.1: reg 0x24: [mem 0xdb100000-0xdb101fff] [ 2.298774] pci 0000:40:08.1: PCI bridge to [bus 42] [ 2.303748] pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] [ 2.303761] pci_bus 0000:40: on NUMA node 1 [ 2.303936] ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 80-bf]) [ 2.310124] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.318335] acpi PNP0A08:02: PCIe AER handled by firmware [ 2.323777] acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug] [ 2.330723] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.338374] acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.346814] PCI host bridge to bus 0000:80 [ 2.350917] pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] [ 2.357703] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] [ 2.365182] pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] [ 2.371968] pci_bus 0000:80: root bus resource [mem 0xab000000-0xc5ffffff window] [ 2.379447] pci_bus 0000:80: root bus resource [mem 0x47e80000000-0x63dbfffffff window] [ 2.387448] pci_bus 0000:80: root bus resource [bus 80-bf] [ 2.392938] pci 0000:80:00.0: [1022:1450] type 00 class 0x060000 [ 2.393008] pci 0000:80:00.2: [1022:1451] type 00 class 0x080600 [ 2.393096] pci 0000:80:01.0: [1022:1452] type 00 class 0x060000 [ 2.393158] pci 0000:80:01.1: [1022:1453] type 01 class 0x060400 [ 2.393776] pci 0000:80:01.1: PME# supported from D0 D3hot D3cold [ 2.393849] pci 0000:80:01.2: [1022:1453] type 01 class 0x060400 [ 2.393977] pci 0000:80:01.2: PME# supported from D0 D3hot D3cold [ 2.394057] pci 0000:80:02.0: [1022:1452] type 00 class 0x060000 [ 2.394131] pci 0000:80:03.0: [1022:1452] type 00 class 0x060000 [ 2.394191] pci 0000:80:03.1: [1022:1453] type 01 class 0x060400 [ 2.394780] pci 0000:80:03.1: PME# supported from D0 D3hot D3cold [ 2.394876] pci 0000:80:04.0: [1022:1452] type 00 class 0x060000 [ 2.394957] pci 0000:80:07.0: [1022:1452] type 00 class 0x060000 [ 2.395019] pci 0000:80:07.1: [1022:1454] type 01 class 0x060400 [ 2.395129] pci 0000:80:07.1: PME# supported from D0 D3hot D3cold [ 2.395205] pci 0000:80:08.0: [1022:1452] type 00 class 0x060000 [ 2.395266] pci 0000:80:08.1: [1022:1454] type 01 class 0x060400 [ 2.395774] pci 0000:80:08.1: PME# supported from D0 D3hot D3cold [ 2.395986] pci 0000:81:00.0: [14e4:165f] type 00 class 0x020000 [ 2.396012] pci 0000:81:00.0: reg 0x10: [mem 0xac230000-0xac23ffff 64bit pref] [ 2.396027] pci 0000:81:00.0: reg 0x18: [mem 0xac240000-0xac24ffff 64bit pref] [ 2.396042] pci 0000:81:00.0: reg 0x20: [mem 0xac250000-0xac25ffff 64bit pref] [ 2.396052] pci 0000:81:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 2.396129] pci 0000:81:00.0: PME# supported from D0 D3hot D3cold [ 2.396221] pci 0000:81:00.1: [14e4:165f] type 00 class 0x020000 [ 2.396247] pci 0000:81:00.1: reg 0x10: [mem 0xac200000-0xac20ffff 64bit pref] [ 2.396262] pci 0000:81:00.1: reg 0x18: [mem 0xac210000-0xac21ffff 64bit pref] [ 2.396276] pci 0000:81:00.1: reg 0x20: [mem 0xac220000-0xac22ffff 64bit pref] [ 2.396286] pci 0000:81:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 2.396362] pci 0000:81:00.1: PME# supported from D0 D3hot D3cold [ 2.396449] pci 0000:80:01.1: PCI bridge to [bus 81] [ 2.401424] pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] [ 2.401754] pci 0000:82:00.0: [1556:be00] type 01 class 0x060400 [ 2.404701] pci 0000:80:01.2: PCI bridge to [bus 82-83] [ 2.409936] pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] [ 2.409940] pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.409988] pci 0000:83:00.0: [102b:0536] type 00 class 0x030000 [ 2.410006] pci 0000:83:00.0: reg 0x10: [mem 0xab000000-0xabffffff pref] [ 2.410017] pci 0000:83:00.0: reg 0x14: [mem 0xc0808000-0xc080bfff] [ 2.410029] pci 0000:83:00.0: reg 0x18: [mem 0xc0000000-0xc07fffff] [ 2.410169] pci 0000:82:00.0: PCI bridge to [bus 83] [ 2.415146] pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] [ 2.415153] pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.415236] pci 0000:84:00.0: [1000:00d1] type 00 class 0x010700 [ 2.415259] pci 0000:84:00.0: reg 0x10: [mem 0xac000000-0xac0fffff 64bit pref] [ 2.415269] pci 0000:84:00.0: reg 0x18: [mem 0xac100000-0xac1fffff 64bit pref] [ 2.415277] pci 0000:84:00.0: reg 0x20: [mem 0xc0d00000-0xc0dfffff] [ 2.415284] pci 0000:84:00.0: reg 0x24: [io 0x8000-0x80ff] [ 2.415293] pci 0000:84:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 2.415344] pci 0000:84:00.0: supports D1 D2 [ 2.417698] pci 0000:80:03.1: PCI bridge to [bus 84] [ 2.422664] pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] [ 2.422667] pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] [ 2.422671] pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] [ 2.422800] pci 0000:85:00.0: [1022:145a] type 00 class 0x130000 [ 2.422905] pci 0000:85:00.2: [1022:1456] type 00 class 0x108000 [ 2.422924] pci 0000:85:00.2: reg 0x18: [mem 0xc0b00000-0xc0bfffff] [ 2.422937] pci 0000:85:00.2: reg 0x24: [mem 0xc0c00000-0xc0c01fff] [ 2.423028] pci 0000:80:07.1: PCI bridge to [bus 85] [ 2.427994] pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] [ 2.428089] pci 0000:86:00.0: [1022:1455] type 00 class 0x130000 [ 2.428206] pci 0000:86:00.1: [1022:1468] type 00 class 0x108000 [ 2.428227] pci 0000:86:00.1: reg 0x18: [mem 0xc0900000-0xc09fffff] [ 2.428241] pci 0000:86:00.1: reg 0x24: [mem 0xc0a00000-0xc0a01fff] [ 2.428328] pci 0000:86:00.2: [1022:7901] type 00 class 0x010601 [ 2.428360] pci 0000:86:00.2: reg 0x24: [mem 0xc0a02000-0xc0a02fff] [ 2.428399] pci 0000:86:00.2: PME# supported from D3hot D3cold [ 2.428465] pci 0000:80:08.1: PCI bridge to [bus 86] [ 2.433439] pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] [ 2.433465] pci_bus 0000:80: on NUMA node 2 [ 2.433636] ACPI: PCI Root Bridge [PC03] (domain 0000 [bus c0-ff]) [ 2.439816] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.448025] acpi PNP0A08:03: PCIe AER handled by firmware [ 2.453467] acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug] [ 2.460415] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 2.468067] acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.476394] acpi PNP0A08:03: host bridge window [mem 0x63dc0000000-0xffffffffffff window] ([0x80000000000-0xffffffffffff] ignored, not CPU addressable) [ 2.490032] PCI host bridge to bus 0000:c0 [ 2.494129] pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] [ 2.500916] pci_bus 0000:c0: root bus resource [mem 0x90000000-0xaaffffff window] [ 2.508396] pci_bus 0000:c0: root bus resource [mem 0x63dc0000000-0x7ffffffffff window] [ 2.516393] pci_bus 0000:c0: root bus resource [bus c0-ff] [ 2.521884] pci 0000:c0:00.0: [1022:1450] type 00 class 0x060000 [ 2.521956] pci 0000:c0:00.2: [1022:1451] type 00 class 0x080600 [ 2.522044] pci 0000:c0:01.0: [1022:1452] type 00 class 0x060000 [ 2.522105] pci 0000:c0:01.1: [1022:1453] type 01 class 0x060400 [ 2.522236] pci 0000:c0:01.1: PME# supported from D0 D3hot D3cold [ 2.522330] pci 0000:c0:02.0: [1022:1452] type 00 class 0x060000 [ 2.522405] pci 0000:c0:03.0: [1022:1452] type 00 class 0x060000 [ 2.522479] pci 0000:c0:04.0: [1022:1452] type 00 class 0x060000 [ 2.522558] pci 0000:c0:07.0: [1022:1452] type 00 class 0x060000 [ 2.522617] pci 0000:c0:07.1: [1022:1454] type 01 class 0x060400 [ 2.523076] pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold [ 2.523153] pci 0000:c0:08.0: [1022:1452] type 00 class 0x060000 [ 2.523216] pci 0000:c0:08.1: [1022:1454] type 01 class 0x060400 [ 2.523329] pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold [ 2.524010] pci 0000:c1:00.0: [1000:005f] type 00 class 0x010400 [ 2.524024] pci 0000:c1:00.0: reg 0x10: [io 0xc000-0xc0ff] [ 2.524034] pci 0000:c1:00.0: reg 0x14: [mem 0xa5500000-0xa550ffff 64bit] [ 2.524044] pci 0000:c1:00.0: reg 0x1c: [mem 0xa5400000-0xa54fffff 64bit] [ 2.524056] pci 0000:c1:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 2.524105] pci 0000:c1:00.0: supports D1 D2 [ 2.524156] pci 0000:c0:01.1: PCI bridge to [bus c1] [ 2.529127] pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] [ 2.529130] pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] [ 2.529221] pci 0000:c2:00.0: [1022:145a] type 00 class 0x130000 [ 2.529325] pci 0000:c2:00.2: [1022:1456] type 00 class 0x108000 [ 2.529345] pci 0000:c2:00.2: reg 0x18: [mem 0xa5200000-0xa52fffff] [ 2.529358] pci 0000:c2:00.2: reg 0x24: [mem 0xa5300000-0xa5301fff] [ 2.529449] pci 0000:c0:07.1: PCI bridge to [bus c2] [ 2.534416] pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] [ 2.534511] pci 0000:c3:00.0: [1022:1455] type 00 class 0x130000 [ 2.534627] pci 0000:c3:00.1: [1022:1468] type 00 class 0x108000 [ 2.534647] pci 0000:c3:00.1: reg 0x18: [mem 0xa5000000-0xa50fffff] [ 2.534661] pci 0000:c3:00.1: reg 0x24: [mem 0xa5100000-0xa5101fff] [ 2.534760] pci 0000:c0:08.1: PCI bridge to [bus c3] [ 2.539729] pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] [ 2.539747] pci_bus 0000:c0: on NUMA node 3 [ 2.541882] vgaarb: device added: PCI:0000:83:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.549972] vgaarb: loaded [ 2.552682] vgaarb: bridge control possible 0000:83:00.0 [ 2.558110] SCSI subsystem initialized [ 2.561885] ACPI: bus type USB registered [ 2.565916] usbcore: registered new interface driver usbfs [ 2.571410] usbcore: registered new interface driver hub [ 2.576937] usbcore: registered new device driver usb [ 2.582311] EDAC MC: Ver: 3.0.0 [ 2.585716] PCI: Using ACPI for IRQ routing [ 2.608878] PCI: pci_cache_line_size set to 64 bytes [ 2.609025] e820: reserve RAM buffer [mem 0x0008f000-0x0008ffff] [ 2.609027] e820: reserve RAM buffer [mem 0x37028020-0x37ffffff] [ 2.609029] e820: reserve RAM buffer [mem 0x37041020-0x37ffffff] [ 2.609031] e820: reserve RAM buffer [mem 0x3704a020-0x37ffffff] [ 2.609032] e820: reserve RAM buffer [mem 0x3707c020-0x37ffffff] [ 2.609033] e820: reserve RAM buffer [mem 0x4f883000-0x4fffffff] [ 2.609034] e820: reserve RAM buffer [mem 0x6cacf000-0x6fffffff] [ 2.609036] e820: reserve RAM buffer [mem 0x107f380000-0x107fffffff] [ 2.609037] e820: reserve RAM buffer [mem 0x207ff80000-0x207fffffff] [ 2.609038] e820: reserve RAM buffer [mem 0x307ff80000-0x307fffffff] [ 2.609039] e820: reserve RAM buffer [mem 0x407ff80000-0x407fffffff] [ 2.609290] NetLabel: Initializing [ 2.612701] NetLabel: domain hash size = 128 [ 2.617059] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.622040] NetLabel: unlabeled traffic allowed by default [ 2.627817] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 2.632799] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 2.640833] Switched to clocksource hpet [ 2.649596] pnp: PnP ACPI init [ 2.652679] ACPI: bus type PNP registered [ 2.656918] system 00:00: [mem 0x80000000-0x8fffffff] has been reserved [ 2.663541] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 2.663598] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.663815] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.663995] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.664142] pnp: PnP ACPI: found 4 devices [ 2.668255] ACPI: bus type PNP unregistered [ 2.679627] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.689544] pci 0000:81:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.699456] pci 0000:81:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.709372] pci 0000:84:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.719287] pci 0000:c1:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.729227] pci 0000:00:03.1: BAR 14: assigned [mem 0xe1000000-0xe10fffff] [ 2.736111] pci 0000:01:00.0: BAR 6: assigned [mem 0xe1000000-0xe10fffff pref] [ 2.743338] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.748313] pci 0000:00:03.1: bridge window [mem 0xe1000000-0xe10fffff] [ 2.755108] pci 0000:00:03.1: bridge window [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.762857] pci 0000:00:07.1: PCI bridge to [bus 02] [ 2.767833] pci 0000:00:07.1: bridge window [mem 0xf7200000-0xf74fffff] [ 2.774629] pci 0000:00:08.1: PCI bridge to [bus 03] [ 2.779608] pci 0000:00:08.1: bridge window [mem 0xf7000000-0xf71fffff] [ 2.786408] pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] [ 2.786410] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] [ 2.786412] pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] [ 2.786413] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] [ 2.786415] pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] [ 2.786417] pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] [ 2.786419] pci_bus 0000:00: resource 10 [mem 0x000d0000-0x000d3fff window] [ 2.786420] pci_bus 0000:00: resource 11 [mem 0x000d4000-0x000d7fff window] [ 2.786422] pci_bus 0000:00: resource 12 [mem 0x000d8000-0x000dbfff window] [ 2.786424] pci_bus 0000:00: resource 13 [mem 0x000dc000-0x000dffff window] [ 2.786425] pci_bus 0000:00: resource 14 [mem 0x000e0000-0x000e3fff window] [ 2.786427] pci_bus 0000:00: resource 15 [mem 0x000e4000-0x000e7fff window] [ 2.786429] pci_bus 0000:00: resource 16 [mem 0x000e8000-0x000ebfff window] [ 2.786430] pci_bus 0000:00: resource 17 [mem 0x000ec000-0x000effff window] [ 2.786432] pci_bus 0000:00: resource 18 [mem 0x000f0000-0x000fffff window] [ 2.786434] pci_bus 0000:00: resource 19 [io 0x0d00-0x3fff window] [ 2.786435] pci_bus 0000:00: resource 20 [mem 0xe1000000-0xfebfffff window] [ 2.786437] pci_bus 0000:00: resource 21 [mem 0x10000000000-0x2bf3fffffff window] [ 2.786439] pci_bus 0000:01: resource 1 [mem 0xe1000000-0xe10fffff] [ 2.786441] pci_bus 0000:01: resource 2 [mem 0xe2000000-0xe3ffffff 64bit pref] [ 2.786443] pci_bus 0000:02: resource 1 [mem 0xf7200000-0xf74fffff] [ 2.786445] pci_bus 0000:03: resource 1 [mem 0xf7000000-0xf71fffff] [ 2.786456] pci 0000:40:07.1: PCI bridge to [bus 41] [ 2.791430] pci 0000:40:07.1: bridge window [mem 0xdb200000-0xdb4fffff] [ 2.798228] pci 0000:40:08.1: PCI bridge to [bus 42] [ 2.803200] pci 0000:40:08.1: bridge window [mem 0xdb000000-0xdb1fffff] [ 2.809997] pci_bus 0000:40: resource 4 [io 0x4000-0x7fff window] [ 2.809999] pci_bus 0000:40: resource 5 [mem 0xc6000000-0xe0ffffff window] [ 2.810001] pci_bus 0000:40: resource 6 [mem 0x2bf40000000-0x47e7fffffff window] [ 2.810002] pci_bus 0000:41: resource 1 [mem 0xdb200000-0xdb4fffff] [ 2.810004] pci_bus 0000:42: resource 1 [mem 0xdb000000-0xdb1fffff] [ 2.810037] pci 0000:80:01.1: BAR 14: assigned [mem 0xac300000-0xac3fffff] [ 2.816920] pci 0000:81:00.0: BAR 6: assigned [mem 0xac300000-0xac33ffff pref] [ 2.824148] pci 0000:81:00.1: BAR 6: assigned [mem 0xac340000-0xac37ffff pref] [ 2.831376] pci 0000:80:01.1: PCI bridge to [bus 81] [ 2.836351] pci 0000:80:01.1: bridge window [mem 0xac300000-0xac3fffff] [ 2.843146] pci 0000:80:01.1: bridge window [mem 0xac200000-0xac2fffff 64bit pref] [ 2.850895] pci 0000:82:00.0: PCI bridge to [bus 83] [ 2.855872] pci 0000:82:00.0: bridge window [mem 0xc0000000-0xc08fffff] [ 2.862664] pci 0000:82:00.0: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.870416] pci 0000:80:01.2: PCI bridge to [bus 82-83] [ 2.875657] pci 0000:80:01.2: bridge window [mem 0xc0000000-0xc08fffff] [ 2.882450] pci 0000:80:01.2: bridge window [mem 0xab000000-0xabffffff 64bit pref] [ 2.890201] pci 0000:84:00.0: BAR 6: no space for [mem size 0x00040000 pref] [ 2.897254] pci 0000:84:00.0: BAR 6: failed to assign [mem size 0x00040000 pref] [ 2.904653] pci 0000:80:03.1: PCI bridge to [bus 84] [ 2.909629] pci 0000:80:03.1: bridge window [io 0x8000-0x8fff] [ 2.915732] pci 0000:80:03.1: bridge window [mem 0xc0d00000-0xc0dfffff] [ 2.922526] pci 0000:80:03.1: bridge window [mem 0xac000000-0xac1fffff 64bit pref] [ 2.930276] pci 0000:80:07.1: PCI bridge to [bus 85] [ 2.935251] pci 0000:80:07.1: bridge window [mem 0xc0b00000-0xc0cfffff] [ 2.942047] pci 0000:80:08.1: PCI bridge to [bus 86] [ 2.947020] pci 0000:80:08.1: bridge window [mem 0xc0900000-0xc0afffff] [ 2.953815] pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] [ 2.953817] pci_bus 0000:80: resource 5 [mem 0x000a0000-0x000bffff window] [ 2.953819] pci_bus 0000:80: resource 6 [io 0x8000-0xbfff window] [ 2.953820] pci_bus 0000:80: resource 7 [mem 0xab000000-0xc5ffffff window] [ 2.953822] pci_bus 0000:80: resource 8 [mem 0x47e80000000-0x63dbfffffff window] [ 2.953824] pci_bus 0000:81: resource 1 [mem 0xac300000-0xac3fffff] [ 2.953826] pci_bus 0000:81: resource 2 [mem 0xac200000-0xac2fffff 64bit pref] [ 2.953827] pci_bus 0000:82: resource 1 [mem 0xc0000000-0xc08fffff] [ 2.953829] pci_bus 0000:82: resource 2 [mem 0xab000000-0xabffffff 64bit pref] [ 2.953831] pci_bus 0000:83: resource 1 [mem 0xc0000000-0xc08fffff] [ 2.953833] pci_bus 0000:83: resource 2 [mem 0xab000000-0xabffffff 64bit pref] [ 2.953834] pci_bus 0000:84: resource 0 [io 0x8000-0x8fff] [ 2.953836] pci_bus 0000:84: resource 1 [mem 0xc0d00000-0xc0dfffff] [ 2.953838] pci_bus 0000:84: resource 2 [mem 0xac000000-0xac1fffff 64bit pref] [ 2.953839] pci_bus 0000:85: resource 1 [mem 0xc0b00000-0xc0cfffff] [ 2.953841] pci_bus 0000:86: resource 1 [mem 0xc0900000-0xc0afffff] [ 2.953857] pci 0000:c1:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.960912] pci 0000:c1:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.968313] pci 0000:c0:01.1: PCI bridge to [bus c1] [ 2.973288] pci 0000:c0:01.1: bridge window [io 0xc000-0xcfff] [ 2.979389] pci 0000:c0:01.1: bridge window [mem 0xa5400000-0xa55fffff] [ 2.986186] pci 0000:c0:07.1: PCI bridge to [bus c2] [ 2.991159] pci 0000:c0:07.1: bridge window [mem 0xa5200000-0xa53fffff] [ 2.997958] pci 0000:c0:08.1: PCI bridge to [bus c3] [ 3.002937] pci 0000:c0:08.1: bridge window [mem 0xa5000000-0xa51fffff] [ 3.009736] pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] [ 3.009738] pci_bus 0000:c0: resource 5 [mem 0x90000000-0xaaffffff window] [ 3.009740] pci_bus 0000:c0: resource 6 [mem 0x63dc0000000-0x7ffffffffff window] [ 3.009741] pci_bus 0000:c1: resource 0 [io 0xc000-0xcfff] [ 3.009743] pci_bus 0000:c1: resource 1 [mem 0xa5400000-0xa55fffff] [ 3.009745] pci_bus 0000:c2: resource 1 [mem 0xa5200000-0xa53fffff] [ 3.009747] pci_bus 0000:c3: resource 1 [mem 0xa5000000-0xa51fffff] [ 3.009845] NET: Registered protocol family 2 [ 3.014910] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 3.023070] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 3.029906] TCP: Hash tables configured (established 524288 bind 65536) [ 3.036568] TCP: reno registered [ 3.039913] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.046517] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 3.053712] NET: Registered protocol family 1 [ 3.058515] pci 0000:83:00.0: Boot video device [ 3.058552] PCI: CLS 64 bytes, default 64 [ 3.058600] Unpacking initramfs... [ 3.328418] Freeing initrd memory: 19600k freed [ 3.335746] AMD-Vi: IOMMU performance counters supported [ 3.341130] AMD-Vi: IOMMU performance counters supported [ 3.346478] AMD-Vi: IOMMU performance counters supported [ 3.351843] AMD-Vi: IOMMU performance counters supported [ 3.358485] iommu: Adding device 0000:00:01.0 to group 0 [ 3.364497] iommu: Adding device 0000:00:02.0 to group 1 [ 3.370488] iommu: Adding device 0000:00:03.0 to group 2 [ 3.376603] iommu: Adding device 0000:00:03.1 to group 3 [ 3.382571] iommu: Adding device 0000:00:04.0 to group 4 [ 3.388607] iommu: Adding device 0000:00:07.0 to group 5 [ 3.394606] iommu: Adding device 0000:00:07.1 to group 6 [ 3.400654] iommu: Adding device 0000:00:08.0 to group 7 [ 3.406648] iommu: Adding device 0000:00:08.1 to group 8 [ 3.412664] iommu: Adding device 0000:00:14.0 to group 9 [ 3.418001] iommu: Adding device 0000:00:14.3 to group 9 [ 3.424088] iommu: Adding device 0000:00:18.0 to group 10 [ 3.429515] iommu: Adding device 0000:00:18.1 to group 10 [ 3.434940] iommu: Adding device 0000:00:18.2 to group 10 [ 3.440365] iommu: Adding device 0000:00:18.3 to group 10 [ 3.445786] iommu: Adding device 0000:00:18.4 to group 10 [ 3.451213] iommu: Adding device 0000:00:18.5 to group 10 [ 3.456643] iommu: Adding device 0000:00:18.6 to group 10 [ 3.462070] iommu: Adding device 0000:00:18.7 to group 10 [ 3.468266] iommu: Adding device 0000:00:19.0 to group 11 [ 3.473691] iommu: Adding device 0000:00:19.1 to group 11 [ 3.479115] iommu: Adding device 0000:00:19.2 to group 11 [ 3.484538] iommu: Adding device 0000:00:19.3 to group 11 [ 3.489964] iommu: Adding device 0000:00:19.4 to group 11 [ 3.495393] iommu: Adding device 0000:00:19.5 to group 11 [ 3.500831] iommu: Adding device 0000:00:19.6 to group 11 [ 3.506260] iommu: Adding device 0000:00:19.7 to group 11 [ 3.512421] iommu: Adding device 0000:00:1a.0 to group 12 [ 3.517850] iommu: Adding device 0000:00:1a.1 to group 12 [ 3.523272] iommu: Adding device 0000:00:1a.2 to group 12 [ 3.528701] iommu: Adding device 0000:00:1a.3 to group 12 [ 3.534126] iommu: Adding device 0000:00:1a.4 to group 12 [ 3.539551] iommu: Adding device 0000:00:1a.5 to group 12 [ 3.544975] iommu: Adding device 0000:00:1a.6 to group 12 [ 3.550401] iommu: Adding device 0000:00:1a.7 to group 12 [ 3.556549] iommu: Adding device 0000:00:1b.0 to group 13 [ 3.561979] iommu: Adding device 0000:00:1b.1 to group 13 [ 3.567407] iommu: Adding device 0000:00:1b.2 to group 13 [ 3.572843] iommu: Adding device 0000:00:1b.3 to group 13 [ 3.578267] iommu: Adding device 0000:00:1b.4 to group 13 [ 3.583688] iommu: Adding device 0000:00:1b.5 to group 13 [ 3.589118] iommu: Adding device 0000:00:1b.6 to group 13 [ 3.594545] iommu: Adding device 0000:00:1b.7 to group 13 [ 3.600659] iommu: Adding device 0000:01:00.0 to group 14 [ 3.606765] iommu: Adding device 0000:02:00.0 to group 15 [ 3.612846] iommu: Adding device 0000:02:00.2 to group 16 [ 3.618957] iommu: Adding device 0000:02:00.3 to group 17 [ 3.625066] iommu: Adding device 0000:03:00.0 to group 18 [ 3.631170] iommu: Adding device 0000:03:00.1 to group 19 [ 3.637224] iommu: Adding device 0000:40:01.0 to group 20 [ 3.643305] iommu: Adding device 0000:40:02.0 to group 21 [ 3.649423] iommu: Adding device 0000:40:03.0 to group 22 [ 3.655520] iommu: Adding device 0000:40:04.0 to group 23 [ 3.661629] iommu: Adding device 0000:40:07.0 to group 24 [ 3.667677] iommu: Adding device 0000:40:07.1 to group 25 [ 3.673682] iommu: Adding device 0000:40:08.0 to group 26 [ 3.679663] iommu: Adding device 0000:40:08.1 to group 27 [ 3.685659] iommu: Adding device 0000:41:00.0 to group 28 [ 3.691694] iommu: Adding device 0000:41:00.2 to group 29 [ 3.697709] iommu: Adding device 0000:41:00.3 to group 30 [ 3.703715] iommu: Adding device 0000:42:00.0 to group 31 [ 3.709712] iommu: Adding device 0000:42:00.1 to group 32 [ 3.715726] iommu: Adding device 0000:80:01.0 to group 33 [ 3.721783] iommu: Adding device 0000:80:01.1 to group 34 [ 3.727944] iommu: Adding device 0000:80:01.2 to group 35 [ 3.734008] iommu: Adding device 0000:80:02.0 to group 36 [ 3.740016] iommu: Adding device 0000:80:03.0 to group 37 [ 3.746017] iommu: Adding device 0000:80:03.1 to group 38 [ 3.752094] iommu: Adding device 0000:80:04.0 to group 39 [ 3.758145] iommu: Adding device 0000:80:07.0 to group 40 [ 3.764170] iommu: Adding device 0000:80:07.1 to group 41 [ 3.770173] iommu: Adding device 0000:80:08.0 to group 42 [ 3.776199] iommu: Adding device 0000:80:08.1 to group 43 [ 3.782280] iommu: Adding device 0000:81:00.0 to group 44 [ 3.787732] iommu: Adding device 0000:81:00.1 to group 44 [ 3.793783] iommu: Adding device 0000:82:00.0 to group 45 [ 3.799203] iommu: Adding device 0000:83:00.0 to group 45 [ 3.805234] iommu: Adding device 0000:84:00.0 to group 46 [ 3.811311] iommu: Adding device 0000:85:00.0 to group 47 [ 3.817308] iommu: Adding device 0000:85:00.2 to group 48 [ 3.823365] iommu: Adding device 0000:86:00.0 to group 49 [ 3.829437] iommu: Adding device 0000:86:00.1 to group 50 [ 3.835438] iommu: Adding device 0000:86:00.2 to group 51 [ 3.841474] iommu: Adding device 0000:c0:01.0 to group 52 [ 3.847536] iommu: Adding device 0000:c0:01.1 to group 53 [ 3.853565] iommu: Adding device 0000:c0:02.0 to group 54 [ 3.859593] iommu: Adding device 0000:c0:03.0 to group 55 [ 3.865664] iommu: Adding device 0000:c0:04.0 to group 56 [ 3.871709] iommu: Adding device 0000:c0:07.0 to group 57 [ 3.877802] iommu: Adding device 0000:c0:07.1 to group 58 [ 3.883856] iommu: Adding device 0000:c0:08.0 to group 59 [ 3.889881] iommu: Adding device 0000:c0:08.1 to group 60 [ 3.898410] iommu: Adding device 0000:c1:00.0 to group 61 [ 3.904437] iommu: Adding device 0000:c2:00.0 to group 62 [ 3.910488] iommu: Adding device 0000:c2:00.2 to group 63 [ 3.916506] iommu: Adding device 0000:c3:00.0 to group 64 [ 3.922522] iommu: Adding device 0000:c3:00.1 to group 65 [ 3.928175] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 3.933495] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.938814] PPR NX GT IA GA PC GA_vAPIC [ 3.942959] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 [ 3.948279] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.953600] PPR NX GT IA GA PC GA_vAPIC [ 3.957735] AMD-Vi: Found IOMMU at 0000:80:00.2 cap 0x40 [ 3.963058] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.968375] PPR NX GT IA GA PC GA_vAPIC [ 3.972513] AMD-Vi: Found IOMMU at 0000:c0:00.2 cap 0x40 [ 3.977834] AMD-Vi: Extended features (0xf77ef22294ada): [ 3.983153] PPR NX GT IA GA PC GA_vAPIC [ 3.987289] AMD-Vi: Interrupt remapping enabled [ 3.991829] AMD-Vi: virtual APIC enabled [ 3.995821] pci 0000:00:00.2: irq 26 for MSI/MSI-X [ 3.995931] pci 0000:40:00.2: irq 27 for MSI/MSI-X [ 3.996020] pci 0000:80:00.2: irq 28 for MSI/MSI-X [ 3.996106] pci 0000:c0:00.2: irq 29 for MSI/MSI-X [ 3.996168] AMD-Vi: Lazy IO/TLB flushing enabled [ 4.002502] perf: AMD NB counters detected [ 4.006647] perf: AMD LLC counters detected [ 4.016591] sha1_ssse3: Using SHA-NI optimized SHA-1 implementation [ 4.022940] sha256_ssse3: Using SHA-256-NI optimized SHA-256 implementation [ 4.031504] futex hash table entries: 32768 (order: 9, 2097152 bytes) [ 4.038129] Initialise system trusted keyring [ 4.042531] audit: initializing netlink socket (disabled) [ 4.047955] type=2000 audit(1584552003.198:1): initialized [ 4.078810] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.085174] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.092840] zpool: loaded [ 4.095469] zbud: loaded [ 4.098363] VFS: Disk quotas dquot_6.6.0 [ 4.102389] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.109167] msgmni has been set to 32768 [ 4.113183] Key type big_key registered [ 4.117033] SELinux: Registering netfilter hooks [ 4.119382] NET: Registered protocol family 38 [ 4.123841] Key type asymmetric registered [ 4.127944] Asymmetric key parser 'x509' registered [ 4.132878] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 4.140427] io scheduler noop registered [ 4.144360] io scheduler deadline registered (default) [ 4.149546] io scheduler cfq registered [ 4.153389] io scheduler mq-deadline registered [ 4.157932] io scheduler kyber registered [ 4.162494] pcieport 0000:00:03.1: irq 30 for MSI/MSI-X [ 4.163445] pcieport 0000:00:07.1: irq 31 for MSI/MSI-X [ 4.164431] pcieport 0000:00:08.1: irq 33 for MSI/MSI-X [ 4.164726] pcieport 0000:40:07.1: irq 34 for MSI/MSI-X [ 4.164984] pcieport 0000:40:08.1: irq 36 for MSI/MSI-X [ 4.165264] pcieport 0000:80:01.1: irq 37 for MSI/MSI-X [ 4.165516] pcieport 0000:80:01.2: irq 38 for MSI/MSI-X [ 4.166230] pcieport 0000:80:03.1: irq 39 for MSI/MSI-X [ 4.166519] pcieport 0000:80:07.1: irq 41 for MSI/MSI-X [ 4.167160] pcieport 0000:80:08.1: irq 43 for MSI/MSI-X [ 4.167498] pcieport 0000:c0:01.1: irq 44 for MSI/MSI-X [ 4.168219] pcieport 0000:c0:07.1: irq 46 for MSI/MSI-X [ 4.168470] pcieport 0000:c0:08.1: irq 48 for MSI/MSI-X [ 4.168589] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt [ 4.175553] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 4.182087] pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded [ 4.182102] pcieport 0000:00:07.1: Signaling PME through PCIe PME interrupt [ 4.189070] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 4.195606] pci 0000:02:00.2: Signaling PME through PCIe PME interrupt [ 4.202141] pci 0000:02:00.3: Signaling PME through PCIe PME interrupt [ 4.208678] pcie_pme 0000:00:07.1:pcie001: service driver pcie_pme loaded [ 4.208691] pcieport 0000:00:08.1: Signaling PME through PCIe PME interrupt [ 4.215653] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 4.222187] pci 0000:03:00.1: Signaling PME through PCIe PME interrupt [ 4.228723] pcie_pme 0000:00:08.1:pcie001: service driver pcie_pme loaded [ 4.228738] pcieport 0000:40:07.1: Signaling PME through PCIe PME interrupt [ 4.235709] pci 0000:41:00.0: Signaling PME through PCIe PME interrupt [ 4.242243] pci 0000:41:00.2: Signaling PME through PCIe PME interrupt [ 4.248779] pci 0000:41:00.3: Signaling PME through PCIe PME interrupt [ 4.255314] pcie_pme 0000:40:07.1:pcie001: service driver pcie_pme loaded [ 4.255327] pcieport 0000:40:08.1: Signaling PME through PCIe PME interrupt [ 4.262289] pci 0000:42:00.0: Signaling PME through PCIe PME interrupt [ 4.268825] pci 0000:42:00.1: Signaling PME through PCIe PME interrupt [ 4.275359] pcie_pme 0000:40:08.1:pcie001: service driver pcie_pme loaded [ 4.275377] pcieport 0000:80:01.1: Signaling PME through PCIe PME interrupt [ 4.282345] pci 0000:81:00.0: Signaling PME through PCIe PME interrupt [ 4.288879] pci 0000:81:00.1: Signaling PME through PCIe PME interrupt [ 4.295415] pcie_pme 0000:80:01.1:pcie001: service driver pcie_pme loaded [ 4.295432] pcieport 0000:80:01.2: Signaling PME through PCIe PME interrupt [ 4.302401] pci 0000:82:00.0: Signaling PME through PCIe PME interrupt [ 4.308935] pci 0000:83:00.0: Signaling PME through PCIe PME interrupt [ 4.315471] pcie_pme 0000:80:01.2:pcie001: service driver pcie_pme loaded [ 4.315486] pcieport 0000:80:03.1: Signaling PME through PCIe PME interrupt [ 4.322454] pci 0000:84:00.0: Signaling PME through PCIe PME interrupt [ 4.328991] pcie_pme 0000:80:03.1:pcie001: service driver pcie_pme loaded [ 4.329007] pcieport 0000:80:07.1: Signaling PME through PCIe PME interrupt [ 4.335975] pci 0000:85:00.0: Signaling PME through PCIe PME interrupt [ 4.342509] pci 0000:85:00.2: Signaling PME through PCIe PME interrupt [ 4.349047] pcie_pme 0000:80:07.1:pcie001: service driver pcie_pme loaded [ 4.349062] pcieport 0000:80:08.1: Signaling PME through PCIe PME interrupt [ 4.356030] pci 0000:86:00.0: Signaling PME through PCIe PME interrupt [ 4.362565] pci 0000:86:00.1: Signaling PME through PCIe PME interrupt [ 4.369100] pci 0000:86:00.2: Signaling PME through PCIe PME interrupt [ 4.375638] pcie_pme 0000:80:08.1:pcie001: service driver pcie_pme loaded [ 4.375654] pcieport 0000:c0:01.1: Signaling PME through PCIe PME interrupt [ 4.382620] pci 0000:c1:00.0: Signaling PME through PCIe PME interrupt [ 4.389157] pcie_pme 0000:c0:01.1:pcie001: service driver pcie_pme loaded [ 4.389172] pcieport 0000:c0:07.1: Signaling PME through PCIe PME interrupt [ 4.396140] pci 0000:c2:00.0: Signaling PME through PCIe PME interrupt [ 4.402675] pci 0000:c2:00.2: Signaling PME through PCIe PME interrupt [ 4.409212] pcie_pme 0000:c0:07.1:pcie001: service driver pcie_pme loaded [ 4.409228] pcieport 0000:c0:08.1: Signaling PME through PCIe PME interrupt [ 4.416196] pci 0000:c3:00.0: Signaling PME through PCIe PME interrupt [ 4.422731] pci 0000:c3:00.1: Signaling PME through PCIe PME interrupt [ 4.429268] pcie_pme 0000:c0:08.1:pcie001: service driver pcie_pme loaded [ 4.429291] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.434878] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.441549] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.448358] efifb: probing for efifb [ 4.451959] efifb: framebuffer at 0xab000000, mapped to 0xffffa62919800000, using 3072k, total 3072k [ 4.461092] efifb: mode is 1024x768x32, linelength=4096, pages=1 [ 4.467106] efifb: scrolling: redraw [ 4.470694] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 4.491904] Console: switching to colour frame buffer device 128x48 [ 4.513582] fb0: EFI VGA frame buffer device [ 4.517961] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 [ 4.526148] ACPI: Power Button [PWRB] [ 4.529881] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 4.537285] ACPI: Power Button [PWRF] [ 4.542159] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 4.549639] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.576831] 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.603374] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.609438] Non-volatile memory driver v1.3 [ 4.613663] Linux agpgart interface v0.103 [ 4.620302] crash memory driver: version 1.1 [ 4.624811] rdac: device handler registered [ 4.629055] hp_sw: device handler registered [ 4.633341] emc: device handler registered [ 4.637599] alua: device handler registered [ 4.641831] libphy: Fixed MDIO Bus: probed [ 4.645989] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.652535] ehci-pci: EHCI PCI platform driver [ 4.657008] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.663200] ohci-pci: OHCI PCI platform driver [ 4.667667] uhci_hcd: USB Universal Host Controller Interface driver [ 4.674171] xhci_hcd 0000:02:00.3: xHCI Host Controller [ 4.679469] xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 1 [ 4.686973] xhci_hcd 0000:02:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 [ 4.695451] xhci_hcd 0000:02:00.3: irq 50 for MSI/MSI-X [ 4.695474] xhci_hcd 0000:02:00.3: irq 51 for MSI/MSI-X [ 4.695493] xhci_hcd 0000:02:00.3: irq 52 for MSI/MSI-X [ 4.695512] xhci_hcd 0000:02:00.3: irq 53 for MSI/MSI-X [ 4.695531] xhci_hcd 0000:02:00.3: irq 54 for MSI/MSI-X [ 4.695551] xhci_hcd 0000:02:00.3: irq 55 for MSI/MSI-X [ 4.695571] xhci_hcd 0000:02:00.3: irq 56 for MSI/MSI-X [ 4.695591] xhci_hcd 0000:02:00.3: irq 57 for MSI/MSI-X [ 4.695730] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 4.702528] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.709757] usb usb1: Product: xHCI Host Controller [ 4.714645] usb usb1: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.722738] usb usb1: SerialNumber: 0000:02:00.3 [ 4.727475] hub 1-0:1.0: USB hub found [ 4.731236] hub 1-0:1.0: 2 ports detected [ 4.735485] xhci_hcd 0000:02:00.3: xHCI Host Controller [ 4.740780] xhci_hcd 0000:02:00.3: new USB bus registered, assigned bus number 2 [ 4.748214] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 4.756319] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 4.763118] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.770346] usb usb2: Product: xHCI Host Controller [ 4.775234] usb usb2: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.783329] usb usb2: SerialNumber: 0000:02:00.3 [ 4.788044] hub 2-0:1.0: USB hub found [ 4.791812] hub 2-0:1.0: 2 ports detected [ 4.796142] xhci_hcd 0000:41:00.3: xHCI Host Controller [ 4.801436] xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 3 [ 4.808952] xhci_hcd 0000:41:00.3: hcc params 0x0270f665 hci version 0x100 quirks 0x00000410 [ 4.817429] xhci_hcd 0000:41:00.3: irq 59 for MSI/MSI-X [ 4.817448] xhci_hcd 0000:41:00.3: irq 60 for MSI/MSI-X [ 4.817466] xhci_hcd 0000:41:00.3: irq 61 for MSI/MSI-X [ 4.817486] xhci_hcd 0000:41:00.3: irq 62 for MSI/MSI-X [ 4.817504] xhci_hcd 0000:41:00.3: irq 63 for MSI/MSI-X [ 4.817523] xhci_hcd 0000:41:00.3: irq 64 for MSI/MSI-X [ 4.817543] xhci_hcd 0000:41:00.3: irq 65 for MSI/MSI-X [ 4.817561] xhci_hcd 0000:41:00.3: irq 66 for MSI/MSI-X [ 4.817702] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 4.824499] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.831727] usb usb3: Product: xHCI Host Controller [ 4.836614] usb usb3: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.844708] usb usb3: SerialNumber: 0000:41:00.3 [ 4.849442] hub 3-0:1.0: USB hub found [ 4.853205] hub 3-0:1.0: 2 ports detected [ 4.857457] xhci_hcd 0000:41:00.3: xHCI Host Controller [ 4.862743] xhci_hcd 0000:41:00.3: new USB bus registered, assigned bus number 4 [ 4.870159] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. [ 4.878268] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 [ 4.885061] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.892289] usb usb4: Product: xHCI Host Controller [ 4.897178] usb usb4: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl2.x86_64 xhci-hcd [ 4.905272] usb usb4: SerialNumber: 0000:41:00.3 [ 4.909994] hub 4-0:1.0: USB hub found [ 4.913764] hub 4-0:1.0: 2 ports detected [ 4.918025] usbcore: registered new interface driver usbserial_generic [ 4.924563] usbserial: USB Serial support registered for generic [ 4.930618] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 5.051867] usb 1-1: new high-speed USB device number 2 using xhci_hcd [ 5.183721] usb 1-1: New USB device found, idVendor=0424, idProduct=2744 [ 5.190435] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 5.197570] usb 1-1: Product: USB2734 [ 5.201245] usb 1-1: Manufacturer: Microchip Tech [ 5.228370] hub 1-1:1.0: USB hub found [ 5.232346] hub 1-1:1.0: 4 ports detected [ 5.299973] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd [ 5.321092] usb 2-1: New USB device found, idVendor=0424, idProduct=5744 [ 5.327801] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=0 [ 5.334942] usb 2-1: Product: USB5734 [ 5.338616] usb 2-1: Manufacturer: Microchip Tech [ 5.356363] hub 2-1:1.0: USB hub found [ 5.360342] hub 2-1:1.0: 4 ports detected [ 5.365265] usb: port power management may be unreliable [ 5.970334] i8042: No controller found [ 5.974133] tsc: Refined TSC clocksource calibration: 1996.248 MHz [ 5.974170] mousedev: PS/2 mouse device common for all mice [ 5.974366] rtc_cmos 00:01: RTC can wake from S4 [ 5.974717] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 5.974816] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 5.974879] cpuidle: using governor menu [ 5.975139] EFI Variables Facility v0.08 2004-May-17 [ 5.998802] hidraw: raw HID events driver (C) Jiri Kosina [ 5.998904] usbcore: registered new interface driver usbhid [ 5.998904] usbhid: USB HID core driver [ 5.999014] drop_monitor: Initializing network drop monitor service [ 5.999160] TCP: cubic registered [ 5.999165] Initializing XFRM netlink socket [ 5.999379] NET: Registered protocol family 10 [ 5.999935] NET: Registered protocol family 17 [ 5.999939] mpls_gso: MPLS GSO support [ 6.001001] mce: Using 23 MCE banks [ 6.001050] microcode: CPU0: patch_level=0x08001250 [ 6.001060] microcode: CPU1: patch_level=0x08001250 [ 6.001071] microcode: CPU2: patch_level=0x08001250 [ 6.001083] microcode: CPU3: patch_level=0x08001250 [ 6.001098] microcode: CPU4: patch_level=0x08001250 [ 6.005073] microcode: CPU5: patch_level=0x08001250 [ 6.005089] microcode: CPU6: patch_level=0x08001250 [ 6.005107] microcode: CPU7: patch_level=0x08001250 [ 6.005117] microcode: CPU8: patch_level=0x08001250 [ 6.005129] microcode: CPU9: patch_level=0x08001250 [ 6.005140] microcode: CPU10: patch_level=0x08001250 [ 6.005150] microcode: CPU11: patch_level=0x08001250 [ 6.005161] microcode: CPU12: patch_level=0x08001250 [ 6.005172] microcode: CPU13: patch_level=0x08001250 [ 6.005183] microcode: CPU14: patch_level=0x08001250 [ 6.005194] microcode: CPU15: patch_level=0x08001250 [ 6.005204] microcode: CPU16: patch_level=0x08001250 [ 6.005216] microcode: CPU17: patch_level=0x08001250 [ 6.005227] microcode: CPU18: patch_level=0x08001250 [ 6.005238] microcode: CPU19: patch_level=0x08001250 [ 6.005248] microcode: CPU20: patch_level=0x08001250 [ 6.005259] microcode: CPU21: patch_level=0x08001250 [ 6.005270] microcode: CPU22: patch_level=0x08001250 [ 6.005282] microcode: CPU23: patch_level=0x08001250 [ 6.005291] microcode: CPU24: patch_level=0x08001250 [ 6.005302] microcode: CPU25: patch_level=0x08001250 [ 6.005312] microcode: CPU26: patch_level=0x08001250 [ 6.005323] microcode: CPU27: patch_level=0x08001250 [ 6.005334] microcode: CPU28: patch_level=0x08001250 [ 6.005343] microcode: CPU29: patch_level=0x08001250 [ 6.005353] microcode: CPU30: patch_level=0x08001250 [ 6.005364] microcode: CPU31: patch_level=0x08001250 [ 6.005374] microcode: CPU32: patch_level=0x08001250 [ 6.005383] microcode: CPU33: patch_level=0x08001250 [ 6.005393] microcode: CPU34: patch_level=0x08001250 [ 6.005404] microcode: CPU35: patch_level=0x08001250 [ 6.005413] microcode: CPU36: patch_level=0x08001250 [ 6.005421] microcode: CPU37: patch_level=0x08001250 [ 6.005432] microcode: CPU38: patch_level=0x08001250 [ 6.005442] microcode: CPU39: patch_level=0x08001250 [ 6.005450] microcode: CPU40: patch_level=0x08001250 [ 6.005461] microcode: CPU41: patch_level=0x08001250 [ 6.005472] microcode: CPU42: patch_level=0x08001250 [ 6.005483] microcode: CPU43: patch_level=0x08001250 [ 6.005491] microcode: CPU44: patch_level=0x08001250 [ 6.005499] microcode: CPU45: patch_level=0x08001250 [ 6.005511] microcode: CPU46: patch_level=0x08001250 [ 6.005521] microcode: CPU47: patch_level=0x08001250 [ 6.005569] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 6.005709] PM: Hibernation image not present or could not be loaded. [ 6.005712] Loading compiled-in X.509 certificates [ 6.005739] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 6.005753] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 6.006138] Loaded X.509 cert 'CentOS Linux kernel signing key: 468656045a39b52ff2152c315f6198c3e658f24d' [ 6.006153] registered taskstats version 1 [ 6.008339] Key type trusted registered [ 6.009896] Key type encrypted registered [ 6.009944] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 6.012203] Magic number: 12:408:340 [ 6.012313] acpi ACPI0007:02: hash matches [ 6.012371] memory memory1423: hash matches [ 6.012414] memory memory636: hash matches [ 6.013312] rtc_cmos 00:01: setting system clock to 2020-03-18 17:20:10 UTC (1584552010) [ 6.390506] Switched to clocksource tsc [ 6.395338] Freeing unused kernel memory: 1876k freed [ 6.400672] Write protecting the kernel read-only data: 12288k [ 6.407890] Freeing unused kernel memory: 504k freed [ 6.414254] Freeing unused kernel memory: 596k freed [ 6.471185] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 6.490259] systemd[1]: Detected architecture x86-64. [ 6.495319] systemd[1]: Running in initial RAM disk. [ 6.505892] usb 3-1: new high-speed USB device number 2 using xhci_hcd [ 6.515035] systemd[1]: Set hostname to . [ 6.550449] systemd[1]: Reached target Timers. [ 6.558983] systemd[1]: Reached target Swap. [ 6.568166] systemd[1]: Created slice Root Slice. [ 6.579006] systemd[1]: Listening on Journal Socket. [ 6.590020] systemd[1]: Created slice System Slice. [ 6.601445] systemd[1]: Starting Setup Virtual Console... [ 6.611336] systemd[1]: Starting Journal Service... [ 6.634439] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 6.635839] usb 3-1: New USB device found, idVendor=1604, idProduct=10c0 [ 6.635840] usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 6.662362] hub 3-1:1.0: USB hub found [ 6.667212] hub 3-1:1.0: 4 ports detected [ 6.674485] systemd[1]: Starting dracut cmdline hook... [ 6.684955] systemd[1]: Reached target Slices. [ 6.693958] systemd[1]: Reached target Local File Systems. [ 6.704988] systemd[1]: Listening on udev Control Socket. [ 6.716455] systemd[1]: Starting Apply Kernel Variables... [ 6.726980] systemd[1]: Listening on udev Kernel Socket. [ 6.737957] systemd[1]: Reached target Sockets. [ 6.747208] systemd[1]: Started Journal Service. [ 6.888304] pps_core: LinuxPPS API ver. 1 registered [ 6.893284] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 6.906735] PTP clock support registered [ 6.906744] megasas: 07.705.02.00-rh1 [ 6.907046] megaraid_sas 0000:c1:00.0: FW now in Ready state [ 6.907049] megaraid_sas 0000:c1:00.0: 64 bit DMA mask and 32 bit consistent mask [ 6.907300] megaraid_sas 0000:c1:00.0: irq 68 for MSI/MSI-X [ 6.907321] megaraid_sas 0000:c1:00.0: irq 69 for MSI/MSI-X [ 6.907345] megaraid_sas 0000:c1:00.0: irq 70 for MSI/MSI-X [ 6.907366] megaraid_sas 0000:c1:00.0: irq 71 for MSI/MSI-X [ 6.907386] megaraid_sas 0000:c1:00.0: irq 72 for MSI/MSI-X [ 6.907408] megaraid_sas 0000:c1:00.0: irq 73 for MSI/MSI-X [ 6.907428] megaraid_sas 0000:c1:00.0: irq 74 for MSI/MSI-X [ 6.907448] megaraid_sas 0000:c1:00.0: irq 75 for MSI/MSI-X [ 6.907467] megaraid_sas 0000:c1:00.0: irq 76 for MSI/MSI-X [ 6.907488] megaraid_sas 0000:c1:00.0: irq 77 for MSI/MSI-X [ 6.907506] megaraid_sas 0000:c1:00.0: irq 78 for MSI/MSI-X [ 6.907525] megaraid_sas 0000:c1:00.0: irq 79 for MSI/MSI-X [ 6.907548] megaraid_sas 0000:c1:00.0: irq 80 for MSI/MSI-X [ 6.907568] megaraid_sas 0000:c1:00.0: irq 81 for MSI/MSI-X [ 6.907590] megaraid_sas 0000:c1:00.0: irq 82 for MSI/MSI-X [ 6.907609] megaraid_sas 0000:c1:00.0: irq 83 for MSI/MSI-X [ 6.907630] megaraid_sas 0000:c1:00.0: irq 84 for MSI/MSI-X [ 6.907650] megaraid_sas 0000:c1:00.0: irq 85 for MSI/MSI-X [ 6.907669] megaraid_sas 0000:c1:00.0: irq 86 for MSI/MSI-X [ 6.907689] megaraid_sas 0000:c1:00.0: irq 87 for MSI/MSI-X [ 6.907710] megaraid_sas 0000:c1:00.0: irq 88 for MSI/MSI-X [ 6.907730] megaraid_sas 0000:c1:00.0: irq 89 for MSI/MSI-X [ 6.907749] megaraid_sas 0000:c1:00.0: irq 90 for MSI/MSI-X [ 6.907768] megaraid_sas 0000:c1:00.0: irq 91 for MSI/MSI-X [ 6.907792] megaraid_sas 0000:c1:00.0: irq 92 for MSI/MSI-X [ 6.907811] megaraid_sas 0000:c1:00.0: irq 93 for MSI/MSI-X [ 6.907831] megaraid_sas 0000:c1:00.0: irq 94 for MSI/MSI-X [ 6.907850] megaraid_sas 0000:c1:00.0: irq 95 for MSI/MSI-X [ 6.907870] megaraid_sas 0000:c1:00.0: irq 96 for MSI/MSI-X [ 6.907891] megaraid_sas 0000:c1:00.0: irq 97 for MSI/MSI-X [ 6.907918] megaraid_sas 0000:c1:00.0: irq 98 for MSI/MSI-X [ 6.907938] megaraid_sas 0000:c1:00.0: irq 99 for MSI/MSI-X [ 6.907957] megaraid_sas 0000:c1:00.0: irq 100 for MSI/MSI-X [ 6.907981] megaraid_sas 0000:c1:00.0: irq 101 for MSI/MSI-X [ 6.908004] megaraid_sas 0000:c1:00.0: irq 102 for MSI/MSI-X [ 6.908028] megaraid_sas 0000:c1:00.0: irq 103 for MSI/MSI-X [ 6.908054] megaraid_sas 0000:c1:00.0: irq 104 for MSI/MSI-X [ 6.908077] megaraid_sas 0000:c1:00.0: irq 105 for MSI/MSI-X [ 6.908099] megaraid_sas 0000:c1:00.0: irq 106 for MSI/MSI-X [ 6.908121] megaraid_sas 0000:c1:00.0: irq 107 for MSI/MSI-X [ 6.908143] megaraid_sas 0000:c1:00.0: irq 108 for MSI/MSI-X [ 6.908166] megaraid_sas 0000:c1:00.0: irq 109 for MSI/MSI-X [ 6.908187] megaraid_sas 0000:c1:00.0: irq 110 for MSI/MSI-X [ 6.908210] megaraid_sas 0000:c1:00.0: irq 111 for MSI/MSI-X [ 6.908233] megaraid_sas 0000:c1:00.0: irq 112 for MSI/MSI-X [ 6.908256] megaraid_sas 0000:c1:00.0: irq 113 for MSI/MSI-X [ 6.908278] megaraid_sas 0000:c1:00.0: irq 114 for MSI/MSI-X [ 6.908302] megaraid_sas 0000:c1:00.0: irq 115 for MSI/MSI-X [ 6.908411] megaraid_sas 0000:c1:00.0: firmware supports msix : (96) [ 6.908413] megaraid_sas 0000:c1:00.0: current msix/online cpus : (48/48) [ 6.908414] megaraid_sas 0000:c1:00.0: RDPQ mode : (disabled) [ 6.908417] megaraid_sas 0000:c1:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 [ 6.908708] megaraid_sas 0000:c1:00.0: Configured max firmware commands: 927 [ 6.910867] megaraid_sas 0000:c1:00.0: FW supports sync cache : No [ 6.971980] mpt3sas: loading out-of-tree module taints kernel. [ 6.981346] mlx_compat: module verification failed: signature and/or required key missing - tainting kernel [ 6.981943] libata version 3.00 loaded. [ 6.992580] Compat-mlnx-ofed backport release: 1c4bf42 [ 6.997797] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git 1c4bf42 [ 7.005800] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 7.016840] mpt3sas version 31.00.00.00 loaded [ 7.022835] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (263565236 kB) [ 7.035240] tg3.c:v3.137 (May 11, 2014) [ 7.040243] ahci 0000:86:00.2: version 3.0 [ 7.040720] ahci 0000:86:00.2: irq 119 for MSI/MSI-X [ 7.040730] ahci 0000:86:00.2: irq 120 for MSI/MSI-X [ 7.040735] ahci 0000:86:00.2: irq 121 for MSI/MSI-X [ 7.040740] ahci 0000:86:00.2: irq 122 for MSI/MSI-X [ 7.040747] ahci 0000:86:00.2: irq 123 for MSI/MSI-X [ 7.040752] ahci 0000:86:00.2: irq 124 for MSI/MSI-X [ 7.040757] ahci 0000:86:00.2: irq 125 for MSI/MSI-X [ 7.040763] ahci 0000:86:00.2: irq 126 for MSI/MSI-X [ 7.040768] ahci 0000:86:00.2: irq 127 for MSI/MSI-X [ 7.040773] ahci 0000:86:00.2: irq 128 for MSI/MSI-X [ 7.040778] ahci 0000:86:00.2: irq 129 for MSI/MSI-X [ 7.040783] ahci 0000:86:00.2: irq 130 for MSI/MSI-X [ 7.040788] ahci 0000:86:00.2: irq 131 for MSI/MSI-X [ 7.040792] ahci 0000:86:00.2: irq 132 for MSI/MSI-X [ 7.040796] ahci 0000:86:00.2: irq 133 for MSI/MSI-X [ 7.040800] ahci 0000:86:00.2: irq 134 for MSI/MSI-X [ 7.040843] ahci 0000:86:00.2: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode [ 7.048950] ahci 0000:86:00.2: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part [ 7.053428] tg3 0000:81:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4b:07 [ 7.053432] tg3 0000:81:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 7.053434] tg3 0000:81:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 7.053436] tg3 0000:81:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 7.075588] tg3 0000:81:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address d0:94:66:34:4b:08 [ 7.075591] tg3 0000:81:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 7.075593] tg3 0000:81:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 7.075595] tg3 0000:81:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] [ 7.090071] scsi host2: ahci [ 7.090192] ata1: SATA max UDMA/133 abar m4096@0xc0a02000 port 0xc0a02100 irq 119 [ 7.111912] mpt3sas_cm0: IOC Number : 0 [ 7.111914] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 7.112019] mpt3sas 0000:84:00.0: irq 136 for MSI/MSI-X [ 7.112043] mpt3sas 0000:84:00.0: irq 137 for MSI/MSI-X [ 7.112065] mpt3sas 0000:84:00.0: irq 138 for MSI/MSI-X [ 7.112087] mpt3sas 0000:84:00.0: irq 139 for MSI/MSI-X [ 7.112108] mpt3sas 0000:84:00.0: irq 140 for MSI/MSI-X [ 7.112129] mpt3sas 0000:84:00.0: irq 141 for MSI/MSI-X [ 7.112151] mpt3sas 0000:84:00.0: irq 142 for MSI/MSI-X [ 7.112174] mpt3sas 0000:84:00.0: irq 143 for MSI/MSI-X [ 7.112194] mpt3sas 0000:84:00.0: irq 144 for MSI/MSI-X [ 7.112217] mpt3sas 0000:84:00.0: irq 145 for MSI/MSI-X [ 7.112239] mpt3sas 0000:84:00.0: irq 146 for MSI/MSI-X [ 7.112261] mpt3sas 0000:84:00.0: irq 147 for MSI/MSI-X [ 7.112281] mpt3sas 0000:84:00.0: irq 148 for MSI/MSI-X [ 7.112302] mpt3sas 0000:84:00.0: irq 149 for MSI/MSI-X [ 7.112324] mpt3sas 0000:84:00.0: irq 150 for MSI/MSI-X [ 7.112348] mpt3sas 0000:84:00.0: irq 151 for MSI/MSI-X [ 7.112370] mpt3sas 0000:84:00.0: irq 152 for MSI/MSI-X [ 7.112393] mpt3sas 0000:84:00.0: irq 153 for MSI/MSI-X [ 7.112415] mpt3sas 0000:84:00.0: irq 154 for MSI/MSI-X [ 7.112436] mpt3sas 0000:84:00.0: irq 155 for MSI/MSI-X [ 7.112460] mpt3sas 0000:84:00.0: irq 156 for MSI/MSI-X [ 7.112483] mpt3sas 0000:84:00.0: irq 157 for MSI/MSI-X [ 7.112507] mpt3sas 0000:84:00.0: irq 158 for MSI/MSI-X [ 7.112533] mpt3sas 0000:84:00.0: irq 159 for MSI/MSI-X [ 7.112555] mpt3sas 0000:84:00.0: irq 160 for MSI/MSI-X [ 7.112586] mpt3sas 0000:84:00.0: irq 161 for MSI/MSI-X [ 7.112606] mpt3sas 0000:84:00.0: irq 162 for MSI/MSI-X [ 7.112628] mpt3sas 0000:84:00.0: irq 163 for MSI/MSI-X [ 7.112652] mpt3sas 0000:84:00.0: irq 164 for MSI/MSI-X [ 7.112675] mpt3sas 0000:84:00.0: irq 165 for MSI/MSI-X [ 7.112696] mpt3sas 0000:84:00.0: irq 166 for MSI/MSI-X [ 7.112722] mpt3sas 0000:84:00.0: irq 167 for MSI/MSI-X [ 7.112744] mpt3sas 0000:84:00.0: irq 168 for MSI/MSI-X [ 7.112766] mpt3sas 0000:84:00.0: irq 169 for MSI/MSI-X [ 7.112788] mpt3sas 0000:84:00.0: irq 170 for MSI/MSI-X [ 7.112810] mpt3sas 0000:84:00.0: irq 171 for MSI/MSI-X [ 7.112832] mpt3sas 0000:84:00.0: irq 172 for MSI/MSI-X [ 7.112853] mpt3sas 0000:84:00.0: irq 173 for MSI/MSI-X [ 7.112873] mpt3sas 0000:84:00.0: irq 174 for MSI/MSI-X [ 7.112892] mpt3sas 0000:84:00.0: irq 175 for MSI/MSI-X [ 7.112924] mpt3sas 0000:84:00.0: irq 176 for MSI/MSI-X [ 7.112946] mpt3sas 0000:84:00.0: irq 177 for MSI/MSI-X [ 7.112968] mpt3sas 0000:84:00.0: irq 178 for MSI/MSI-X [ 7.112990] mpt3sas 0000:84:00.0: irq 179 for MSI/MSI-X [ 7.113012] mpt3sas 0000:84:00.0: irq 180 for MSI/MSI-X [ 7.113036] mpt3sas 0000:84:00.0: irq 181 for MSI/MSI-X [ 7.113058] mpt3sas 0000:84:00.0: irq 182 for MSI/MSI-X [ 7.113083] mpt3sas 0000:84:00.0: irq 183 for MSI/MSI-X [ 7.113752] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 136 [ 7.113754] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 137 [ 7.113754] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 138 [ 7.113755] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 139 [ 7.113756] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 140 [ 7.113757] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 141 [ 7.113757] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 142 [ 7.113758] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 143 [ 7.113759] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 144 [ 7.113759] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 145 [ 7.113760] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 146 [ 7.113761] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 147 [ 7.113761] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 148 [ 7.113762] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 149 [ 7.113762] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 150 [ 7.113763] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 151 [ 7.113764] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 152 [ 7.113764] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 153 [ 7.113765] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 154 [ 7.113766] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 155 [ 7.113766] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 156 [ 7.113767] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 157 [ 7.113767] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 158 [ 7.113768] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 159 [ 7.113769] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 160 [ 7.113769] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 161 [ 7.113770] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 162 [ 7.113770] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 163 [ 7.113771] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 164 [ 7.113771] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 165 [ 7.113772] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 166 [ 7.113773] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 167 [ 7.113773] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 168 [ 7.113774] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 169 [ 7.113774] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 170 [ 7.113775] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 171 [ 7.113776] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 172 [ 7.113776] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 173 [ 7.113777] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 174 [ 7.113777] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 175 [ 7.113778] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 176 [ 7.113779] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 177 [ 7.113779] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 178 [ 7.113780] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 179 [ 7.113780] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 180 [ 7.113781] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 181 [ 7.113782] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 182 [ 7.113782] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 183 [ 7.113784] mpt3sas_cm0: iomem(0x00000000ac000000), mapped(0xffffa6291a000000), size(1048576) [ 7.113785] mpt3sas_cm0: ioport(0x0000000000008000), size(256) [ 7.190909] mpt3sas_cm0: IOC Number : 0 [ 7.190911] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 7.266912] megaraid_sas 0000:c1:00.0: Init cmd return status SUCCESS for SCSI host 0 [ 7.287909] megaraid_sas 0000:c1:00.0: firmware type : Legacy(64 VD) firmware [ 7.287911] megaraid_sas 0000:c1:00.0: controller type : iMR(0MB) [ 7.287912] megaraid_sas 0000:c1:00.0: Online Controller Reset(OCR) : Enabled [ 7.287912] megaraid_sas 0000:c1:00.0: Secure JBOD support : No [ 7.287913] megaraid_sas 0000:c1:00.0: NVMe passthru support : No [ 7.309419] megaraid_sas 0000:c1:00.0: INIT adapter done [ 7.309421] megaraid_sas 0000:c1:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 [ 7.335510] megaraid_sas 0000:c1:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) [ 7.335512] megaraid_sas 0000:c1:00.0: unevenspan support : yes [ 7.335514] megaraid_sas 0000:c1:00.0: firmware crash dump : no [ 7.335515] megaraid_sas 0000:c1:00.0: jbod sync map : no [ 7.335519] scsi host0: Avago SAS based MegaRAID driver [ 7.352342] mpt3sas_cm0: Allocated physical memory: size(38831 kB) [ 7.352344] mpt3sas_cm0: Current Controller Queue Depth(7564), Max Controller Queue Depth(7680) [ 7.352345] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 7.352942] usb 3-1.1: new high-speed USB device number 3 using xhci_hcd [ 7.353843] mlx5_core 0000:01:00.0: firmware version: 20.26.1040 [ 7.353872] mlx5_core 0000:01:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16 GT/s x16 link) [ 7.355372] scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.30 PQ: 0 ANSI: 5 [ 7.396924] ata1: SATA link down (SStatus 0 SControl 300) [ 7.437860] usb 3-1.1: New USB device found, idVendor=1604, idProduct=10c0 [ 7.437862] usb 3-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 7.462390] hub 3-1.1:1.0: USB hub found [ 7.462734] hub 3-1.1:1.0: 4 ports detected [ 7.496523] mpt3sas_cm0: FW Package Version(12.00.00.00) [ 7.496777] mpt3sas_cm0: SAS3616: FWVersion(12.00.00.00), ChipRevision(0x02), BiosVersion(00.00.00.00) [ 7.496781] mpt3sas_cm0: Protocol=(Initiator,Target,NVMe), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ) [ 7.496850] mpt3sas 0000:84:00.0: Enabled Extended Tags as Controller Supports [ 7.496865] mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 [ 7.496875] scsi host1: Fusion MPT SAS Host [ 7.497125] mpt3sas_cm0: sending port enable !! [ 7.527919] usb 3-1.4: new high-speed USB device number 4 using xhci_hcd [ 7.601864] usb 3-1.4: New USB device found, idVendor=1604, idProduct=10c0 [ 7.601866] usb 3-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 7.608660] mlx5_core 0000:01:00.0: irq 185 for MSI/MSI-X [ 7.608683] mlx5_core 0000:01:00.0: irq 186 for MSI/MSI-X [ 7.608703] mlx5_core 0000:01:00.0: irq 187 for MSI/MSI-X [ 7.608724] mlx5_core 0000:01:00.0: irq 188 for MSI/MSI-X [ 7.608742] mlx5_core 0000:01:00.0: irq 189 for MSI/MSI-X [ 7.608760] mlx5_core 0000:01:00.0: irq 190 for MSI/MSI-X [ 7.608779] mlx5_core 0000:01:00.0: irq 191 for MSI/MSI-X [ 7.608798] mlx5_core 0000:01:00.0: irq 192 for MSI/MSI-X [ 7.608818] mlx5_core 0000:01:00.0: irq 193 for MSI/MSI-X [ 7.608836] mlx5_core 0000:01:00.0: irq 194 for MSI/MSI-X [ 7.608854] mlx5_core 0000:01:00.0: irq 195 for MSI/MSI-X [ 7.608874] mlx5_core 0000:01:00.0: irq 196 for MSI/MSI-X [ 7.608892] mlx5_core 0000:01:00.0: irq 197 for MSI/MSI-X [ 7.608918] mlx5_core 0000:01:00.0: irq 198 for MSI/MSI-X [ 7.608943] mlx5_core 0000:01:00.0: irq 199 for MSI/MSI-X [ 7.608961] mlx5_core 0000:01:00.0: irq 200 for MSI/MSI-X [ 7.608979] mlx5_core 0000:01:00.0: irq 201 for MSI/MSI-X [ 7.608998] mlx5_core 0000:01:00.0: irq 202 for MSI/MSI-X [ 7.609017] mlx5_core 0000:01:00.0: irq 203 for MSI/MSI-X [ 7.609034] mlx5_core 0000:01:00.0: irq 204 for MSI/MSI-X [ 7.609053] mlx5_core 0000:01:00.0: irq 205 for MSI/MSI-X [ 7.609076] mlx5_core 0000:01:00.0: irq 206 for MSI/MSI-X [ 7.609102] mlx5_core 0000:01:00.0: irq 207 for MSI/MSI-X [ 7.609121] mlx5_core 0000:01:00.0: irq 208 for MSI/MSI-X [ 7.609139] mlx5_core 0000:01:00.0: irq 209 for MSI/MSI-X [ 7.609157] mlx5_core 0000:01:00.0: irq 210 for MSI/MSI-X [ 7.609177] mlx5_core 0000:01:00.0: irq 211 for MSI/MSI-X [ 7.609194] mlx5_core 0000:01:00.0: irq 212 for MSI/MSI-X [ 7.609218] mlx5_core 0000:01:00.0: irq 213 for MSI/MSI-X [ 7.609237] mlx5_core 0000:01:00.0: irq 214 for MSI/MSI-X [ 7.609257] mlx5_core 0000:01:00.0: irq 215 for MSI/MSI-X [ 7.609276] mlx5_core 0000:01:00.0: irq 216 for MSI/MSI-X [ 7.609295] mlx5_core 0000:01:00.0: irq 217 for MSI/MSI-X [ 7.609313] mlx5_core 0000:01:00.0: irq 218 for MSI/MSI-X [ 7.609333] mlx5_core 0000:01:00.0: irq 219 for MSI/MSI-X [ 7.609352] mlx5_core 0000:01:00.0: irq 220 for MSI/MSI-X [ 7.609372] mlx5_core 0000:01:00.0: irq 221 for MSI/MSI-X [ 7.609390] mlx5_core 0000:01:00.0: irq 222 for MSI/MSI-X [ 7.609409] mlx5_core 0000:01:00.0: irq 223 for MSI/MSI-X [ 7.609428] mlx5_core 0000:01:00.0: irq 224 for MSI/MSI-X [ 7.609446] mlx5_core 0000:01:00.0: irq 225 for MSI/MSI-X [ 7.609465] mlx5_core 0000:01:00.0: irq 226 for MSI/MSI-X [ 7.609484] mlx5_core 0000:01:00.0: irq 227 for MSI/MSI-X [ 7.609502] mlx5_core 0000:01:00.0: irq 228 for MSI/MSI-X [ 7.609520] mlx5_core 0000:01:00.0: irq 229 for MSI/MSI-X [ 7.609538] mlx5_core 0000:01:00.0: irq 230 for MSI/MSI-X [ 7.609557] mlx5_core 0000:01:00.0: irq 231 for MSI/MSI-X [ 7.609575] mlx5_core 0000:01:00.0: irq 232 for MSI/MSI-X [ 7.609593] mlx5_core 0000:01:00.0: irq 233 for MSI/MSI-X [ 7.610895] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged [ 7.611148] mlx5_core 0000:01:00.0: mlx5_pcie_event:303:(pid 319): PCIe slot advertised sufficient power (27W). [ 7.618949] mlx5_core 0000:01:00.0: mlx5_fw_tracer_start:776:(pid 294): FWTracer: Ownership granted and active [ 7.622394] hub 3-1.4:1.0: USB hub found [ 7.622737] hub 3-1.4:1.0: 4 ports detected [ 7.758374] mlx5_ib: Mellanox Connect-IB Infiniband driver v4.7-1.0.0 [ 7.982747] random: crng init done [ 9.561031] mpt3sas_cm0: hba_port entry: ffff8afd6858bbc0, port: 255 is added to hba_port list [ 9.572375] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00db90900), phys(21) [ 9.580983] mpt3sas_cm0: detecting: handle(0x0011), sas_address(0x300705b00db90900), phy(16) [ 9.589432] mpt3sas_cm0: REPORT_LUNS: handle(0x0011), retries(0) [ 9.598694] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0011), lun(0) [ 9.727352] scsi 1:0:0:0: Enclosure LSI VirtualSES 03 PQ: 0 ANSI: 7 [ 9.735476] scsi 1:0:0:0: set ignore_delay_remove for handle(0x0011) [ 9.741829] scsi 1:0:0:0: SES: handle(0x0011), sas_addr(0x300705b00db90900), phy(16), device_name(0x300705b00db90900) [ 9.752429] scsi 1:0:0:0: enclosure logical id(0x300605b00d110900), slot(16) [ 9.759561] scsi 1:0:0:0: enclosure level(0x0000), connector name( C3 ) [ 9.766267] scsi 1:0:0:0: serial_number(300605B00D110900) [ 9.771674] scsi 1:0:0:0: qdepth(1), tagged(0), simple(0), ordered(0), scsi_level(8), cmd_que(0) [ 9.780484] mpt3sas_cm0: log_info(0x31200206): originator(PL), code(0x20), sub_code(0x0206) [ 9.799310] mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984da0f9b20), phy(8) [ 9.807675] mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) [ 9.813824] mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(1) [ 9.820893] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) [ 9.827279] mpt3sas_cm0: detecting: handle(0x0017), sas_address(0x500a0984da0f9b20), phy(8) [ 9.835640] mpt3sas_cm0: REPORT_LUNS: handle(0x0017), retries(0) [ 9.842289] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0017), lun(0) [ 9.848977] scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 9.857731] scsi 1:0:1:0: SSP: handle(0x0017), sas_addr(0x500a0984da0f9b20), phy(8), device_name(0x500a0984da0f9b20) [ 9.868246] scsi 1:0:1:0: enclosure logical id(0x300605b00d110900), slot(5) [ 9.875290] scsi 1:0:1:0: enclosure level(0x0000), connector name( C1 ) [ 9.882009] scsi 1:0:1:0: serial_number(021812047179 ) [ 9.887411] scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.931905] scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 9.940076] scsi 1:0:1:1: SSP: handle(0x0017), sas_addr(0x500a0984da0f9b20), phy(8), device_name(0x500a0984da0f9b20) [ 9.950590] scsi 1:0:1:1: enclosure logical id(0x300605b00d110900), slot(5) [ 9.957634] scsi 1:0:1:1: enclosure level(0x0000), connector name( C1 ) [ 9.964355] scsi 1:0:1:1: serial_number(021812047179 ) [ 9.969755] scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 9.992175] scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.000338] scsi 1:0:1:2: SSP: handle(0x0017), sas_addr(0x500a0984da0f9b20), phy(8), device_name(0x500a0984da0f9b20) [ 10.010849] scsi 1:0:1:2: enclosure logical id(0x300605b00d110900), slot(5) [ 10.017896] scsi 1:0:1:2: enclosure level(0x0000), connector name( C1 ) [ 10.024615] scsi 1:0:1:2: serial_number(021812047179 ) [ 10.030016] scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.089177] scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 10.097421] scsi 1:0:1:31: SSP: handle(0x0017), sas_addr(0x500a0984da0f9b20), phy(8), device_name(0x500a0984da0f9b20) [ 10.108023] scsi 1:0:1:31: enclosure logical id(0x300605b00d110900), slot(5) [ 10.115156] scsi 1:0:1:31: enclosure level(0x0000), connector name( C1 ) [ 10.121963] scsi 1:0:1:31: serial_number(021812047179 ) [ 10.127449] scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.147504] mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa20c20), phy(0) [ 10.155852] mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) [ 10.161996] mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(1) [ 10.169788] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) [ 10.176102] mpt3sas_cm0: detecting: handle(0x0018), sas_address(0x500a0984dfa20c20), phy(0) [ 10.184468] mpt3sas_cm0: REPORT_LUNS: handle(0x0018), retries(0) [ 10.191148] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0018), lun(0) [ 10.197854] scsi 1:0:2:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.206081] scsi 1:0:2:0: SSP: handle(0x0018), sas_addr(0x500a0984dfa20c20), phy(0), device_name(0x500a0984dfa20c20) [ 10.216595] scsi 1:0:2:0: enclosure logical id(0x300605b00d110900), slot(13) [ 10.223728] scsi 1:0:2:0: enclosure level(0x0000), connector name( C3 ) [ 10.230444] scsi 1:0:2:0: serial_number(021825001558 ) [ 10.235847] scsi 1:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.258850] scsi 1:0:2:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.267044] scsi 1:0:2:1: SSP: handle(0x0018), sas_addr(0x500a0984dfa20c20), phy(0), device_name(0x500a0984dfa20c20) [ 10.277557] scsi 1:0:2:1: enclosure logical id(0x300605b00d110900), slot(13) [ 10.284689] scsi 1:0:2:1: enclosure level(0x0000), connector name( C3 ) [ 10.291411] scsi 1:0:2:1: serial_number(021825001558 ) [ 10.296810] scsi 1:0:2:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.319243] scsi 1:0:2:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 10.327499] scsi 1:0:2:31: SSP: handle(0x0018), sas_addr(0x500a0984dfa20c20), phy(0), device_name(0x500a0984dfa20c20) [ 10.338095] scsi 1:0:2:31: enclosure logical id(0x300605b00d110900), slot(13) [ 10.345316] scsi 1:0:2:31: enclosure level(0x0000), connector name( C3 ) [ 10.352121] scsi 1:0:2:31: serial_number(021825001558 ) [ 10.357607] scsi 1:0:2:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.383772] mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa1fa14), phy(4) [ 10.392135] mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) [ 10.398278] mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(1) [ 10.407829] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) [ 10.414147] mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa1fa14), phy(4) [ 10.422513] mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) [ 10.429298] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) [ 10.435631] mpt3sas_cm0: detecting: handle(0x0019), sas_address(0x500a0984dfa1fa14), phy(4) [ 10.443997] mpt3sas_cm0: REPORT_LUNS: handle(0x0019), retries(0) [ 10.450765] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0019), lun(0) [ 10.457307] scsi 1:0:3:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.465492] scsi 1:0:3:0: SSP: handle(0x0019), sas_addr(0x500a0984dfa1fa14), phy(4), device_name(0x500a0984dfa1fa14) [ 10.476003] scsi 1:0:3:0: enclosure logical id(0x300605b00d110900), slot(9) [ 10.483049] scsi 1:0:3:0: enclosure level(0x0000), connector name( C2 ) [ 10.489767] scsi 1:0:3:0: serial_number(021825001369 ) [ 10.495170] scsi 1:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.515932] scsi 1:0:3:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.524096] scsi 1:0:3:1: SSP: handle(0x0019), sas_addr(0x500a0984dfa1fa14), phy(4), device_name(0x500a0984dfa1fa14) [ 10.534608] scsi 1:0:3:1: enclosure logical id(0x300605b00d110900), slot(9) [ 10.541655] scsi 1:0:3:1: enclosure level(0x0000), connector name( C2 ) [ 10.548374] scsi 1:0:3:1: serial_number(021825001369 ) [ 10.553775] scsi 1:0:3:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.563007] scsi 1:0:3:1: Mode parameters changed [ 10.575640] sd 0:2:0:0: [sda] 233308160 512-byte logical blocks: (119 GB/111 GiB) [ 10.583315] sd 0:2:0:0: [sda] Write Protect is off [ 10.588115] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 [ 10.588164] sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA [ 10.592200] scsi 1:0:3:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 10.592280] scsi 1:0:3:31: SSP: handle(0x0019), sas_addr(0x500a0984dfa1fa14), phy(4), device_name(0x500a0984dfa1fa14) [ 10.592282] scsi 1:0:3:31: enclosure logical id(0x300605b00d110900), slot(9) [ 10.592283] scsi 1:0:3:31: enclosure level(0x0000), connector name( C2 ) [ 10.592284] scsi 1:0:3:31: serial_number(021825001369 ) [ 10.592286] scsi 1:0:3:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.646862] sda: sda1 sda2 sda3 [ 10.650515] sd 0:2:0:0: [sda] Attached SCSI disk [ 10.661717] mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984db2fa914), phy(12) [ 10.670159] mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) [ 10.676309] mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(1) [ 10.684933] mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) [ 10.691309] mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984db2fa914), phy(12) [ 10.699759] mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) [ 10.706731] mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) [ 10.713028] mpt3sas_cm0: detecting: handle(0x001a), sas_address(0x500a0984db2fa914), phy(12) [ 10.721476] mpt3sas_cm0: REPORT_LUNS: handle(0x001a), retries(0) [ 10.728246] mpt3sas_cm0: TEST_UNIT_READY: handle(0x001a), lun(0) [ 10.734823] scsi 1:0:4:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.743007] scsi 1:0:4:0: SSP: handle(0x001a), sas_addr(0x500a0984db2fa914), phy(12), device_name(0x500a0984db2fa914) [ 10.753606] scsi 1:0:4:0: enclosure logical id(0x300605b00d110900), slot(1) [ 10.760651] scsi 1:0:4:0: enclosure level(0x0000), connector name( C0 ) [ 10.767369] scsi 1:0:4:0: serial_number(021815000354 ) [ 10.772772] scsi 1:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.795944] scsi 1:0:4:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.804110] scsi 1:0:4:1: SSP: handle(0x001a), sas_addr(0x500a0984db2fa914), phy(12), device_name(0x500a0984db2fa914) [ 10.814708] scsi 1:0:4:1: enclosure logical id(0x300605b00d110900), slot(1) [ 10.821752] scsi 1:0:4:1: enclosure level(0x0000), connector name( C0 ) [ 10.828470] scsi 1:0:4:1: serial_number(021815000354 ) [ 10.833872] scsi 1:0:4:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.843060] scsi 1:0:4:1: Mode parameters changed [ 10.859206] scsi 1:0:4:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 10.867380] scsi 1:0:4:2: SSP: handle(0x001a), sas_addr(0x500a0984db2fa914), phy(12), device_name(0x500a0984db2fa914) [ 10.877976] scsi 1:0:4:2: enclosure logical id(0x300605b00d110900), slot(1) [ 10.885021] scsi 1:0:4:2: enclosure level(0x0000), connector name( C0 ) [ 10.891737] scsi 1:0:4:2: serial_number(021815000354 ) [ 10.897139] scsi 1:0:4:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 10.906327] scsi 1:0:4:2: Mode parameters changed [ 10.923207] scsi 1:0:4:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 10.931471] scsi 1:0:4:31: SSP: handle(0x001a), sas_addr(0x500a0984db2fa914), phy(12), device_name(0x500a0984db2fa914) [ 10.942153] scsi 1:0:4:31: enclosure logical id(0x300605b00d110900), slot(1) [ 10.949287] scsi 1:0:4:31: enclosure level(0x0000), connector name( C0 ) [ 10.956090] scsi 1:0:4:31: serial_number(021815000354 ) [ 10.961578] scsi 1:0:4:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 15.563098] mpt3sas_cm0: port enable: SUCCESS [ 15.568087] scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (unowned) [ 15.573687] sd 1:0:1:0: [sdb] 926167040 512-byte logical blocks: (474 GB/441 GiB) [ 15.581180] sd 1:0:1:0: [sdb] 4096-byte physical blocks [ 15.586634] scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (owned) [ 15.591982] sd 1:0:1:0: [sdb] Write Protect is off [ 15.596791] sd 1:0:1:0: [sdb] Mode Sense: 83 00 10 08 [ 15.596793] sd 1:0:1:1: [sdc] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.604883] sd 1:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.613417] scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (unowned) [ 15.613593] sd 1:0:1:1: [sdc] Write Protect is off [ 15.613595] sd 1:0:1:1: [sdc] Mode Sense: 83 00 10 08 [ 15.613760] sd 1:0:1:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.616745] sd 1:0:1:1: [sdc] Attached SCSI disk [ 15.616869] sd 1:0:1:0: [sdb] Attached SCSI disk [ 15.641494] sd 1:0:1:2: [sdd] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.649593] scsi 1:0:2:0: rdac: LUN 0 (IOSHIP) (unowned) [ 15.654999] sd 1:0:1:2: [sdd] Write Protect is off [ 15.655101] sd 1:0:2:0: [sde] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.655508] scsi 1:0:2:1: rdac: LUN 1 (IOSHIP) (owned) [ 15.655692] sd 1:0:2:0: [sde] Write Protect is off [ 15.655694] sd 1:0:2:0: [sde] Mode Sense: 83 00 10 08 [ 15.655791] sd 1:0:2:1: [sdf] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.655857] sd 1:0:2:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.656090] scsi 1:0:3:0: rdac: LUN 0 (IOSHIP) (owned) [ 15.656353] sd 1:0:3:0: [sdg] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.656448] sd 1:0:2:1: [sdf] Write Protect is off [ 15.656449] sd 1:0:2:1: [sdf] Mode Sense: 83 00 10 08 [ 15.656646] scsi 1:0:3:1: rdac: LUN 1 (IOSHIP) (unowned) [ 15.656647] sd 1:0:2:1: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.656900] sd 1:0:3:0: [sdg] Write Protect is off [ 15.656902] sd 1:0:3:0: [sdg] Mode Sense: 83 00 10 08 [ 15.656926] sd 1:0:3:1: [sdh] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.657079] sd 1:0:3:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.657247] scsi 1:0:4:0: rdac: LUN 0 (IOSHIP) (owned) [ 15.657486] sd 1:0:4:0: [sdi] 926167040 512-byte logical blocks: (474 GB/441 GiB) [ 15.657488] sd 1:0:4:0: [sdi] 4096-byte physical blocks [ 15.657732] sd 1:0:3:1: [sdh] Write Protect is off [ 15.657734] sd 1:0:3:1: [sdh] Mode Sense: 83 00 10 08 [ 15.657907] scsi 1:0:4:1: rdac: LUN 1 (IOSHIP) (unowned) [ 15.657988] sd 1:0:3:1: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.658047] sd 1:0:4:0: [sdi] Write Protect is off [ 15.658048] sd 1:0:4:0: [sdi] Mode Sense: 83 00 10 08 [ 15.658288] sd 1:0:4:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.658307] sd 1:0:4:1: [sdj] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.658659] scsi 1:0:4:2: rdac: LUN 2 (IOSHIP) (owned) [ 15.659032] sd 1:0:4:2: [sdk] 37449707520 512-byte logical blocks: (19.1 TB/17.4 TiB) [ 15.659065] sd 1:0:4:1: [sdj] Write Protect is off [ 15.659067] sd 1:0:4:1: [sdj] Mode Sense: 83 00 10 08 [ 15.659331] sd 1:0:4:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.659392] sd 1:0:2:1: [sdf] Attached SCSI disk [ 15.659739] sd 1:0:4:2: [sdk] Write Protect is off [ 15.659741] sd 1:0:4:2: [sdk] Mode Sense: 83 00 10 08 [ 15.659963] sd 1:0:4:2: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.660274] sd 1:0:2:0: [sde] Attached SCSI disk [ 15.660758] sd 1:0:3:0: [sdg] Attached SCSI disk [ 15.661878] sd 1:0:3:1: [sdh] Attached SCSI disk [ 15.662239] sd 1:0:4:0: [sdi] Attached SCSI disk [ 15.663258] sd 1:0:4:2: [sdk] Attached SCSI disk [ 15.663667] sd 1:0:4:1: [sdj] Attached SCSI disk [ 15.876207] sd 1:0:1:2: [sdd] Mode Sense: 83 00 10 08 [ 15.876371] sd 1:0:1:2: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.887704] sd 1:0:1:2: [sdd] Attached SCSI disk [ 15.971038] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 16.193836] systemd-journald[354]: Received SIGTERM from PID 1 (systemd). [ 16.221229] SELinux: Disabled at runtime. [ 16.225754] SELinux: Unregistering netfilter hooks [ 16.266120] type=1404 audit(1584552020.752:2): selinux=0 auid=4294967295 ses=4294967295 [ 16.295368] ip_tables: (C) 2000-2006 Netfilter Core Team [ 16.301909] systemd[1]: Inserted module 'ip_tables' [ 16.383997] EXT4-fs (sda2): re-mounted. Opts: (null) [ 16.395902] systemd-journald[4902]: Received request to flush runtime journal from PID 1 [ 16.450395] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 16.458074] piix4_smbus 0000:00:14.0: Using register 0x2e for SMBus port selection [ 16.477061] ACPI Error: No handler for Region [SYSI] (ffff8ae514b42a68) [IPMI] (20130517/evregion-162) [ 16.490196] ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) [ 16.500472] ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8ade29e795a0), AE_NOT_EXIST (20130517/psparse-536) [ 16.519350] ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8ade29e79500), AE_NOT_EXIST (20130517/psparse-536) [ 16.538288] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) [ 16.550948] ccp 0000:02:00.2: 3 command queues available [ 16.557044] ccp 0000:02:00.2: irq 235 for MSI/MSI-X [ 16.557067] ccp 0000:02:00.2: irq 236 for MSI/MSI-X [ 16.557131] ccp 0000:02:00.2: Queue 2 can access 4 LSB regions [ 16.564222] ccp 0000:02:00.2: Queue 3 can access 4 LSB regions [ 16.571462] ccp 0000:02:00.2: Queue 4 can access 4 LSB regions [ 16.578692] ccp 0000:02:00.2: Queue 0 gets LSB 4 [ 16.584705] ccp 0000:02:00.2: Queue 1 gets LSB 5 [ 16.590717] ccp 0000:02:00.2: Queue 2 gets LSB 6 [ 16.598361] cryptd: max_cpu_qlen set to 1000 [ 16.598527] ccp 0000:02:00.2: enabled [ 16.598845] ccp 0000:03:00.1: 5 command queues available [ 16.598902] ccp 0000:03:00.1: irq 238 for MSI/MSI-X [ 16.598954] ccp 0000:03:00.1: Queue 0 can access 7 LSB regions [ 16.598956] ccp 0000:03:00.1: Queue 1 can access 7 LSB regions [ 16.598958] ccp 0000:03:00.1: Queue 2 can access 7 LSB regions [ 16.598960] ccp 0000:03:00.1: Queue 3 can access 7 LSB regions [ 16.598962] ccp 0000:03:00.1: Queue 4 can access 7 LSB regions [ 16.598963] ccp 0000:03:00.1: Queue 0 gets LSB 1 [ 16.598964] ccp 0000:03:00.1: Queue 1 gets LSB 2 [ 16.598965] ccp 0000:03:00.1: Queue 2 gets LSB 3 [ 16.598966] ccp 0000:03:00.1: Queue 3 gets LSB 4 [ 16.598967] ccp 0000:03:00.1: Queue 4 gets LSB 5 [ 16.600450] ccp 0000:03:00.1: enabled [ 16.600741] ccp 0000:41:00.2: 3 command queues available [ 16.600833] ccp 0000:41:00.2: irq 240 for MSI/MSI-X [ 16.600859] ccp 0000:41:00.2: irq 241 for MSI/MSI-X [ 16.600932] ccp 0000:41:00.2: Queue 2 can access 4 LSB regions [ 16.600934] ccp 0000:41:00.2: Queue 3 can access 4 LSB regions [ 16.600936] ccp 0000:41:00.2: Queue 4 can access 4 LSB regions [ 16.600937] ccp 0000:41:00.2: Queue 0 gets LSB 4 [ 16.600938] ccp 0000:41:00.2: Queue 1 gets LSB 5 [ 16.600939] ccp 0000:41:00.2: Queue 2 gets LSB 6 [ 16.601582] ccp 0000:41:00.2: enabled [ 16.601761] ccp 0000:42:00.1: 5 command queues available [ 16.601878] ccp 0000:42:00.1: irq 243 for MSI/MSI-X [ 16.601954] ccp 0000:42:00.1: Queue 0 can access 7 LSB regions [ 16.601956] ccp 0000:42:00.1: Queue 1 can access 7 LSB regions [ 16.601958] ccp 0000:42:00.1: Queue 2 can access 7 LSB regions [ 16.601961] ccp 0000:42:00.1: Queue 3 can access 7 LSB regions [ 16.601964] ccp 0000:42:00.1: Queue 4 can access 7 LSB regions [ 16.601965] ccp 0000:42:00.1: Queue 0 gets LSB 1 [ 16.601967] ccp 0000:42:00.1: Queue 1 gets LSB 2 [ 16.601968] ccp 0000:42:00.1: Queue 2 gets LSB 3 [ 16.601969] ccp 0000:42:00.1: Queue 3 gets LSB 4 [ 16.601970] ccp 0000:42:00.1: Queue 4 gets LSB 5 [ 16.602577] ccp 0000:42:00.1: enabled [ 16.602791] ccp 0000:85:00.2: 3 command queues available [ 16.602852] ccp 0000:85:00.2: irq 245 for MSI/MSI-X [ 16.602878] ccp 0000:85:00.2: irq 246 for MSI/MSI-X [ 16.602927] ccp 0000:85:00.2: Queue 2 can access 4 LSB regions [ 16.602929] ccp 0000:85:00.2: Queue 3 can access 4 LSB regions [ 16.602931] ccp 0000:85:00.2: Queue 4 can access 4 LSB regions [ 16.602933] ccp 0000:85:00.2: Queue 0 gets LSB 4 [ 16.602934] ccp 0000:85:00.2: Queue 1 gets LSB 5 [ 16.602935] ccp 0000:85:00.2: Queue 2 gets LSB 6 [ 16.604661] ccp 0000:85:00.2: enabled [ 16.604930] ccp 0000:86:00.1: 5 command queues available [ 16.604986] ccp 0000:86:00.1: irq 248 for MSI/MSI-X [ 16.605018] ccp 0000:86:00.1: Queue 0 can access 7 LSB regions [ 16.605020] ccp 0000:86:00.1: Queue 1 can access 7 LSB regions [ 16.605022] ccp 0000:86:00.1: Queue 2 can access 7 LSB regions [ 16.605025] ccp 0000:86:00.1: Queue 3 can access 7 LSB regions [ 16.605027] ccp 0000:86:00.1: Queue 4 can access 7 LSB regions [ 16.605030] ccp 0000:86:00.1: Queue 0 gets LSB 1 [ 16.605031] ccp 0000:86:00.1: Queue 1 gets LSB 2 [ 16.605032] ccp 0000:86:00.1: Queue 2 gets LSB 3 [ 16.605034] ccp 0000:86:00.1: Queue 3 gets LSB 4 [ 16.605035] ccp 0000:86:00.1: Queue 4 gets LSB 5 [ 16.605655] ccp 0000:86:00.1: enabled [ 16.605869] ccp 0000:c2:00.2: 3 command queues available [ 16.605929] ccp 0000:c2:00.2: irq 250 for MSI/MSI-X [ 16.605955] ccp 0000:c2:00.2: irq 251 for MSI/MSI-X [ 16.606010] ccp 0000:c2:00.2: Queue 2 can access 4 LSB regions [ 16.606013] ccp 0000:c2:00.2: Queue 3 can access 4 LSB regions [ 16.606015] ccp 0000:c2:00.2: Queue 4 can access 4 LSB regions [ 16.606017] ccp 0000:c2:00.2: Queue 0 gets LSB 4 [ 16.606018] ccp 0000:c2:00.2: Queue 1 gets LSB 5 [ 16.606020] ccp 0000:c2:00.2: Queue 2 gets LSB 6 [ 16.606948] ccp 0000:c2:00.2: enabled [ 16.607074] ccp 0000:c3:00.1: 5 command queues available [ 16.607133] ccp 0000:c3:00.1: irq 253 for MSI/MSI-X [ 16.607163] ccp 0000:c3:00.1: Queue 0 can access 7 LSB regions [ 16.607165] ccp 0000:c3:00.1: Queue 1 can access 7 LSB regions [ 16.607167] ccp 0000:c3:00.1: Queue 2 can access 7 LSB regions [ 16.607169] ccp 0000:c3:00.1: Queue 3 can access 7 LSB regions [ 16.607171] ccp 0000:c3:00.1: Queue 4 can access 7 LSB regions [ 16.607172] ccp 0000:c3:00.1: Queue 0 gets LSB 1 [ 16.607173] ccp 0000:c3:00.1: Queue 1 gets LSB 2 [ 16.607175] ccp 0000:c3:00.1: Queue 2 gets LSB 3 [ 16.607176] ccp 0000:c3:00.1: Queue 3 gets LSB 4 [ 16.607177] ccp 0000:c3:00.1: Queue 4 gets LSB 5 [ 16.607910] ccp 0000:c3:00.1: enabled [ 16.652198] ipmi message handler version 39.2 [ 16.701191] input: PC Speaker as /devices/platform/pcspkr/input/input2 [ 16.752377] ipmi device interface [ 16.844913] device-mapper: uevent: version 1.0.3 [ 16.845083] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 17.109201] sd 0:2:0:0: Attached scsi generic sg0 type 0 [ 17.116234] scsi 1:0:0:0: Attached scsi generic sg1 type 13 [ 17.116450] sd 1:0:1:0: Attached scsi generic sg2 type 0 [ 17.116560] sd 1:0:1:1: Attached scsi generic sg3 type 0 [ 17.116615] sd 1:0:1:2: Attached scsi generic sg4 type 0 [ 17.116818] scsi 1:0:1:31: Attached scsi generic sg5 type 0 [ 17.116883] sd 1:0:2:0: Attached scsi generic sg6 type 0 [ 17.116960] sd 1:0:2:1: Attached scsi generic sg7 type 0 [ 17.117014] scsi 1:0:2:31: Attached scsi generic sg8 type 0 [ 17.117173] sd 1:0:3:0: Attached scsi generic sg9 type 0 [ 17.117578] sd 1:0:3:1: Attached scsi generic sg10 type 0 [ 17.117764] scsi 1:0:3:31: Attached scsi generic sg11 type 0 [ 17.117849] sd 1:0:4:0: Attached scsi generic sg12 type 0 [ 17.117922] sd 1:0:4:1: Attached scsi generic sg13 type 0 [ 17.118005] sd 1:0:4:2: Attached scsi generic sg14 type 0 [ 17.118057] scsi 1:0:4:31: Attached scsi generic sg15 type 0 [ 17.161457] IPMI System Interface driver [ 17.161482] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 17.161485] ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 [ 17.161486] ipmi_si: Adding SMBIOS-specified kcs state machine [ 17.161525] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI [ 17.161551] ipmi_si IPI0001:00: [io 0x0ca8] regsize 1 spacing 4 irq 10 [ 17.161553] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI [ 17.161553] ipmi_si: Adding ACPI-specified kcs state machine [ 17.161684] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 [ 17.230150] ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. [ 17.236244] ipmi_si IPI0001:00: Using irq 10 [ 17.259562] ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) [ 17.339571] ipmi_si IPI0001:00: IPMI kcs interface initialized [ 17.346997] sd 1:0:1:0: Embedded Enclosure Device [ 17.353191] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) [ 17.353206] AVX2 version of gcm_enc/dec engaged. [ 17.353206] AES CTR mode by8 optimization enabled [ 17.355363] sd 1:0:1:1: Embedded Enclosure Device [ 17.355693] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 17.355779] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 17.357488] sd 1:0:1:2: Embedded Enclosure Device [ 17.359586] scsi 1:0:1:31: Embedded Enclosure Device [ 17.361676] sd 1:0:2:0: Embedded Enclosure Device [ 17.363909] sd 1:0:2:1: Embedded Enclosure Device [ 17.366004] scsi 1:0:2:31: Embedded Enclosure Device [ 17.368093] sd 1:0:3:0: Embedded Enclosure Device [ 17.376322] sd 1:0:3:1: Embedded Enclosure Device [ 17.378418] scsi 1:0:3:31: Embedded Enclosure Device [ 17.380522] sd 1:0:4:0: Embedded Enclosure Device [ 17.382757] sd 1:0:4:1: Embedded Enclosure Device [ 17.384882] sd 1:0:4:2: Embedded Enclosure Device [ 17.387006] scsi 1:0:4:31: Embedded Enclosure Device [ 17.389148] ses 1:0:0:0: Attached Enclosure device [ 17.503384] kvm: Nested Paging enabled [ 17.510670] MCE: In-kernel MCE decoding enabled. [ 17.519576] AMD64 EDAC driver v3.4.0 [ 17.523180] EDAC amd64: DRAM ECC enabled. [ 17.527212] EDAC amd64: F17h detected (node 0). [ 17.531793] EDAC MC: UMC0 chip selects: [ 17.531795] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.536505] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.541218] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.545924] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.550632] EDAC MC: UMC1 chip selects: [ 17.550635] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.555346] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.560053] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.564766] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.569471] EDAC amd64: using x8 syndromes. [ 17.573659] EDAC amd64: MCT channel count: 2 [ 17.578108] EDAC MC0: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:18.3 [ 17.585514] EDAC amd64: DRAM ECC enabled. [ 17.589536] EDAC amd64: F17h detected (node 1). [ 17.594110] EDAC MC: UMC0 chip selects: [ 17.594112] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.598818] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.603525] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.608232] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.612947] EDAC MC: UMC1 chip selects: [ 17.612951] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.617659] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.622366] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.627073] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.631779] EDAC amd64: using x8 syndromes. [ 17.635965] EDAC amd64: MCT channel count: 2 [ 17.640361] EDAC MC1: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:19.3 [ 17.647765] EDAC amd64: DRAM ECC enabled. [ 17.651782] EDAC amd64: F17h detected (node 2). [ 17.656354] EDAC MC: UMC0 chip selects: [ 17.656356] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.661065] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.665771] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.670486] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.675194] EDAC MC: UMC1 chip selects: [ 17.675197] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.679906] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.684613] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.689326] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.694033] EDAC amd64: using x8 syndromes. [ 17.698221] EDAC amd64: MCT channel count: 2 [ 17.702626] EDAC MC2: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1a.3 [ 17.710029] EDAC amd64: DRAM ECC enabled. [ 17.714046] EDAC amd64: F17h detected (node 3). [ 17.718628] EDAC MC: UMC0 chip selects: [ 17.718629] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.723335] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.728042] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.732749] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.737456] EDAC MC: UMC1 chip selects: [ 17.737459] EDAC amd64: MC: 0: 0MB 1: 0MB [ 17.742170] EDAC amd64: MC: 2: 16383MB 3: 16383MB [ 17.746875] EDAC amd64: MC: 4: 0MB 5: 0MB [ 17.751580] EDAC amd64: MC: 6: 0MB 7: 0MB [ 17.756288] EDAC amd64: using x8 syndromes. [ 17.760474] EDAC amd64: MCT channel count: 2 [ 17.764871] EDAC MC3: Giving out device to 'amd64_edac' 'F17h': DEV 0000:00:1b.3 [ 17.772282] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.0' (POLLED) [ 42.400362] device-mapper: multipath round-robin: version 1.2.0 loaded [ 50.138218] Adding 4194300k swap on /dev/sda3. Priority:-2 extents:1 across:4194300k FS [ 50.180186] type=1305 audit(1584552054.665:3): audit_pid=10361 old=0 auid=4294967295 ses=4294967295 res=1 [ 50.200702] RPC: Registered named UNIX socket transport module. [ 50.207800] RPC: Registered udp transport module. [ 50.213890] RPC: Registered tcp transport module. [ 50.219984] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 50.846270] mlx5_core 0000:01:00.0: slow_pci_heuristic:5575:(pid 10644): Max link speed = 100000, PCI BW = 126016 [ 50.856593] mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 50.864873] mlx5_core 0000:01:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0) [ 51.416890] tg3 0000:81:00.0: irq 254 for MSI/MSI-X [ 51.416913] tg3 0000:81:00.0: irq 255 for MSI/MSI-X [ 51.416924] tg3 0000:81:00.0: irq 256 for MSI/MSI-X [ 51.416952] tg3 0000:81:00.0: irq 257 for MSI/MSI-X [ 51.416963] tg3 0000:81:00.0: irq 258 for MSI/MSI-X [ 51.543036] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [ 55.113422] tg3 0000:81:00.0 em1: Link is up at 1000 Mbps, full duplex [ 55.119954] tg3 0000:81:00.0 em1: Flow control is on for TX and on for RX [ 55.126746] tg3 0000:81:00.0 em1: EEE is enabled [ 55.131381] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 55.906269] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 56.188412] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 60.322244] FS-Cache: Loaded [ 60.351515] FS-Cache: Netfs 'nfs' registered for caching [ 60.361051] Key type dns_resolver registered [ 60.390841] NFS: Registering the id_resolver key type [ 60.397391] Key type id_resolver registered [ 60.402926] Key type id_legacy registered [ 199.839177] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [ 199.846744] alg: No test for adler32 (adler32-zlib) [ 200.646830] Lustre: Lustre: Build Version: 2.12.4 [ 200.750653] LNet: 20171:0:(config.c:1627:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down [ 200.760432] LNet: Using FastReg for registration [ 200.777380] LNet: Added LNI 10.0.10.52@o2ib7 [8/256/0/180] [ 308.753055] LNetError: 20215:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [ 308.763230] LNetError: 20215:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.216@o2ib7 (6): c: 7, oc: 0, rc: 8 [ 308.785216] LNetError: 20224:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.216@o2ib7 added to recovery queue. Health = 900 [ 411.433064] LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 [ 411.518678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 411.970062] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.49.26.15@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.987430] LustreError: Skipped 1 previous similar message [ 412.050602] Lustre: fir-MDT0001: Not available for connect from 10.50.7.56@o2ib2 (not set up) [ 412.364435] Lustre: fir-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 412.375828] Lustre: 20423:0:(llog_cat.c:1059:llog_cat_reverse_process()) fir-MDD0001: catalog [0x5:0xa:0x0] crosses index zero [ 412.504767] Lustre: fir-MDD0001: changelog on [ 412.509142] Lustre: 20423:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x5:0xa:0x0] crosses index zero [ 412.524435] Lustre: fir-MDT0001: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 413.531025] Lustre: fir-MDT0001: Connection restored to e0b3c403-4bb2-4 (at 10.50.14.6@o2ib2) [ 413.539563] Lustre: Skipped 90 previous similar messages [ 414.042375] Lustre: fir-MDT0001: Connection restored to e361cff3-f0f6-4 (at 10.50.9.49@o2ib2) [ 414.050903] Lustre: Skipped 134 previous similar messages [ 415.042327] Lustre: fir-MDT0001: Connection restored to b04d2d5e-f4ea-4 (at 10.49.26.23@o2ib1) [ 415.050940] Lustre: Skipped 103 previous similar messages [ 417.042845] Lustre: fir-MDT0001: Connection restored to 80cdbcdc-4411-4 (at 10.50.10.70@o2ib2) [ 417.051456] Lustre: Skipped 208 previous similar messages [ 421.043355] Lustre: fir-MDT0001: Connection restored to 3e1a7dd1-f48f-4 (at 10.50.2.17@o2ib2) [ 421.051880] Lustre: Skipped 588 previous similar messages [ 499.689451] Lustre: fir-MDT0001: Connection restored to fir-MDT0001-lwp-OST001e_UUID (at 10.0.10.105@o2ib7) [ 499.699192] Lustre: Skipped 152 previous similar messages [ 504.799579] Lustre: fir-MDT0001: Recovery over after 1:33, of 1290 clients 1290 recovered and 0 were evicted. [ 504.831276] Lustre: 21096:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 504.844351] LustreError: 21097:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 505.330811] Lustre: 21239:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 505.342810] Lustre: 21239:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 7461 previous similar messages [ 505.352159] LustreError: 21217:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 505.362195] LustreError: 21217:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 7778 previous similar messages [ 506.330830] Lustre: 21135:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 506.342839] Lustre: 21135:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 19205 previous similar messages [ 506.352713] LustreError: 21218:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 506.362720] LustreError: 21218:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 16801 previous similar messages [ 508.330828] Lustre: 21176:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 508.342830] Lustre: 21176:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 40185 previous similar messages [ 508.353020] LustreError: 21177:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 508.363033] LustreError: 21177:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 34114 previous similar messages [ 512.330888] Lustre: 21030:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 512.342885] Lustre: 21030:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 79423 previous similar messages [ 512.352917] LustreError: 21213:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 512.362931] LustreError: 21213:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 67416 previous similar messages [ 520.331132] Lustre: 21084:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 520.343129] Lustre: 21084:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 159923 previous similar messages [ 520.353111] LustreError: 21179:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 520.363117] LustreError: 21179:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 137757 previous similar messages [ 524.957681] LustreError: 21095:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = 'alignment.eigen.indiv', t = [0x240049459:0x9f67:0x0], p = [0x2400478b1:0x1e9d6:0x0]: rc = -28 [ 530.656413] LustreError: 20878:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 6, name = 'alignment.eigen.indiv', t = [0x240049419:0xea19:0x0], p = [0x24004ac39:0x22e6:0x0]: rc = -5 [ 530.676655] LustreError: 20878:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 2 previous similar messages [ 531.887370] LustreError: 20887:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = 'alignment.eigen.indiv', t = [0x240049419:0xea1f:0x0], p = [0x24004ac39:0x22e6:0x0]: rc = -28 [ 531.907696] LustreError: 20887:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 2 previous similar messages [ 534.437187] LustreError: 21091:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = '.state.rob002.nwwUCh', t = [0x24003e2c1:0x1a4be:0x0], p = [0x24003e2c1:0x1a2cd:0x0]: rc = -28 [ 534.457604] LustreError: 21091:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 476 previous similar messages [ 536.331595] Lustre: 21208:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 536.343598] Lustre: 21208:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 318896 previous similar messages [ 536.353580] LustreError: 21233:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 536.363592] LustreError: 21233:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 278389 previous similar messages [ 538.793759] LustreError: 20903:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = '.state.rob008.lIDtrw', t = [0x24003e2c1:0x1a4c4:0x0], p = [0x24003e2c1:0x1a2cd:0x0]: rc = -5 [ 538.814093] LustreError: 20903:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 4 previous similar messages [ 546.814069] LustreError: 20826:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = '.state.rob019.9XaMV2', t = [0x24003e2c1:0x1a4cf:0x0], p = [0x24003e2c1:0x1a2cd:0x0]: rc = -28 [ 546.834485] LustreError: 20826:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 10 previous similar messages [ 562.867247] LustreError: 20872:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = '.state.rob038.uqeKld', t = [0x24003e2c1:0x1a702:0x0], p = [0x24003e2c1:0x1a4dd:0x0]: rc = -28 [ 562.887661] LustreError: 20872:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 536 previous similar messages [ 568.332611] Lustre: 21146:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 568.344609] Lustre: 21146:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 632422 previous similar messages [ 568.354939] LustreError: 21214:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 568.364951] LustreError: 21214:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 550759 previous similar messages [ 595.094431] LustreError: 20852:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) fir-MDD0001: cannot store changelog record: type = 1, name = '.state.rob144.edbipg', t = [0x24003e2c1:0x1a797:0x0], p = [0x24003e2c1:0x1a4dd:0x0]: rc = -5 [ 595.114751] LustreError: 20852:0:(mdd_dir.c:1065:mdd_changelog_ns_store()) Skipped 154 previous similar messages [ 632.334659] Lustre: 21210:0:(llog_cat.c:98:llog_cat_new_log()) fir-MDD0001: there are no more free slots in catalog changelog_catalog [ 632.346653] Lustre: 21210:0:(llog_cat.c:98:llog_cat_new_log()) Skipped 1243835 previous similar messages [ 632.356529] LustreError: 20480:0:(llog_cat.c:530:llog_cat_current_log()) fir-MDD0001: next log does not exist! [ 632.366527] LustreError: 20480:0:(llog_cat.c:530:llog_cat_current_log()) Skipped 1115377 previous similar messages [ 637.373911] Lustre: Failing over fir-MDT0001 [ 637.538894] Lustre: fir-MDT0001: Not available for connect from 10.49.0.71@o2ib1 (stopping) [ 637.547259] Lustre: Skipped 9 previous similar messages [ 638.564213] Lustre: fir-MDT0001: Not available for connect from 10.50.2.1@o2ib2 (stopping) [ 638.572489] Lustre: Skipped 91 previous similar messages [ 640.572209] Lustre: fir-MDT0001: Not available for connect from 10.49.25.11@o2ib1 (stopping) [ 640.580646] Lustre: Skipped 269 previous similar messages [ 641.349649] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.49.18.32@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.367024] LustreError: Skipped 9 previous similar messages [ 641.885493] Lustre: server umount fir-MDT0001 complete [ 642.798839] LNetError: 20284:0:(o2iblnd_cb.c:2496:kiblnd_passive_connect()) Can't accept conn from 10.0.10.224@o2ib7 on NA (ib0:1:10.0.10.52): bad dst nid 10.0.10.52@o2ib7 [ 643.310440] LNetError: 21557:0:(o2iblnd_cb.c:2496:kiblnd_passive_connect()) Can't accept conn from 10.0.10.218@o2ib7 on NA (ib0:1:10.0.10.52): bad dst nid 10.0.10.52@o2ib7 [ 643.325744] LNetError: 21557:0:(o2iblnd_cb.c:2496:kiblnd_passive_connect()) Skipped 21 previous similar messages [ 644.800599] LNet: Removed LNI 10.0.10.52@o2ib7 [ 731.122093] LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 [ 731.208724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 908.220225] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [ 908.227858] alg: No test for adler32 (adler32-zlib) [ 909.028572] Lustre: Lustre: Build Version: 2.12.4 [ 909.132162] LNet: 21811:0:(config.c:1627:lnet_inet_enumerate()) lnet: Ignoring interface em2: it's down [ 909.141782] LNet: Using FastReg for registration [ 909.157585] LNet: Added LNI 10.0.10.52@o2ib7 [8/256/0/180] [ 910.343024] LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 [ 910.429954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 911.334090] Lustre: fir-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 911.346291] Lustre: fir-MDT0001: in recovery but waiting for the first client to connect [ 912.807358] Lustre: fir-MDT0001: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 913.816154] Lustre: fir-MDT0001: Connection restored to (at 10.50.4.10@o2ib2) [ 916.346313] Lustre: fir-MDT0001: Connection restored to dc2c90e2-2fae-4 (at 10.49.7.10@o2ib1) [ 916.354858] Lustre: Skipped 2 previous similar messages [ 917.347015] Lustre: fir-MDT0001: Connection restored to 79513c1e-38af-4 (at 10.50.1.12@o2ib2) [ 917.355540] Lustre: Skipped 262 previous similar messages [ 919.346610] Lustre: fir-MDT0001: Connection restored to 7a4b5ab1-0a05-4 (at 10.50.10.40@o2ib2) [ 919.355224] Lustre: Skipped 519 previous similar messages [ 925.853335] Lustre: fir-MDT0001: Connection restored to 9bb420b9-4b7e-4 (at 10.49.8.24@o2ib1) [ 925.861876] Lustre: Skipped 484 previous similar messages [ 952.450239] Lustre: fir-MDT0001: Connection restored to 95eea94a-c8bd-4 (at 10.50.6.1@o2ib2) [ 977.094182] Lustre: fir-MDT0001: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.53@o2ib7) [ 977.103427] Lustre: Skipped 94 previous similar messages [ 977.141845] Lustre: fir-MDT0001: Recovery over after 1:04, of 1290 clients 1290 recovered and 0 were evicted. [ 1256.698668] LustreError: 11-0: fir-MDT0002-osp-MDT0001: operation mds_statfs to node 10.0.10.53@o2ib7 failed: rc = -107 [ 1256.709459] Lustre: fir-MDT0002-osp-MDT0001: Connection to fir-MDT0002 (at 10.0.10.53@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 1326.252157] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.3@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.927146] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.6.28@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.685554] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1328.771290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 1328.907069] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.7.59@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.683859] Lustre: fir-MDT0002: Not available for connect from 10.50.1.10@o2ib2 (not set up) [ 1330.367735] Lustre: fir-MDT0002: Not available for connect from 10.0.10.51@o2ib7 (not set up) [ 1330.376261] Lustre: Skipped 1 previous similar message [ 1331.417567] Lustre: fir-MDT0002: Not available for connect from 10.0.10.113@o2ib7 (not set up) [ 1331.426189] Lustre: Skipped 25 previous similar messages [ 1331.542791] Lustre: fir-MDT0001: Connection restored to fir-MDT0002-mdtlov_UUID (at 0@lo) [ 1331.550982] Lustre: Skipped 7 previous similar messages [ 1331.842596] Lustre: fir-MDT0002: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 1332.016545] Lustre: fir-MDD0002: changelog on [ 1332.028636] Lustre: fir-MDT0002: in recovery but waiting for the first client to connect [ 1332.073057] Lustre: fir-MDT0002: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 1430.722510] Lustre: fir-MDT0002: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 1430.731131] Lustre: Skipped 1376 previous similar messages [ 1431.893954] Lustre: fir-MDT0002: Recovery over after 1:40, of 1290 clients 1290 recovered and 0 were evicted. [ 3222.730973] Lustre: Failing over fir-MDT0002 [ 3222.767144] Lustre: fir-MDT0002: Not available for connect from 10.50.4.26@o2ib2 (stopping) [ 3222.775500] Lustre: Skipped 2 previous similar messages [ 3223.286675] Lustre: fir-MDT0002: Not available for connect from 10.50.8.2@o2ib2 (stopping) [ 3223.294946] Lustre: Skipped 119 previous similar messages [ 3224.291658] Lustre: fir-MDT0002: Not available for connect from 10.50.10.49@o2ib2 (stopping) [ 3224.300099] Lustre: Skipped 176 previous similar messages [ 3224.800652] LustreError: 11-0: fir-MDT0002-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3224.810404] Lustre: fir-MDT0002-osp-MDT0001: Connection to fir-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.312907] Lustre: fir-MDT0002: Not available for connect from 10.50.2.7@o2ib2 (stopping) [ 3226.321182] Lustre: Skipped 289 previous similar messages [ 3230.643415] Lustre: fir-MDT0002: Not available for connect from 10.50.6.17@o2ib2 (stopping) [ 3230.651767] Lustre: Skipped 125 previous similar messages [ 3230.720972] LustreError: 22568:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.13.9@o2ib2 arrived at 1584555234 with bad export cookie 13699018698230690058 [ 3230.736522] LustreError: 22568:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3233.053872] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.7.66@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.071157] LustreError: Skipped 1 previous similar message [ 3233.091450] Lustre: server umount fir-MDT0002 complete [ 3233.963037] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.4.2@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.968970] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.9.72@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.986249] LustreError: Skipped 3 previous similar messages [ 3237.066184] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.7.34@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.083470] LustreError: Skipped 6 previous similar messages [ 3241.072136] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.8.24@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.089423] LustreError: Skipped 151 previous similar messages [ 3244.293339] Lustre: fir-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [ 3244.301959] Lustre: Skipped 8 previous similar messages [ 3249.517385] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.6.29@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.534668] LustreError: Skipped 483 previous similar messages [ 3284.663654] Lustre: fir-MDT0002-osp-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [ 3400.181442] LustreError: 11-0: fir-MDT0003-osp-MDT0001: operation mds_statfs to node 10.0.10.54@o2ib7 failed: rc = -107 [ 3400.192234] Lustre: fir-MDT0003-osp-MDT0001: Connection to fir-MDT0003 (at 10.0.10.54@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 3443.575752] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.4.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.593031] LustreError: Skipped 2 previous similar messages [ 3479.128062] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.62@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.145340] LustreError: Skipped 1383 previous similar messages [ 3532.738794] Lustre: fir-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [ 3548.177848] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.32@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.287959] Lustre: fir-MDT0003-osp-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [ 5611.506941] LustreError: 11-0: fir-MDT0003-osp-MDT0001: operation mds_statfs to node 10.0.10.53@o2ib7 failed: rc = -107 [ 5611.517726] Lustre: fir-MDT0003-osp-MDT0001: Connection to fir-MDT0003 (at 10.0.10.53@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 5626.958568] Lustre: fir-MDT0001: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [ 5655.658556] Lustre: fir-MDT0003-osp-MDT0001: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [ 5784.039806] LustreError: 11-0: fir-MDT0000-osp-MDT0001: operation mds_statfs to node 10.0.10.51@o2ib7 failed: rc = -107 [ 5784.050606] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 5797.624198] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 5804.065093] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.12@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.082378] LustreError: Skipped 1 previous similar message [ 5820.107515] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.63@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.124803] LustreError: Skipped 477 previous similar messages [ 5904.419952] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.12@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.437241] LustreError: Skipped 908 previous similar messages [ 6004.358902] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [ 6004.444793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 6004.774763] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.12@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.792064] LustreError: Skipped 1388 previous similar messages [ 6006.364017] Lustre: fir-MDT0000: Not available for connect from 10.50.1.5@o2ib2 (not set up) [ 6006.372459] Lustre: Skipped 4 previous similar messages [ 6006.425119] Lustre: fir-MDT0001: Connection restored to 10.0.10.52@o2ib7 (at 0@lo) [ 6006.642664] Lustre: fir-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 6006.886318] Lustre: fir-MDD0000: changelog on [ 6006.906722] Lustre: fir-MDT0000: in recovery but waiting for the first client to connect [ 6006.970182] Lustre: fir-MDT0000: Will be in recovery for at least 2:30, or until 1290 clients reconnect [ 6019.342147] LustreError: 22533:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b0cedae0d80 x1661526579728960/t0(0) o601->fir-MDT0000-lwp-MDT0002_UUID@10.0.10.53@o2ib7:268/0 lens 336/0 e 0 to 0 dl 1584558028 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6019.368011] LustreError: 22533:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 38 previous similar messages [ 6023.332027] LustreError: 22344:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8add39c01f80 x1652476184025616/t0(0) o601->fir-MDT0000-lwp-OST0050_UUID@10.0.10.113@o2ib7:466/0 lens 336/0 e 0 to 0 dl 1584558226 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6023.357988] LustreError: 22344:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 20 previous similar messages [ 6024.331861] LustreError: 22594:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8add39c00000 x1652476184173472/t0(0) o601->fir-MDT0000-lwp-OST004a_UUID@10.0.10.113@o2ib7:467/0 lens 336/0 e 0 to 0 dl 1584558227 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6024.357818] LustreError: 22594:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 1841 previous similar messages [ 6026.335584] LustreError: 22534:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afbc6781b00 x1660618473253904/t0(0) o601->fir-MDT0000-lwp-OST0021_UUID@10.0.10.106@o2ib7:275/0 lens 336/0 e 0 to 0 dl 1584558035 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6026.361538] LustreError: 22534:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 377 previous similar messages [ 6030.338137] LustreError: 22514:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8aed70839680 x1652572522238656/t0(0) o601->fir-MDT0000-lwp-OST0015_UUID@10.0.10.104@o2ib7:279/0 lens 336/0 e 0 to 0 dl 1584558039 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6030.364081] LustreError: 22514:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 9962 previous similar messages [ 6031.679724] LustreError: 167-0: fir-MDT0000-lwp-MDT0001: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [ 6038.349771] LustreError: 22531:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afbd1a9f980 x1660618473263440/t0(0) o601->fir-MDT0000-lwp-OST001d_UUID@10.0.10.106@o2ib7:287/0 lens 336/0 e 0 to 0 dl 1584558047 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 6038.375735] LustreError: 22531:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 5804 previous similar messages [ 6041.869739] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [ 6041.869740] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [ 6041.889564] LustreError: Skipped 55 previous similar messages [ 6052.326020] Lustre: fir-MDT0000: Recovery over after 0:45, of 1290 clients 1290 recovered and 0 were evicted. [ 6062.274420] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [ 6062.284591] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (49): c: 7, oc: 0, rc: 8 [ 6062.296793] LNetError: 21867:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 [ 6063.086717] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6063.863464] Lustre: 21903:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1584558060/real 0] req@ffff8afc3c713600 x1661524266554368/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1584558067 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6063.890721] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 6064.086630] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6065.086768] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6114.049204] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6134.094601] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6134.106537] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [ 6164.226544] LNetError: 23422:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6189.277983] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6189.288088] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6189.300002] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [ 6220.278855] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6227.279042] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [ 6227.289139] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6227.301069] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [ 6233.279220] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6239.279397] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6270.280249] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6290.281799] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [ 6290.291881] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [ 6339.282194] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6339.292296] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6339.304212] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 6 previous similar messages [ 6420.284448] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6420.294534] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 4 previous similar messages [ 6489.286383] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6489.298297] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 8 previous similar messages [ 6639.290548] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6639.300632] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 7 previous similar messages [ 6747.293551] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 6747.305466] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 14 previous similar messages [ 6939.298950] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [ 6939.309037] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 14 previous similar messages [ 7261.307964] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 7261.319880] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 26 previous similar messages [ 7749.189046] Lustre: Failing over fir-MDT0000 [ 7749.211707] LustreError: 23643:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.110@o2ib7 arrived at 1584559752 with bad export cookie 13699018698713817514 [ 7749.211710] LustreError: 21964:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.110@o2ib7 arrived at 1584559752 with bad export cookie 13699018698713817563 [ 7749.211839] LustreError: 21962:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffff8add62b98900/0xbe1cb0ed0cc8ed99 lrc: 4/0,0 mode: CR/CR res: [0x200000006:0x2020000:0x59e36].0x0 rrc: 78 type: PLN flags: 0x54801000000000 nid: 10.0.10.110@o2ib7 remote: 0xd45f74c9f5bc313d expref: 135 pid: 42123 timeout: 0 lvb_type: 2 [ 7749.212095] Lustre: fir-MDT0000: Not available for connect from 10.0.10.110@o2ib7 (stopping) [ 7749.212096] Lustre: fir-MDT0000: Not available for connect from 10.0.10.110@o2ib7 (stopping) [ 7749.212096] Lustre: fir-MDT0000: Not available for connect from 10.0.10.110@o2ib7 (stopping) [ 7749.212097] Lustre: Skipped 1 previous similar message [ 7749.212098] Lustre: Skipped 1 previous similar message [ 7749.212099] Lustre: Skipped 1 previous similar message [ 7749.314737] LustreError: 23643:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 18 previous similar messages [ 7749.731930] Lustre: fir-MDT0000: Not available for connect from 10.50.5.19@o2ib2 (stopping) [ 7749.740288] Lustre: Skipped 88 previous similar messages [ 7750.019182] LustreError: 11-0: fir-MDT0000-osp-MDT0001: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 7750.029114] LustreError: Skipped 21 previous similar messages [ 7750.034876] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.608869] LustreError: 21967:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.112@o2ib7 arrived at 1584559754 with bad export cookie 13699018698713796990 [ 7750.624509] LustreError: 21967:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 64 previous similar messages [ 7750.792846] Lustre: fir-MDT0000: Not available for connect from 10.50.10.14@o2ib2 (stopping) [ 7750.801286] Lustre: Skipped 133 previous similar messages [ 7752.134873] LustreError: 22370:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.116@o2ib7 arrived at 1584559755 with bad export cookie 13699018698713381225 [ 7752.150515] LustreError: 22370:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 20 previous similar messages [ 7752.975391] Lustre: fir-MDT0000: Not available for connect from 10.50.7.20@o2ib2 (stopping) [ 7752.983746] Lustre: Skipped 113 previous similar messages [ 7753.191918] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.17.34@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7753.668824] Lustre: server umount fir-MDT0000 complete [ 7754.765062] LustreError: 23643:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.113@o2ib7 arrived at 1584559758 with bad export cookie 13699018698713168635 [ 7754.780699] LustreError: 23643:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 64 previous similar messages [ 7759.844015] LustreError: 23642:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.102@o2ib7 arrived at 1584559763 with bad export cookie 13699018698713318827 [ 7759.859657] LustreError: 23642:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 42 previous similar messages [ 7769.904245] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.919299] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.935537] LustreError: Skipped 843 previous similar messages [ 7810.442600] Lustre: fir-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 7810.451214] Lustre: Skipped 1391 previous similar messages [ 7846.414270] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.20.20@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.431641] LustreError: Skipped 162 previous similar messages [ 7895.346808] LustreError: 167-0: fir-MDT0000-lwp-MDT0001: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [ 7895.367380] Lustre: fir-MDT0000-lwp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 7945.524193] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x65ffc3416476acb6 to 0x22d27906083e0c79 [ 7945.536238] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [ 7945.545550] Lustre: Skipped 1 previous similar message [ 8954.573331] LustreError: 11-0: fir-OST0004-osc-MDT0001: operation ost_destroy to node 10.0.10.101@o2ib7 failed: rc = -107 [ 8954.584292] Lustre: fir-OST0004-osc-MDT0001: Connection to fir-OST0004 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 8955.152536] LustreError: 11-0: fir-OST0002-osc-MDT0001: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 [ 8955.152543] Lustre: fir-OST0008-osc-MDT0001: Connection to fir-OST0008 (at 10.0.10.101@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 8955.179469] LustreError: Skipped 2 previous similar messages [ 8981.089235] LustreError: 11-0: fir-OST001a-osc-MDT0001: operation ost_statfs to node 10.0.10.105@o2ib7 failed: rc = -107 [ 8981.100103] LustreError: Skipped 1 previous similar message [ 8981.105689] Lustre: fir-OST001a-osc-MDT0001: Connection to fir-OST001a (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 8981.121782] Lustre: Skipped 4 previous similar messages [ 8989.249382] Lustre: 21876:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584560985/real 1584560985] req@ffff8add52eab180 x1661524333038912/t0(0) o13->fir-OST0022-osc-MDT0001@10.0.10.105@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584560992 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 8989.277502] Lustre: fir-OST0022-osc-MDT0001: Connection to fir-OST0022 (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 8989.293573] Lustre: Skipped 3 previous similar messages [ 8990.265405] Lustre: 21877:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584560986/real 1584560986] req@ffff8add75cb0480 x1661524333039232/t0(0) o13->fir-OST001e-osc-MDT0001@10.0.10.105@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584560993 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9010.338054] LustreError: 11-0: fir-OST0038-osc-MDT0001: operation ost_statfs to node 10.0.10.109@o2ib7 failed: rc = -107 [ 9010.348923] LustreError: Skipped 3 previous similar messages [ 9010.354597] Lustre: fir-OST0038-osc-MDT0001: Connection to fir-OST0038 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9010.370677] Lustre: Skipped 1 previous similar message [ 9016.357152] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 14 seconds [ 9016.367408] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 13 previous similar messages [ 9016.376818] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 9034.903214] Lustre: fir-MDT0001: Client 725911fc-a014-4 (at 10.50.17.7@o2ib2) reconnecting [ 9034.911500] Lustre: fir-MDT0001: Connection restored to 725911fc-a014-4 (at 10.50.17.7@o2ib2) [ 9042.962588] Lustre: fir-MDT0001: Client 4d904238-d00c-4 (at 10.50.9.46@o2ib2) reconnecting [ 9042.970885] Lustre: fir-MDT0001: Connection restored to 4d904238-d00c-4 (at 10.50.9.46@o2ib2) [ 9045.430208] Lustre: fir-MDT0001: Client 541f81d4-bd4f-4 (at 10.50.7.3@o2ib2) reconnecting [ 9045.438551] Lustre: Skipped 1 previous similar message [ 9049.663906] Lustre: fir-MDT0001: Client ccea6ca8-94f6-4 (at 10.50.15.3@o2ib2) reconnecting [ 9050.381163] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.30.1@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.398442] LustreError: Skipped 736 previous similar messages [ 9054.169181] Lustre: fir-MDT0001: Client 11210b27-8de6-4 (at 10.49.26.6@o2ib1) reconnecting [ 9054.177445] Lustre: Skipped 1 previous similar message [ 9054.182602] Lustre: fir-MDT0001: Connection restored to 11210b27-8de6-4 (at 10.49.26.6@o2ib1) [ 9054.191137] Lustre: Skipped 4 previous similar messages [ 9055.331358] LustreError: 11-0: fir-OST004e-osc-MDT0001: operation ost_statfs to node 10.0.10.113@o2ib7 failed: rc = -107 [ 9055.342235] LustreError: Skipped 5 previous similar messages [ 9055.347905] Lustre: fir-OST004e-osc-MDT0001: Connection to fir-OST004e (at 10.0.10.113@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9055.364007] Lustre: Skipped 5 previous similar messages [ 9063.762483] Lustre: fir-MDT0001: Client 17f2724d-4d81-4 (at 10.50.5.63@o2ib2) reconnecting [ 9063.770752] Lustre: Skipped 9 previous similar messages [ 9070.287430] Lustre: fir-MDT0001: Connection restored to db7814de-e91d-4 (at 10.50.7.2@o2ib2) [ 9070.295877] Lustre: Skipped 57 previous similar messages [ 9070.496696] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.18@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9079.989064] Lustre: fir-MDT0001: Client bd34050e-30d7-4 (at 10.50.10.66@o2ib2) reconnecting [ 9079.997421] Lustre: Skipped 93 previous similar messages [ 9084.359083] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 0 seconds [ 9084.369257] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 70 previous similar messages [ 9089.489703] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.19.3@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9089.506986] LustreError: Skipped 5 previous similar messages [ 9102.321859] Lustre: fir-MDT0001: Connection restored to e7e26766-3907-4 (at 10.50.3.36@o2ib2) [ 9102.330393] Lustre: Skipped 79 previous similar messages [ 9113.341236] Lustre: fir-MDT0001: Client 6479814a-e93b-4 (at 10.50.2.67@o2ib2) reconnecting [ 9113.349505] Lustre: Skipped 76 previous similar messages [ 9122.862630] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.2.54@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.879910] LustreError: Skipped 23 previous similar messages [ 9141.360699] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 9141.372608] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 17 previous similar messages [ 9167.315444] Lustre: fir-MDT0001: Connection restored to 25f82366-6817-4 (at 10.50.3.16@o2ib2) [ 9167.323970] Lustre: Skipped 533 previous similar messages [ 9177.718911] Lustre: fir-MDT0001: Client e7e26766-3907-4 (at 10.50.3.36@o2ib2) reconnecting [ 9177.727178] Lustre: Skipped 542 previous similar messages [ 9186.989857] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.15.6@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9187.007143] LustreError: Skipped 1722 previous similar messages [ 9216.362794] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.101@o2ib7: 4 seconds [ 9216.372965] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 162 previous similar messages [ 9260.603510] Lustre: fir-MDT0001: haven't heard from client 5b88c4a2-39df-4 (at 10.50.17.45@o2ib2) in 194 seconds. I think it's dead, and I am evicting it. exp ffff8add5c65b800, cur 1584561264 expire 1584561114 last 1584561070 [ 9295.511840] Lustre: fir-MDT0001: Connection restored to 60becd34-63f8-4 (at 10.49.30.12@o2ib1) [ 9295.520455] Lustre: Skipped 1257 previous similar messages [ 9307.753529] Lustre: fir-MDT0001: Client 2226722d-03ed-4 (at 10.50.9.71@o2ib2) reconnecting [ 9307.761802] Lustre: Skipped 1188 previous similar messages [ 9332.603167] LustreError: 11-0: fir-OST000e-osc-MDT0001: operation ost_statfs to node 10.0.10.103@o2ib7 failed: rc = -19 [ 9332.613953] LustreError: Skipped 5 previous similar messages [ 9332.619619] Lustre: fir-OST000e-osc-MDT0001: Connection to fir-OST000e (at 10.0.10.103@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9332.635706] Lustre: Skipped 5 previous similar messages [ 9375.836419] LustreError: 11-0: fir-OST002a-osc-MDT0001: operation ost_statfs to node 10.0.10.107@o2ib7 failed: rc = -107 [ 9375.847291] LustreError: Skipped 5 previous similar messages [ 9375.852392] Lustre: fir-OST0026-osc-MDT0001: Connection to fir-OST0026 (at 10.0.10.107@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9375.852394] Lustre: Skipped 5 previous similar messages [ 9382.612512] Lustre: 21891:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561379/real 1584561379] req@ffff8aed12e7a400 x1661524333456960/t0(0) o13->fir-OST002e-osc-MDT0001@10.0.10.107@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584561386 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9409.368284] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [ 9409.380189] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [ 9413.045369] Lustre: 21908:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584561409/real 1584561409] req@ffff8afbfeb55580 x1661524333488512/t0(0) o13->fir-OST003c-osc-MDT0001@10.0.10.111@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584561416 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9441.902288] LustreError: 11-0: fir-OST0058-osc-MDT0001: operation ost_statfs to node 10.0.10.115@o2ib7 failed: rc = -107 [ 9441.913160] LustreError: Skipped 4 previous similar messages [ 9441.918834] Lustre: fir-OST0058-osc-MDT0001: Connection to fir-OST0058 (at 10.0.10.115@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [ 9441.934914] Lustre: Skipped 11 previous similar messages [ 9463.238607] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.14.6@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.255892] LustreError: Skipped 532 previous similar messages [ 9471.460016] LustreError: 42123:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8add3eeb0900 x1660724395218048/t0(0) o37->8c27500b-ffc7-4@10.50.0.64@o2ib2:706/0 lens 448/440 e 2 to 0 dl 1584561486 ref 1 fl Interpret:/0/0 rc 0/0 [ 9472.370038] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.103@o2ib7: 0 seconds [ 9472.380209] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 169 previous similar messages [ 9520.240387] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 9520.254218] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561223, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add5dbc1200/0xbe1cb0ed12f43543 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060a26001a expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [ 9520.291541] LustreError: 43551:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afc6c6b0d80) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9542.732844] Lustre: fir-MDT0001: haven't heard from client 430e4894-d38d-4 (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5c659c00, cur 1584561546 expire 1584561396 last 1584561319 [ 9542.752836] Lustre: Skipped 7 previous similar messages [ 9552.027442] Lustre: fir-OST0048-osc-MDT0001: Connection restored to 10.0.10.114@o2ib7 (at 10.0.10.114@o2ib7) [ 9552.037270] Lustre: Skipped 522 previous similar messages [ 9564.351883] Lustre: fir-MDT0001: Client a52f1005-c8d6-4 (at 10.50.1.15@o2ib2) reconnecting [ 9564.360149] Lustre: Skipped 740 previous similar messages [ 9618.634638] Lustre: fir-MDT0001: haven't heard from client 262f777c-aaac-4 (at 10.50.5.14@o2ib2) in 204 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b334400, cur 1584561622 expire 1584561472 last 1584561418 [ 9618.654548] Lustre: Skipped 2 previous similar messages [ 9694.619278] Lustre: fir-MDT0001: haven't heard from client 12d78f05-c42c-4 (at 10.50.7.13@o2ib2) in 184 seconds. I think it's dead, and I am evicting it. exp ffff8b0d723a4400, cur 1584561698 expire 1584561548 last 1584561514 [ 9694.639178] Lustre: Skipped 21 previous similar messages [ 9770.612334] Lustre: fir-MDT0001: haven't heard from client a52f1005-c8d6-4 (at 10.50.1.15@o2ib2) in 167 seconds. I think it's dead, and I am evicting it. exp ffff8b0d54a80800, cur 1584561774 expire 1584561624 last 1584561607 [ 9770.632249] Lustre: Skipped 4 previous similar messages [ 9827.352056] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [ 9827.365900] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561530, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add5dbc2880/0xbe1cb0ed12f58e26 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060a4abeed expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [ 9827.403255] LustreError: 43876:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afc6c6b0f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [10132.692682] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [10132.706514] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584561836, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add5dbc5340/0xbe1cb0ed12f7a03c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060b88d7a6 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [10132.743851] LustreError: 43968:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afc423d92c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [10132.763400] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [10132.772710] Lustre: Skipped 1589 previous similar messages [10440.314424] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [10440.328256] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562143, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add6139f2c0/0xbe1cb0ed13034eee lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bb22787 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [10440.365601] LustreError: 44070:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afd2f670d80) refcount nonzero (1) after lock cleanup; forcing cleanup. [10746.316183] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [10746.330015] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562449, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add4bc81d40/0xbe1cb0ed13192f57 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bd6d2e2 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [10746.367371] LustreError: 44163:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afd767c3500) refcount nonzero (1) after lock cleanup; forcing cleanup. [10746.386913] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [10746.396214] Lustre: Skipped 1 previous similar message [11055.187981] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11055.201804] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584562758, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add4bc81d40/0xbe1cb0ed132f6fd6 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060bfc8af2 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [11055.239149] LustreError: 44258:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add0e65a900) refcount nonzero (1) after lock cleanup; forcing cleanup. [11348.773767] Lustre: fir-OST002a-osc-MDT0001: Connection restored to 10.0.10.108@o2ib7 (at 10.0.10.108@o2ib7) [11348.783594] Lustre: Skipped 5 previous similar messages [11360.580661] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11360.594493] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563063, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add0ebdb180/0xbe1cb0ed13335bca lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060c23ce97 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [11360.631851] LustreError: 44398:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add3925bd40) refcount nonzero (1) after lock cleanup; forcing cleanup. [11667.073333] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11667.087162] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563370, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed0da74380/0xbe1cb0ed1336bae5 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060d4d0d9f expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [11667.124536] LustreError: 44561:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add3925b680) refcount nonzero (1) after lock cleanup; forcing cleanup. [11974.496935] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [11974.510773] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563677, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afd264aa1c0/0xbe1cb0ed136e660c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060e759af2 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [11974.548121] LustreError: 44651:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add0da2d800) refcount nonzero (1) after lock cleanup; forcing cleanup. [11974.567666] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [11974.576969] Lustre: Skipped 4 previous similar messages [12283.329541] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [12283.343376] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584563986, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add41440b40/0xbe1cb0ed13e30616 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060eebae57 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [12283.380758] LustreError: 44756:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add23b22180) refcount nonzero (1) after lock cleanup; forcing cleanup. [12309.983385] LustreError: 11-0: fir-OST001e-osc-MDT0001: operation ost_statfs to node 10.0.10.106@o2ib7 failed: rc = -107 [12309.994258] LustreError: Skipped 5 previous similar messages [12309.999932] Lustre: fir-OST001e-osc-MDT0001: Connection to fir-OST001e (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [12310.016026] Lustre: Skipped 6 previous similar messages [12324.303804] LustreError: 11-0: fir-OST0040-osc-MDT0001: operation ost_statfs to node 10.0.10.112@o2ib7 failed: rc = -107 [12324.314701] LustreError: Skipped 24 previous similar messages [12326.207840] Lustre: fir-OST0046-osc-MDT0001: Connection to fir-OST0046 (at 10.0.10.112@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [12326.223907] Lustre: Skipped 26 previous similar messages [12511.377859] LustreError: 11-0: fir-OST001b-osc-MDT0001: operation ost_destroy to node 10.0.10.106@o2ib7 failed: rc = -19 [12511.378102] Lustre: fir-OST0023-osc-MDT0001: Connection to fir-OST0023 (at 10.0.10.106@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [12511.378104] Lustre: Skipped 19 previous similar messages [12511.410109] LustreError: Skipped 26 previous similar messages [12522.885272] Lustre: 21889:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564519/real 1584564519] req@ffff8aed12e60d80 x1661524342123968/t0(0) o13->fir-OST0015-osc-MDT0001@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584564526 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [12522.913382] Lustre: 21889:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [12523.421288] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564519/real 1584564519] req@ffff8add4d1a3a80 x1661524342129600/t0(0) o13->fir-OST0017-osc-MDT0001@10.0.10.104@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584564526 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [12523.449409] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12524.429316] Lustre: 21891:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564519/real 1584564519] req@ffff8aecfbe64c80 x1661524342124800/t0(0) o6->fir-OST000f-osc-MDT0001@10.0.10.104@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1584564527 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [12524.457428] Lustre: 21891:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [12527.741414] Lustre: 21890:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564524/real 1584564524] req@ffff8aecfb618d80 x1661524342134080/t0(0) o13->fir-OST002f-osc-MDT0001@10.0.10.108@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1584564531 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [12527.769528] Lustre: 21890:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [12528.798427] LNetError: 326:0:(o2iblnd_cb.c:2962:kiblnd_rejected()) 10.0.10.108@o2ib7 rejected: o2iblnd fatal error [12528.808789] LNetError: 326:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [12528.820528] LNetError: 326:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [12530.456485] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.108@o2ib7: 0 seconds [12530.466659] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 198 previous similar messages [12555.747192] Lustre: 22470:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564552/real 1584564552] req@ffff8add52eae780 x1661524342154496/t0(0) o104->fir-MDT0001@10.50.0.61@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584564559 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [12555.774450] Lustre: 22470:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [12557.804757] Lustre: fir-MDT0001: Client 629e5bf9-3bd8-4 (at 10.49.21.4@o2ib1) reconnecting [12557.813027] Lustre: Skipped 1260 previous similar messages [12569.784583] Lustre: 22470:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584564566/real 1584564566] req@ffff8add52eae780 x1661524342154496/t0(0) o104->fir-MDT0001@10.50.0.61@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584564573 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [12569.811827] Lustre: 22470:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12579.457862] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [12579.469773] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [12581.832136] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.20.12@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.849502] LustreError: Skipped 2503 previous similar messages [12592.820296] Lustre: fir-MDT0001: Connection restored to 8ddae0cd-994d-4 (at 10.50.2.63@o2ib2) [12592.828824] Lustre: Skipped 150 previous similar messages [12593.282239] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [12593.296067] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584564296, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add3979dc40/0xbe1cb0ed1458c37c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279060f5682af expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [12593.333390] LustreError: 44850:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8adcffaf9800) refcount nonzero (1) after lock cleanup; forcing cleanup. [12594.458296] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.106@o2ib7: 0 seconds [12594.468468] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 144 previous similar messages [12615.420808] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.6.29@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.438088] LustreError: Skipped 19 previous similar messages [12623.820741] Lustre: fir-MDT0001: Client a90b3cdd-0751-4 (at 10.50.8.61@o2ib2) reconnecting [12623.829013] Lustre: Skipped 16 previous similar messages [12679.834970] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.30.19@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [12679.852351] LustreError: Skipped 161 previous similar messages [12736.723617] Lustre: fir-MDT0001: haven't heard from client b428c775-980e-4 (at 10.49.26.31@o2ib1) in 211 seconds. I think it's dead, and I am evicting it. exp ffff8b0d5b64b800, cur 1584564740 expire 1584564590 last 1584564529 [12751.898232] Lustre: fir-MDT0001: Client 99a50c64-49e6-4 (at 10.49.22.26@o2ib1) reconnecting [12751.906588] Lustre: Skipped 1291 previous similar messages [13207.479479] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [13207.493298] LustreError: Skipped 1 previous similar message [13207.498893] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584564910, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed072218c0/0xbe1cb0ed149ace7b lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906100a193c expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [13207.536045] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [13207.547028] LustreError: 45080:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add23f7d800) refcount nonzero (1) after lock cleanup; forcing cleanup. [13207.566568] LustreError: 45080:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [13207.577008] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [13207.586318] Lustre: Skipped 1861 previous similar messages [13823.953806] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [13823.967622] LustreError: Skipped 1 previous similar message [13823.973215] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584565527, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8add2c247500/0xbe1cb0ed158fd22c lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790610e2def7 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [13824.010373] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [13824.021409] LustreError: 45288:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add4c7d1980) refcount nonzero (1) after lock cleanup; forcing cleanup. [13824.040956] LustreError: 45288:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [13824.051401] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [13824.060710] Lustre: Skipped 1 previous similar message [14441.290042] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [14441.303860] LustreError: Skipped 1 previous similar message [14441.309453] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584566144, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed052cb840/0xbe1cb0ed1699b5ae lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790611b997b2 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [14441.346613] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [14441.357670] LustreError: 45474:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aed691ef080) refcount nonzero (1) after lock cleanup; forcing cleanup. [14441.377221] LustreError: 45474:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [14441.387659] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [14441.396969] Lustre: Skipped 1 previous similar message [14717.282796] LustreError: 11-0: fir-OST0007-osc-MDT0001: operation ost_statfs to node 10.0.10.101@o2ib7 failed: rc = -107 [14717.293671] LustreError: Skipped 32 previous similar messages [14717.298811] Lustre: fir-OST0049-osc-MDT0001: Connection to fir-OST0049 (at 10.0.10.113@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [14717.298813] Lustre: Skipped 46 previous similar messages [14721.714887] LustreError: 11-0: fir-OST0031-osc-MDT0001: operation ost_statfs to node 10.0.10.109@o2ib7 failed: rc = -107 [14721.725758] LustreError: Skipped 39 previous similar messages [15056.497121] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [15056.510937] LustreError: Skipped 1 previous similar message [15056.516528] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584566759, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed0726c5c0/0xbe1cb0ed176e1bee lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906127f6fd7 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [15056.553655] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [15056.564671] LustreError: 45905:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aecfe3eb800) refcount nonzero (1) after lock cleanup; forcing cleanup. [15056.584207] LustreError: 45905:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [15056.594649] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [15056.603959] Lustre: Skipped 97 previous similar messages [15298.831653] Lustre: fir-MDD0001: changelog on [15673.305288] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [15673.319122] LustreError: Skipped 1 previous similar message [15673.324716] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584567376, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed06fd2f40/0xbe1cb0ed1831e243 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061300a55b expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [15673.361845] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [15673.372831] LustreError: 46129:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aed67599d40) refcount nonzero (1) after lock cleanup; forcing cleanup. [15673.392372] LustreError: 46129:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [15673.402816] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [15673.412124] Lustre: Skipped 1 previous similar message [16286.613385] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [16286.627204] LustreError: Skipped 1 previous similar message [16286.632802] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584567989, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed50ad8fc0/0xbe1cb0ed1a23779d lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790613bc5355 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [16286.669934] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [16286.680976] LustreError: 46349:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afcd3f4ca80) refcount nonzero (1) after lock cleanup; forcing cleanup. [16286.700520] LustreError: 46349:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [16286.710961] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [16286.720268] Lustre: Skipped 1 previous similar message [16905.643773] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [16905.657587] LustreError: Skipped 1 previous similar message [16905.663180] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584568608, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aece7f50000/0xbe1cb0ed1be9606f lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061478ce32 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [16905.700324] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [16905.711341] LustreError: 46551:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aeceaa00900) refcount nonzero (1) after lock cleanup; forcing cleanup. [16905.730888] LustreError: 46551:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [16905.741336] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [16905.750644] Lustre: Skipped 1 previous similar message [17520.083946] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [17520.097762] LustreError: Skipped 1 previous similar message [17520.103355] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584569223, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aecf5b098c0/0xbe1cb0ed1d8a1f84 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906153cc786 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [17520.140544] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [17520.151560] LustreError: 46749:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aecdfb9e900) refcount nonzero (1) after lock cleanup; forcing cleanup. [17520.171110] LustreError: 46749:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [17520.181565] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [17520.190878] Lustre: Skipped 1 previous similar message [18135.816017] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [18135.829833] LustreError: Skipped 1 previous similar message [18135.835429] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584569838, 301s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aecd524f980/0xbe1cb0ed251099e6 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906160b1b9c expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [18135.872598] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18135.883668] LustreError: 46939:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8ad583ef7200) refcount nonzero (1) after lock cleanup; forcing cleanup. [18135.903257] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [18135.912574] Lustre: Skipped 1 previous similar message [18332.721397] list passed to list_sort() too long for efficiency [18336.642840] sched: RT throttling activated [18748.187609] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [18748.201429] LustreError: Skipped 1 previous similar message [18748.207022] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584570451, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aed12e35100/0xbe1cb0ed30e44b87 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790616e7537a expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [18748.244157] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18748.255259] LustreError: 47124:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8aeced790fc0) refcount nonzero (1) after lock cleanup; forcing cleanup. [18748.274813] LustreError: 47124:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [18748.285269] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [18748.294595] Lustre: Skipped 1 previous similar message [18880.562892] perf: interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [19367.341435] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [19367.355250] LustreError: Skipped 1 previous similar message [19367.360847] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584571070, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8aecf7e3ee40/0xbe1cb0ed3c90fa90 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d27906179b369e expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [19367.397980] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [19367.409096] LustreError: 47310:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b0d6f6acf00) refcount nonzero (1) after lock cleanup; forcing cleanup. [19367.428673] LustreError: 47310:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [19367.439112] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [19367.448454] Lustre: Skipped 1 previous similar message [19486.863720] Lustre: fir-MDT0001: haven't heard from client 3c0f2777-beb3-4 (at 10.50.1.60@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d70109800, cur 1584571490 expire 1584571340 last 1584571263 [19486.883607] Lustre: Skipped 58 previous similar messages [19981.348110] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [19981.361926] LustreError: Skipped 1 previous similar message [19981.367522] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584571684, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afcc3f34380/0xbe1cb0ed4459f371 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061855b0d0 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [19981.404648] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [19981.415709] LustreError: 47496:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8ad8b62d7080) refcount nonzero (1) after lock cleanup; forcing cleanup. [19981.435260] LustreError: 47496:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [19981.445711] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [19981.455033] Lustre: Skipped 1 previous similar message [20594.303785] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [20594.317600] LustreError: Skipped 1 previous similar message [20594.323200] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584572297, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afd51a57bc0/0xbe1cb0ed4d03ebf5 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d2790618ed7802 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [20594.360340] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [20594.371403] LustreError: 47709:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afcef34ef00) refcount nonzero (1) after lock cleanup; forcing cleanup. [20594.390944] LustreError: 47709:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [20594.401384] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [20594.410692] Lustre: Skipped 1 previous similar message [20689.682381] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [20689.692559] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 7, oc: 0, rc: 8 [20808.900719] Lustre: fir-MDT0001: haven't heard from client f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73058800, cur 1584572812 expire 1584572662 last 1584572585 [21209.909518] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [21209.923345] LustreError: Skipped 1 previous similar message [21209.928947] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584572912, 301s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afc3822b600/0xbe1cb0ed52f82511 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061989a0d8 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [21209.966160] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [21209.977245] LustreError: 47899:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8b0d7575d680) refcount nonzero (1) after lock cleanup; forcing cleanup. [21209.996796] LustreError: 47899:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [21210.007247] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [21210.016573] Lustre: Skipped 2 previous similar messages [21824.285227] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [21824.299043] LustreError: Skipped 1 previous similar message [21824.304639] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584573527, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afd00e64800/0xbe1cb0ed56392271 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061a38be75 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [21824.341763] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [21824.352800] LustreError: 48091:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afd3726b200) refcount nonzero (1) after lock cleanup; forcing cleanup. [21824.372346] LustreError: 48091:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [21824.382788] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [21824.392099] Lustre: Skipped 1 previous similar message [22438.033047] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [22438.046860] LustreError: Skipped 1 previous similar message [22438.052455] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574141, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8afd1fa406c0/0xbe1cb0ed57eb86d2 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061af85eec expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [22438.089591] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [22438.100609] LustreError: 48276:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8afd2c664a80) refcount nonzero (1) after lock cleanup; forcing cleanup. [22438.120151] LustreError: 48276:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [22438.130592] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [22438.139901] Lustre: Skipped 1 previous similar message [22583.394537] LustreError: 22450:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8acd8969de80 x1661524459155264 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8add01f9e0c0/0xbe1cb0ed2390a211 lrc: 4/0,0 mode: PR/PR res: [0x24003ddc1:0x15aad:0x0].0x0 bits 0x12/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d45cbf87 expref: 2530766 pid: 22449 timeout: 22731 lvb_type: 0 [22583.406654] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22583.406683] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad600c5c140/0xbe1cb0ed2423e6dd lrc: 3/0,0 mode: PR/PR res: [0x24003ddc1:0x15aaf:0x0].0x0 bits 0x12/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d48e8658 expref: 2530749 pid: 22333 timeout: 0 lvb_type: 0 [22583.487819] LustreError: 22450:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 2 previous similar messages [22587.231933] LustreError: 22459:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8acd8969cc80 x1661524459284096 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad3f4e55340/0xbe1cb0ed28076288 lrc: 4/0,0 mode: PR/PR res: [0x24003ddc1:0x15ab3:0x0].0x0 bits 0x1b/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d6084f85 expref: 2525087 pid: 22413 timeout: 22734 lvb_type: 0 [22587.275227] LustreError: 22459:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message [22587.285230] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22587.297664] LustreError: Skipped 3 previous similar messages [22587.303356] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad3f4e55340/0xbe1cb0ed28076288 lrc: 3/0,0 mode: PR/PR res: [0x24003ddc1:0x15ab3:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d6084f85 expref: 2524982 pid: 22413 timeout: 0 lvb_type: 0 [22587.340830] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [22605.081086] LustreError: 22377:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8acea25cf080 x1661524459691072 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8aceee351f80/0xbe1cb0ed32e79fe1 lrc: 4/0,0 mode: PR/PR res: [0x240044fb5:0x103ff:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d9b59731 expref: 2498772 pid: 22473 timeout: 22752 lvb_type: 0 [22605.124470] LustreError: 22377:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message [22605.134477] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22605.146927] LustreError: Skipped 1 previous similar message [22605.152527] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8aceee351f80/0xbe1cb0ed32e79fe1 lrc: 3/0,0 mode: PR/PR res: [0x240044fb5:0x103ff:0x0].0x0 bits 0x12/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d9b59731 expref: 2498667 pid: 22473 timeout: 0 lvb_type: 0 [22605.190095] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [22616.495674] LustreError: 22449:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8acd89698d80 x1661524460021312 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad14d886540/0xbe1cb0ed2dc46332 lrc: 4/0,0 mode: PR/PR res: [0x240047b8a:0x120aa:0x0].0x0 bits 0x20/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d7eee57a expref: 2482002 pid: 22485 timeout: 22764 lvb_type: 0 [22616.539059] LustreError: 22449:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message [22616.549070] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22616.561500] LustreError: Skipped 1 previous similar message [22616.567105] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad14d886540/0xbe1cb0ed2dc46332 lrc: 3/0,0 mode: PR/PR res: [0x240047b8a:0x120aa:0x0].0x0 bits 0x20/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d7eee57a expref: 2481898 pid: 22485 timeout: 0 lvb_type: 0 [22616.604613] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [22621.499592] LustreError: 22442:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8add10382880 x1661524460190080 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad07d911b00/0xbe1cb0ed2fa63d9b lrc: 4/0,0 mode: PR/PR res: [0x2400481bc:0x94d3:0x0].0x0 bits 0x20/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d88c75b8 expref: 2474683 pid: 22443 timeout: 22769 lvb_type: 0 [22621.542878] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22621.555337] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad07d911b00/0xbe1cb0ed2fa63d9b lrc: 3/0,0 mode: PR/PR res: [0x2400481bc:0x94d3:0x0].0x0 bits 0x20/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d88c75b8 expref: 2474603 pid: 22443 timeout: 0 lvb_type: 0 [22643.889454] LustreError: 22418:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8acd8969e780 x1661524460754368 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8acfdc30a400/0xbe1cb0ed310d11ec lrc: 4/0,0 mode: PR/PR res: [0x240048a2f:0x358e:0x0].0x0 bits 0x12/0x0 rrc: 13 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d90ae698 expref: 2441991 pid: 22443 timeout: 22791 lvb_type: 0 [22643.932744] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22643.945203] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8acfdc30a400/0xbe1cb0ed310d11ec lrc: 3/0,0 mode: PR/PR res: [0x240048a2f:0x358e:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d90ae698 expref: 2441910 pid: 22443 timeout: 0 lvb_type: 0 [22664.766168] LustreError: 22449:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8add10385580 x1661524461169280 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad51aeb3f00/0xbe1cb0ed25c2c617 lrc: 4/0,0 mode: PR/PR res: [0x240048c52:0xd725:0x0].0x0 bits 0x20/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d52bbe91 expref: 2411651 pid: 22478 timeout: 22812 lvb_type: 0 [22664.809449] LustreError: 22449:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 2 previous similar messages [22664.819543] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22664.831974] LustreError: Skipped 2 previous similar messages [22664.837662] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad51aeb3f00/0xbe1cb0ed25c2c617 lrc: 3/0,0 mode: PR/PR res: [0x240048c52:0xd725:0x0].0x0 bits 0x20/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d52bbe91 expref: 2411549 pid: 22478 timeout: 0 lvb_type: 0 [22664.875047] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages [22783.696526] LNet: Service thread pid 22386 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [22783.713468] Pid: 22386, comm: mdt00_018 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [22783.723645] Call Trace: [22783.726112] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [22783.733068] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [22783.740289] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [22783.747134] [] mdt_object_lock_internal+0x70/0x360 [mdt] [22783.754140] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [22783.760802] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [22783.767372] [] mdt_reint_rec+0x83/0x210 [mdt] [22783.773434] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [22783.780010] [] mdt_reint+0x67/0x140 [mdt] [22783.785735] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [22783.792690] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [22783.800435] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [22783.806771] [] kthread+0xd1/0xe0 [22783.811688] [] ret_from_fork_nospec_begin+0xe/0x21 [22783.818165] [] 0xffffffffffffffff [22783.823190] LustreError: dumping log to /tmp/lustre-log.1584574786.22386 [22783.927697] Pid: 22450, comm: mdt00_046 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [22783.937875] Call Trace: [22783.940340] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [22783.947297] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [22783.954529] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [22783.961374] [] mdt_object_lock_internal+0x70/0x360 [mdt] [22783.968382] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [22783.975039] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [22783.981613] [] mdt_reint_rec+0x83/0x210 [mdt] [22783.987670] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [22783.994265] [] mdt_reint+0x67/0x140 [mdt] [22783.999985] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [22784.006959] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [22784.014681] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [22784.021026] [] kthread+0xd1/0xe0 [22784.025943] [] ret_from_fork_nospec_begin+0xe/0x21 [22784.032434] [] 0xffffffffffffffff [22787.280631] LNet: Service thread pid 22459 was inactive for 200.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [22787.297571] LNet: Skipped 1 previous similar message [22787.302555] Pid: 22459, comm: mdt00_049 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [22787.312734] Call Trace: [22787.315203] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [22787.322161] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [22787.329392] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [22787.336233] [] mdt_object_lock_internal+0x70/0x360 [mdt] [22787.343234] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [22787.349911] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [22787.356485] [] mdt_reint_rec+0x83/0x210 [mdt] [22787.362540] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [22787.369118] [] mdt_reint+0x67/0x140 [mdt] [22787.374844] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [22787.381799] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [22787.389541] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [22787.395871] [] kthread+0xd1/0xe0 [22787.400802] [] ret_from_fork_nospec_begin+0xe/0x21 [22787.407279] [] 0xffffffffffffffff [22787.412329] LustreError: dumping log to /tmp/lustre-log.1584574790.22459 [22805.713133] LNet: Service thread pid 22377 was inactive for 200.62s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [22805.730070] Pid: 22377, comm: mdt00_015 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [22805.740246] Call Trace: [22805.742714] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [22805.749673] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [22805.756883] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [22805.763731] [] mdt_object_lock_internal+0x70/0x360 [mdt] [22805.770751] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [22805.777421] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [22805.783995] [] mdt_reint_rec+0x83/0x210 [mdt] [22805.790050] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [22805.796627] [] mdt_reint+0x67/0x140 [mdt] [22805.802333] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [22805.809300] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [22805.817031] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [22805.823365] [] kthread+0xd1/0xe0 [22805.828279] [] ret_from_fork_nospec_begin+0xe/0x21 [22805.834754] [] 0xffffffffffffffff [22805.839791] LustreError: dumping log to /tmp/lustre-log.1584574808.22377 [22811.492915] LustreError: 22442:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8ad874307500 x1661524461531264 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8afd2615f980/0xbe1cb0ed1d86fb41 lrc: 4/0,0 mode: PR/PR res: [0x240048d08:0xf32:0x0].0x0 bits 0x1b/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d228f654 expref: 2203214 pid: 22343 timeout: 22959 lvb_type: 0 [22811.536125] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [22811.548581] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8afd2615f980/0xbe1cb0ed1d86fb41 lrc: 3/0,0 mode: PR/PR res: [0x240048d08:0xf32:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d228f654 expref: 2203138 pid: 22343 timeout: 0 lvb_type: 0 [22844.626206] LNet: Service thread pid 22418 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [22844.643147] Pid: 22418, comm: mdt00_029 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [22844.653323] Call Trace: [22844.655792] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [22844.662747] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [22844.669942] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [22844.676780] [] mdt_object_lock_internal+0x70/0x360 [mdt] [22844.683796] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [22844.690453] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [22844.697026] [] mdt_reint_rec+0x83/0x210 [mdt] [22844.703082] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [22844.709655] [] mdt_reint+0x67/0x140 [mdt] [22844.715365] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [22844.722333] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [22844.730065] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [22844.736395] [] kthread+0xd1/0xe0 [22844.741315] [] ret_from_fork_nospec_begin+0xe/0x21 [22844.747795] [] 0xffffffffffffffff [22844.752828] LustreError: dumping log to /tmp/lustre-log.1584574847.22418 [22877.395080] LNet: Service thread pid 22449 was inactive for 212.62s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [22877.407945] LustreError: dumping log to /tmp/lustre-log.1584574880.22449 [22883.415245] LustreError: 22386:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574586, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ada11e18b40/0xbe1cb0ed5888cd6d lrc: 3/0,1 mode: --/EX res: [0x24003ddc1:0x15aaf:0x0].0x0 bits 0x22/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22386 timeout: 0 lvb_type: 0 [22883.454849] LustreError: dumping log to /tmp/lustre-log.1584574886.22386 [22887.311361] LustreError: 22459:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574590, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8acd7d5706c0/0xbe1cb0ed588b77fb lrc: 3/0,1 mode: --/EX res: [0x24003ddc1:0x15ab3:0x0].0x0 bits 0x22/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22459 timeout: 0 lvb_type: 0 [22887.350926] LustreError: 22459:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [22905.159839] LustreError: 22377:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574608, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad62b4260c0/0xbe1cb0ed5896d5bd lrc: 3/0,1 mode: --/EX res: [0x240044fb5:0x103ff:0x0].0x0 bits 0x22/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22377 timeout: 0 lvb_type: 0 [22943.952931] LustreError: 22418:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574646, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad1306fb180/0xbe1cb0ed58b0e3ba lrc: 3/0,1 mode: --/EX res: [0x240048a2f:0x358e:0x0].0x0 bits 0x22/0x0 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22418 timeout: 0 lvb_type: 0 [22964.845531] LustreError: 22449:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574667, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ace91c9e9c0/0xbe1cb0ed58be8df3 lrc: 3/0,1 mode: --/EX res: [0x240048c52:0xd725:0x0].0x0 bits 0x22/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22449 timeout: 0 lvb_type: 0 [23050.781925] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [23050.795739] LustreError: Skipped 1 previous similar message [23050.801352] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574753, 300s ago), entering recovery for MGS@10.0.10.51@o2ib7 ns: MGC10.0.10.51@o2ib7 lock: ffff8b0cce6ba400/0xbe1cb0ed58dd84d9 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061bcc1791 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [23050.838515] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [23050.849553] LustreError: 48493:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8add2df6b680) refcount nonzero (1) after lock cleanup; forcing cleanup. [23050.869088] LustreError: 48493:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [23050.879532] Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [23050.888839] Lustre: Skipped 1 previous similar message [23111.556589] LustreError: 22442:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584574814, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad5f5463a80/0xbe1cb0ed58f41610 lrc: 3/0,1 mode: --/EX res: [0x240048d08:0xf32:0x0].0x0 bits 0x22/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22442 timeout: 0 lvb_type: 0 [23178.619426] Lustre: 22495:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8add3bdd1200 x1661545158214208/t0(0) o36->3dab8abe-e790-3878-3898-4444ee422524@10.0.10.3@o2ib7:61/0 lens 528/440 e 24 to 0 dl 1584575186 ref 2 fl Interpret:/0/0 rc 0/0 [23182.665535] Lustre: 22495:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8add5ecea880 x1661545159079808/t0(0) o36->3dab8abe-e790-3878-3898-4444ee422524@10.0.10.3@o2ib7:65/0 lens 528/440 e 23 to 0 dl 1584575190 ref 2 fl Interpret:/0/0 rc 0/0 [23182.694088] Lustre: 22495:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [23184.422679] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [23184.432678] Lustre: Skipped 448 previous similar messages [23227.967034] Lustre: fir-MDT0001: haven't heard from client 15c6dd40-b461-4 (at 10.50.1.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d7305d800, cur 1584575231 expire 1584575081 last 1584575004 [23254.413674] LustreError: 22442:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8ad753276300 x1661524462488896 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad501e86780/0xbe1cb0ed25f26faa lrc: 4/0,0 mode: PR/PR res: [0x240048d24:0xf7ec:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d53dbfa1 expref: 1616481 pid: 22439 timeout: 23402 lvb_type: 0 [23254.456962] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [23254.469416] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad501e86780/0xbe1cb0ed25f26faa lrc: 3/0,0 mode: PR/PR res: [0x240048d24:0xf7ec:0x0].0x0 bits 0x12/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d53dbfa1 expref: 1616412 pid: 22439 timeout: 0 lvb_type: 0 [23307.120459] LNet: Service thread pid 22386 completed after 723.69s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [23439.613031] LNet: Service thread pid 22449 completed after 774.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [23439.615892] LustreError: 22483:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8add5bf3b600 x1661524462944256 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad3dc31e300/0xbe1cb0ed28471769 lrc: 4/0,0 mode: PR/PR res: [0x240048d24:0xf7f5:0x0].0x0 bits 0x12/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d620b6e9 expref: 1386814 pid: 22478 timeout: 23587 lvb_type: 0 [23439.615895] LustreError: 22483:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 3 previous similar messages [23439.615898] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [23439.615899] LustreError: Skipped 3 previous similar messages [23439.615922] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8ad3dc31e300/0xbe1cb0ed28471769 lrc: 3/0,0 mode: PR/PR res: [0x240048d24:0xf7f5:0x0].0x0 bits 0x12/0x0 rrc: 10 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d620b6e9 expref: 1386815 pid: 22478 timeout: 0 lvb_type: 0 [23439.615925] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages [23554.476719] LustreError: 22442:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584575257, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad743b02880/0xbe1cb0ed599795be lrc: 3/0,1 mode: --/EX res: [0x240048d24:0xf7ec:0x0].0x0 bits 0x22/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22442 timeout: 0 lvb_type: 0 [23561.252844] Lustre: fir-MDT0001: Client 541f81d4-bd4f-4 (at 10.50.7.3@o2ib2) reconnecting [23566.958066] Lustre: 21907:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584575561/real 1584575561] req@ffff8afba2af0d80 x1661524463403968/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1584575568 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23571.735822] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.14.10@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [23571.753192] LustreError: Skipped 3506 previous similar messages [23587.774955] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.3.17@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [23587.792234] LustreError: Skipped 155 previous similar messages [23607.185178] LustreError: 22495:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584575310, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad45aa70b40/0xbe1cb0ed59a6806f lrc: 3/0,1 mode: --/EX res: [0x240048d24:0xf7f1:0x0].0x0 bits 0x22/0x0 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22495 timeout: 0 lvb_type: 0 [23622.644105] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.27@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [23622.661382] LustreError: Skipped 143 previous similar messages [23652.763429] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [23652.773599] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (7): c: 6, oc: 0, rc: 8 [23652.785704] LNetError: 21865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.51@o2ib7 added to recovery queue. Health = 900 [23653.020443] Lustre: 42005:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1584575649/real 0] req@ffff8aed12e61b00 x1661524463579584/t0(0) o101->fir-MDT0000-osp-MDT0001@10.0.10.51@o2ib7:24/4 lens 328/344 e 0 to 1 dl 1584575656 ref 3 fl Rpc:X/0/ffffffff rc 0/-1 [23653.047871] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [23653.063848] Lustre: Skipped 47 previous similar messages [23653.069403] LNetError: 47326:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [23664.651760] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584575367, 300s ago), entering recovery for MGS@MGC10.0.10.51@o2ib7_1 ns: MGC10.0.10.51@o2ib7 lock: ffff8ad2f2ddd100/0xbe1cb0ed59b55e54 lrc: 4/1,0 mode: --/CR res: [0x726966:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x22d279061c7e24c0 expref: -99 pid: 21977 timeout: 0 lvb_type: 0 [23664.689322] LustreError: 21977:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [23666.722438] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [23678.121503] LNetError: 47326:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [23678.133412] LNetError: 47326:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [23679.346339] Lustre: 21895:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1584575681/real 1584575682] req@ffff8aed1aede780 x1661524463618816/t0(0) o400->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1584575834 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [23679.375056] Lustre: 21895:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [23679.384717] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [23686.692109] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.8.29@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [23686.709395] LustreError: Skipped 500 previous similar messages [23708.736211] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [23709.347201] LNetError: 47326:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [23709.359109] LNetError: 47326:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [23711.678685] LDISKFS-fs (dm-0): recovery complete [23711.683557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [23735.076695] Lustre: 22409:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-492), not sending early reply req@ffff8ad20413f080 x1661545174145344/t0(0) o36->3dab8abe-e790-3878-3898-4444ee422524@10.0.10.3@o2ib7:618/0 lens 528/440 e 6 to 0 dl 1584575743 ref 2 fl Interpret:/0/0 rc 0/0 [23739.623851] LustreError: 22483:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1584575442, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ad1d75e0240/0xbe1cb0ed59d56e3d lrc: 3/0,1 mode: --/EX res: [0x240048d24:0xf7f5:0x0].0x0 bits 0x22/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22483 timeout: 0 lvb_type: 0 [23741.498881] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [23741.508905] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [23741.519162] Lustre: Skipped 4 previous similar messages [23764.766509] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [23764.776596] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [23764.785907] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [23764.797811] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [23782.766990] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds [23782.777074] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 3 previous similar messages [23814.792883] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.52@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [23814.810165] LustreError: Skipped 1437 previous similar messages [23827.768254] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [23827.778337] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 5 previous similar messages [23840.768597] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [23840.780514] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 11 previous similar messages [23846.125723] LNet: Service thread pid 42005 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [23846.142653] Pid: 42005, comm: mdt01_040 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [23846.152827] Call Trace: [23846.155292] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [23846.161892] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [23846.168564] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [23846.175237] [] osp_md_object_lock+0x162/0x2d0 [osp] [23846.181810] [] lod_object_lock+0xf4/0x780 [lod] [23846.188032] [] mdd_object_lock+0x3e/0xe0 [mdd] [23846.194169] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [23846.201429] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [23846.208171] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [23846.214395] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [23846.220879] [] mdt_reint_rec+0x83/0x210 [mdt] [23846.226929] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [23846.233497] [] mdt_reint+0x67/0x140 [mdt] [23846.239199] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [23846.246143] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [23846.253857] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [23846.260183] [] kthread+0xd1/0xe0 [23846.265098] [] ret_from_fork_nospec_begin+0xe/0x21 [23846.271573] [] 0xffffffffffffffff [23846.276591] LustreError: dumping log to /tmp/lustre-log.1584575849.42005 [23912.355972] Lustre: fir-MDT0000: Not available for connect from 10.50.16.15@o2ib2 (not set up) [23912.364586] Lustre: Skipped 6 previous similar messages [23913.383551] Lustre: fir-MDT0000: Not available for connect from 10.50.5.52@o2ib2 (not set up) [23913.392074] Lustre: Skipped 8 previous similar messages [23913.592842] Lustre: fir-MDT0001: Received LWP connection from 0@lo, removing former export from 10.0.10.51@o2ib7 [24075.994077] Lustre: fir-MDT0001: haven't heard from client 48a86d34-282f-4 (at 10.50.5.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d654e4800, cur 1584576079 expire 1584575929 last 1584575852 [24104.153531] LNet: Service thread pid 22377 completed after 1499.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24169.462918] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x22d27906083e0c79 to 0xae6d2c631bd3f37c [24169.474921] LustreError: 49181:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.10.51@o2ib7: namespace resource [0x726966:0x2:0x0].0x0 (ffff8ad8985552c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [24169.494512] LustreError: 49181:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message [24169.523326] Lustre: fir-MDT0000: Imperative Recovery not enabled, recovery window 300-900 [24169.541877] Lustre: fir-MDD0000: changelog on [24169.564638] Lustre: fir-MDT0000: in recovery but waiting for the first client to connect [24169.574466] Lustre: fir-MDT0000: Will be in recovery for at least 5:00, or until 1289 clients reconnect [24172.345393] LustreError: 42044:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afd720b4050 x1661538582546944/t0(0) o601->fir-MDT0000-lwp-OST0023_UUID@10.0.10.106@o2ib7:446/0 lens 336/0 e 0 to 0 dl 1584576326 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24172.371358] LustreError: 42044:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 984 previous similar messages [24174.404956] LustreError: 21997:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afb75f66300 x1661535676014016/t0(0) o601->fir-MDT0000-lwp-OST0006_UUID@10.0.10.101@o2ib7:448/0 lens 336/0 e 0 to 0 dl 1584576328 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24174.430905] LustreError: 21997:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 39 previous similar messages [24178.446152] LustreError: 42070:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8acf9b004c80 x1661538563375488/t0(0) o601->fir-MDT0000-lwp-OST0037_UUID@10.0.10.110@o2ib7:301/0 lens 336/0 e 0 to 0 dl 1584576936 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24178.472107] LustreError: 42070:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 65 previous similar messages [24186.542623] LustreError: 42117:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afb75346300 x1661538583160448/t0(0) o601->fir-MDT0000-lwp-OST001d_UUID@10.0.10.106@o2ib7:460/0 lens 336/0 e 0 to 0 dl 1584576340 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24186.568574] LustreError: 42117:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 232 previous similar messages [24194.552480] LustreError: 167-0: fir-MDT0000-lwp-MDT0001: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [24202.552858] LustreError: 22530:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8add53e17850 x1661535950923200/t0(0) o601->fir-MDT0000-lwp-OST0050_UUID@10.0.10.113@o2ib7:325/0 lens 336/0 e 0 to 0 dl 1584576960 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24202.578804] LustreError: 22530:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 705 previous similar messages [24209.047433] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1276 recovered, 11 in progress, and 0 evicted) to recover in 4:20 [24234.071963] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1276 recovered, 11 in progress, and 0 evicted) to recover in 3:55 [24234.636759] LustreError: 42115:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8afb74a3f500 x1661538583303040/t0(0) o601->fir-MDT0000-lwp-OST001d_UUID@10.0.10.106@o2ib7:508/0 lens 336/0 e 0 to 0 dl 1584576388 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24234.662710] LustreError: 42115:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 1082 previous similar messages [24241.272708] Lustre: 21983:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8aec73285e80 x1659165561069760/t0(0) o36->5ab066c6-3005-4@10.50.13.9@o2ib2:369/0 lens 560/2888 e 24 to 0 dl 1584576249 ref 2 fl Interpret:/0/0 rc 0/0 [24247.101579] Lustre: fir-MDT0001: Client 5ab066c6-3005-4 (at 10.50.13.9@o2ib2) reconnecting [24251.138833] LNet: Service thread pid 22459 completed after 1663.86s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24259.160530] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 3:30 [24283.901094] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [24283.911007] LustreError: Skipped 3 previous similar messages [24284.249590] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 3:05 [24298.724645] LustreError: 41943:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8ad5e48b9f80 x1661538583775680/t0(0) o601->fir-MDT0000-lwp-OST001d_UUID@10.0.10.106@o2ib7:572/0 lens 336/0 e 0 to 0 dl 1584576452 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24298.750597] LustreError: 41943:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 16342 previous similar messages [24309.337505] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 2:39 [24309.750372] LNet: Service thread pid 22450 completed after 1726.30s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24334.427305] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 2:14 [24343.902650] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [24351.632015] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [24359.515757] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 1:49 [24403.904320] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [24403.914240] LustreError: Skipped 2 previous similar messages [24407.860472] LNet: Service thread pid 22418 completed after 1763.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24409.692833] Lustre: fir-MDT0000: Denying connection for new client 18e71bec-a099-4 (at 10.50.5.38@o2ib2), waiting for 1289 known clients (1277 recovered, 11 in progress, and 0 evicted) to recover in 0:59 [24409.710895] Lustre: Skipped 1 previous similar message [24426.822147] LustreError: 22599:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8ad7e229e780 x1661535951445952/t0(0) o601->fir-MDT0000-lwp-OST004a_UUID@10.0.10.113@o2ib7:549/0 lens 336/0 e 0 to 0 dl 1584577184 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [24426.848097] LustreError: 22599:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 3930 previous similar messages [24463.905957] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11 [24469.336035] Lustre: fir-MDT0000: recovery is timed out, evict stale exports [24469.343305] Lustre: fir-MDT0000: disconnecting 1 stale clients [24469.351999] Lustre: 49200:0:(ldlm_lib.c:1782:extend_recovery_timer()) fir-MDT0000: extended recovery timer reached hard limit: 900, extend: 1 [24469.411974] Lustre: fir-MDT0000-osp-MDT0001: Connection restored to 10.0.10.52@o2ib7 (at 0@lo) [24469.412033] Lustre: fir-MDT0000: Recovery over after 5:00, of 1289 clients 1288 recovered and 1 was evicted. [24469.430413] Lustre: Skipped 1394 previous similar messages [24469.494775] LNet: Service thread pid 42005 completed after 823.45s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24470.793691] LustreError: 22442:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.0.10.3@o2ib7) returned error from blocking AST (req@ffff8add6ef72880 x1661524465312000 status -107 rc -107), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8add42ea4a40/0xbe1cb0ed20ea29bf lrc: 4/0,0 mode: PR/PR res: [0x24004e4a6:0x25e8:0x0].0x0 bits 0x1b/0x0 rrc: 11 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d374bfc8 expref: 234308 pid: 22478 timeout: 24618 lvb_type: 0 [24470.836897] LustreError: 22442:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message [24470.846909] LustreError: 138-a: fir-MDT0001: A client on nid 10.0.10.3@o2ib7 was evicted due to a lock blocking callback time out: rc -107 [24470.859341] LustreError: Skipped 1 previous similar message [24470.864945] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.0.10.3@o2ib7 ns: mdt-fir-MDT0001_UUID lock: ffff8add42ea4a40/0xbe1cb0ed20ea29bf lrc: 3/0,0 mode: PR/PR res: [0x24004e4a6:0x25e8:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.0.10.3@o2ib7 remote: 0x4de8f570d374bfc8 expref: 234235 pid: 22478 timeout: 0 lvb_type: 0 [24470.902246] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [24509.696159] LNet: Service thread pid 22495 was inactive for 1202.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [24509.713184] Pid: 22495, comm: mdt00_074 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [24509.723356] Call Trace: [24509.725825] [] ldlm_completion_ast+0x4e5/0x860 [ptlrpc] [24509.732773] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [24509.739967] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [24509.746796] [] mdt_object_lock_internal+0x70/0x360 [mdt] [24509.753789] [] mdt_object_find_lock+0x6a/0x1a0 [mdt] [24509.760437] [] mdt_reint_setxattr+0x1ce/0xfd0 [mdt] [24509.767011] [] mdt_reint_rec+0x83/0x210 [mdt] [24509.773066] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [24509.779635] [] mdt_reint+0x67/0x140 [mdt] [24509.785328] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [24509.792271] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [24509.799985] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [24509.806311] [] kthread+0xd1/0xe0 [24509.811226] [] ret_from_fork_nospec_begin+0xe/0x21 [24509.817701] [] 0xffffffffffffffff [24509.822728] LustreError: dumping log to /tmp/lustre-log.1584576512.22495 [24628.794237] LNet: Service thread pid 22495 completed after 1321.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [33910.420966] perf: interrupt took too long (3133 > 3132), lowering kernel.perf_event_max_sample_rate to 63000 [38102.215177] perf: interrupt took too long (3981 > 3916), lowering kernel.perf_event_max_sample_rate to 50000 [43137.516477] Lustre: fir-MDT0001: haven't heard from client ca4d9d7f-c632-4 (at 10.50.9.23@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add59baac00, cur 1584595140 expire 1584594990 last 1584594913 [43186.345696] Lustre: fir-MDT0001: Connection restored to ca4d9d7f-c632-4 (at 10.50.9.23@o2ib2) [43186.354226] Lustre: Skipped 2 previous similar messages [43910.538333] Lustre: fir-MDT0001: haven't heard from client f4363950-d6c3-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd77522000, cur 1584595913 expire 1584595763 last 1584595686 [43910.558213] Lustre: Skipped 1 previous similar message [43967.217898] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [43967.226429] Lustre: Skipped 1 previous similar message [43992.424555] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [50169.719951] Lustre: fir-MDT0001: haven't heard from client eceee209-ec05-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d54a84400, cur 1584602172 expire 1584602022 last 1584601945 [50169.739835] Lustre: Skipped 1 previous similar message [50182.716536] Lustre: fir-MDT0000: haven't heard from client eceee209-ec05-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add4dff2c00, cur 1584602185 expire 1584602035 last 1584601958 [50245.624479] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [50270.655226] Lustre: fir-MDT0000: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [68360.239324] Lustre: fir-MDT0000: haven't heard from client 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add76175000, cur 1584620362 expire 1584620212 last 1584620135 [68415.081547] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [68440.173840] Lustre: fir-MDT0000: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [86931.760750] Lustre: fir-MDT0001: haven't heard from client a2a51302-74aa-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd62e01000, cur 1584638933 expire 1584638783 last 1584638706 [86931.780646] Lustre: Skipped 1 previous similar message [86986.392716] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [87011.440747] Lustre: fir-MDT0000: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [91636.520364] Lustre: 22377:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584643630/real 1584643630] req@ffff8add397b5e80 x1661526924243520/t0(0) o104->fir-MDT0001@10.50.8.20@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584643637 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [91643.547547] Lustre: 22377:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584643637/real 1584643637] req@ffff8add397b5e80 x1661526924243520/t0(0) o104->fir-MDT0001@10.50.8.20@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584643644 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [91650.574737] Lustre: 22377:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584643644/real 1584643644] req@ffff8add397b5e80 x1661526924243520/t0(0) o104->fir-MDT0001@10.50.8.20@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584643651 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [91653.888839] Lustre: fir-MDT0000: haven't heard from client e584cba3-332a-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd453d3000, cur 1584643655 expire 1584643505 last 1584643428 [91653.908732] Lustre: Skipped 1 previous similar message [91653.931195] LustreError: 22377:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.8.20@o2ib2) failed to reply to blocking AST (req@ffff8add397b5e80 x1661526924243520 status 0 rc -5), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ad4ef1fc140/0xbe1cb0f24ac7943a lrc: 4/0,0 mode: PR/PR res: [0x240000402:0x5:0x0].0x0 bits 0x13/0x0 rrc: 58 type: IBT flags: 0x50a01400000020 nid: 10.50.8.20@o2ib2 remote: 0x2557de43b4f016d6 expref: 8 pid: 22343 timeout: 91796 lvb_type: 0 [91653.973421] LustreError: 138-a: fir-MDT0001: A client on nid 10.50.8.20@o2ib2 was evicted due to a lock blocking callback time out: rc -5 [91704.745829] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [91729.761992] Lustre: fir-MDT0000: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [118301.503368] LNet: Service thread pid 22449 was inactive for 200.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [118301.520404] Pid: 22449, comm: mdt00_045 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [118301.530687] Call Trace: [118301.533258] [] call_rwsem_down_write_failed+0x17/0x30 [118301.540134] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [118301.547173] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [118301.554729] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [118301.561922] [] llog_process_thread+0x82f/0x18e0 [obdclass] [118301.569247] [] llog_process_or_fork+0xbc/0x450 [obdclass] [118301.576464] [] llog_cat_process_cb+0x239/0x250 [obdclass] [118301.583717] [] llog_process_thread+0x82f/0x18e0 [obdclass] [118301.591022] [] llog_process_or_fork+0xbc/0x450 [obdclass] [118301.598228] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [118301.605864] [] llog_cat_process+0x2e/0x30 [obdclass] [118301.612652] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [118301.620181] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [118301.627392] [] mdd_changelog_clear+0x503/0x690 [mdd] [118301.634174] [] mdd_iocontrol+0x163/0x540 [mdd] [118301.640410] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [118301.646668] [] mdt_set_info+0x484/0x490 [mdt] [118301.652838] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [118301.659929] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [118301.667779] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [118301.674236] [] kthread+0xd1/0xe0 [118301.679238] [] ret_from_fork_nospec_begin+0xe/0x21 [118301.685807] [] 0xffffffffffffffff [118301.690941] LustreError: dumping log to /tmp/lustre-log.1584670302.22449 [118306.874717] LNet: Service thread pid 22449 completed after 205.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [121746.599515] Lustre: fir-MDT0000: Client 0219ca75-6687-4 (at 10.50.1.2@o2ib2) reconnecting [121746.607864] Lustre: Skipped 2 previous similar messages [121746.613250] Lustre: fir-MDT0000: Connection restored to 0219ca75-6687-4 (at 10.50.1.2@o2ib2) [123085.278056] Lustre: fir-MDT0000: Client 1b87e37c-9096-4 (at 10.50.1.29@o2ib2) reconnecting [123085.286501] Lustre: fir-MDT0000: Connection restored to 1b87e37c-9096-4 (at 10.50.1.29@o2ib2) [123670.995606] LNet: Service thread pid 22343 was inactive for 228.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [123671.012639] Pid: 22343, comm: mdt00_009 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [123671.022935] Call Trace: [123671.025501] [] call_rwsem_down_write_failed+0x17/0x30 [123671.032344] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [123671.039392] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [123671.046931] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [123671.054118] [] llog_process_thread+0x82f/0x18e0 [obdclass] [123671.061399] [] llog_process_or_fork+0xbc/0x450 [obdclass] [123671.068617] [] llog_cat_process_cb+0x239/0x250 [obdclass] [123671.075809] [] llog_process_thread+0x82f/0x18e0 [obdclass] [123671.083110] [] llog_process_or_fork+0xbc/0x450 [obdclass] [123671.090327] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [123671.097954] [] llog_cat_process+0x2e/0x30 [obdclass] [123671.104714] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [123671.112236] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [123671.119424] [] mdd_changelog_clear+0x503/0x690 [mdd] [123671.126174] [] mdd_iocontrol+0x163/0x540 [mdd] [123671.132405] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [123671.138639] [] mdt_set_info+0x484/0x490 [mdt] [123671.144784] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [123671.151862] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [123671.159680] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [123671.166121] [] kthread+0xd1/0xe0 [123671.171146] [] ret_from_fork_nospec_begin+0xe/0x21 [123671.177724] [] 0xffffffffffffffff [123671.182843] LustreError: dumping log to /tmp/lustre-log.1584675671.22343 [123878.773277] LNet: Service thread pid 22343 completed after 436.18s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [130815.709847] Lustre: fir-MDT0001: Connection restored to 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) [130815.968496] Lustre: fir-MDT0000: haven't heard from client 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add66fb8400, cur 1584682816 expire 1584682666 last 1584682589 [130815.988377] Lustre: Skipped 1 previous similar message [130825.968586] Lustre: fir-MDT0001: haven't heard from client 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d7855bc00, cur 1584682826 expire 1584682676 last 1584682599 [130840.718591] Lustre: fir-MDT0000: Connection restored to 4f32e72b-02b6-4 (at 10.50.9.6@o2ib2) [133306.043212] Lustre: fir-MDT0000: haven't heard from client 417f1855-bc48-4 (at 10.50.9.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed795be800, cur 1584685306 expire 1584685156 last 1584685079 [133327.040154] Lustre: fir-MDT0001: haven't heard from client 417f1855-bc48-4 (at 10.50.9.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d723a5400, cur 1584685327 expire 1584685177 last 1584685100 [161831.889624] LustreError: 52772:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.9.7@o2ib2 arrived at 1584713831 with bad export cookie 13699018700021419089 [161831.889947] Lustre: fir-MDT0001: Connection restored to 417f1855-bc48-4 (at 10.50.9.7@o2ib2) [161831.913729] LustreError: 52772:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 43 previous similar messages [162058.869213] Lustre: fir-MDT0001: haven't heard from client 417f1855-bc48-4 (at 10.50.9.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0aa2219000, cur 1584714058 expire 1584713908 last 1584713831 [172899.686015] Lustre: 21898:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584724297/real 1584724297] req@ffff8aed52b26780 x1661529201460352/t0(0) o6->fir-OST0045-osc-MDT0001@10.0.10.112@o2ib7:28/4 lens 544/432 e 24 to 1 dl 1584724898 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [172899.714316] Lustre: fir-OST0045-osc-MDT0001: Connection to fir-OST0045 (at 10.0.10.112@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [172899.754490] Lustre: fir-OST0045-osc-MDT0001: Connection restored to 10.0.10.112@o2ib7 (at 10.0.10.112@o2ib7) [172899.764402] Lustre: Skipped 1 previous similar message [172900.790053] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584724297/real 1584724297] req@ffff8acfb241f080 x1661529201460608/t0(0) o6->fir-OST005f-osc-MDT0001@10.0.10.116@o2ib7:28/4 lens 544/432 e 24 to 1 dl 1584724898 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [172900.818348] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [172900.828099] Lustre: fir-OST005f-osc-MDT0001: Connection to fir-OST005f (at 10.0.10.116@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [172900.844260] Lustre: Skipped 1 previous similar message [172900.873414] Lustre: fir-OST005f-osc-MDT0001: Connection restored to 10.0.10.116@o2ib7 (at 10.0.10.116@o2ib7) [172900.883325] Lustre: Skipped 1 previous similar message [178930.389143] Lustre: 21888:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584730328/real 1584730328] req@ffff8b0d78676c00 x1661529431227200/t0(0) o6->fir-OST004f-osc-MDT0001@10.0.10.114@o2ib7:28/4 lens 544/432 e 24 to 1 dl 1584730929 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [178930.417603] Lustre: fir-OST004f-osc-MDT0001: Connection to fir-OST004f (at 10.0.10.114@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [178930.457964] Lustre: fir-OST004f-osc-MDT0001: Connection restored to 10.0.10.114@o2ib7 (at 10.0.10.114@o2ib7) [184063.029773] Lustre: fir-MDT0000: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [184063.038472] Lustre: fir-MDT0000: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [184097.154930] Lustre: fir-MDT0001: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [184097.163393] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [184100.544549] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [184100.561980] LustreError: Skipped 1367 previous similar messages [184175.810434] Lustre: fir-MDT0000: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [184175.818829] Lustre: fir-MDT0000: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [184225.987897] Lustre: fir-MDT0001: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [184225.996361] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [186873.555497] Lustre: fir-MDT0001: haven't heard from client 2bdad291-4dd1-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aecb36fc000, cur 1584738872 expire 1584738722 last 1584738645 [186873.575518] Lustre: Skipped 1 previous similar message [186927.005442] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [186951.956753] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [187101.670058] Lustre: fir-MDT0000: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [187101.678524] Lustre: fir-MDT0000: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [187212.069354] Lustre: fir-MDT0001: Client eb57335a-b614-4 (at 10.49.0.62@o2ib1) reconnecting [187212.077868] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [187212.330414] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [187212.347880] LustreError: Skipped 1 previous similar message [187328.568468] Lustre: fir-MDT0000: haven't heard from client eb57335a-b614-4 (at 10.49.0.62@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed052de400, cur 1584739327 expire 1584739177 last 1584739100 [187328.588469] Lustre: Skipped 1 previous similar message [187438.569938] Lustre: fir-MDT0001: haven't heard from client eb57335a-b614-4 (at 10.49.0.62@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5c091000, cur 1584739437 expire 1584739287 last 1584739210 [187705.578297] Lustre: fir-MDT0001: haven't heard from client 22cd96b1-4e55-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0393725800, cur 1584739704 expire 1584739554 last 1584739477 [187747.704435] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [187772.717248] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [189078.614503] Lustre: fir-MDT0001: haven't heard from client 4119ac7e-3e26-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af3b0ec1400, cur 1584741077 expire 1584740927 last 1584740850 [189078.634514] Lustre: Skipped 1 previous similar message [189108.479887] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [189133.496882] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [190840.663561] Lustre: fir-MDT0001: haven't heard from client e1f1914e-b10c-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afbc66c2400, cur 1584742839 expire 1584742689 last 1584742612 [190840.683616] Lustre: Skipped 1 previous similar message [190892.531566] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [190917.551645] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [192621.436689] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [192646.530670] Lustre: fir-MDT0000: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [192850.051005] Lustre: fir-MDT0001: Connection restored to d8ba59b7-d352-4 (at 10.50.10.30@o2ib2) [192857.718430] Lustre: fir-MDT0001: haven't heard from client d8ba59b7-d352-4 (at 10.50.10.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6f892000, cur 1584744856 expire 1584744706 last 1584744629 [192857.738516] Lustre: Skipped 1 previous similar message [192874.994366] Lustre: fir-MDT0000: Connection restored to d8ba59b7-d352-4 (at 10.50.10.30@o2ib2) [194054.751002] Lustre: fir-MDT0001: haven't heard from client 7521def8-3cfd-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec33f17400, cur 1584746053 expire 1584745903 last 1584745826 [194054.770999] Lustre: Skipped 1 previous similar message [194279.900094] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [194304.960281] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [200051.915675] Lustre: fir-MDT0001: haven't heard from client 1a29ad5f-9961-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd6993a800, cur 1584752050 expire 1584751900 last 1584751823 [200051.935672] Lustre: Skipped 1 previous similar message [200099.687049] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [200124.739354] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [201580.958702] Lustre: fir-MDT0001: haven't heard from client 34d41e1f-f6fe-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d3af5c000, cur 1584753579 expire 1584753429 last 1584753352 [201580.978671] Lustre: Skipped 1 previous similar message [201627.043444] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [201652.101015] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [202907.997739] Lustre: fir-MDT0001: haven't heard from client 809d0a51-afb6-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b06e776d400, cur 1584754906 expire 1584754756 last 1584754679 [202908.017718] Lustre: Skipped 1 previous similar message [202962.005705] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [202987.010385] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [204764.343410] LNet: Service thread pid 22383 was inactive for 200.59s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [204764.360453] Pid: 22383, comm: mdt00_017 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [204764.370720] Call Trace: [204764.373282] [] call_rwsem_down_write_failed+0x17/0x30 [204764.380144] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [204764.387227] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [204764.394794] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [204764.402047] [] llog_process_thread+0x82f/0x18e0 [obdclass] [204764.409382] [] llog_process_or_fork+0xbc/0x450 [obdclass] [204764.416644] [] llog_cat_process_cb+0x239/0x250 [obdclass] [204764.423910] [] llog_process_thread+0x82f/0x18e0 [obdclass] [204764.431221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [204764.438451] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [204764.446105] [] llog_cat_process+0x2e/0x30 [obdclass] [204764.452914] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [204764.460462] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [204764.467657] [] mdd_changelog_clear+0x503/0x690 [mdd] [204764.474424] [] mdd_iocontrol+0x163/0x540 [mdd] [204764.480682] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [204764.486952] [] mdt_set_info+0x484/0x490 [mdt] [204764.493137] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [204764.500204] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [204764.508070] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [204764.514527] [] kthread+0xd1/0xe0 [204764.519561] [] ret_from_fork_nospec_begin+0xe/0x21 [204764.526138] [] 0xffffffffffffffff [204764.531273] LustreError: dumping log to /tmp/lustre-log.1584756762.22383 [204851.607787] LNet: Service thread pid 22383 completed after 287.85s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [205655.248280] LNet: Service thread pid 22479 was inactive for 248.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [205655.265322] Pid: 22479, comm: mdt00_063 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [205655.275611] Call Trace: [205655.278170] [] call_rwsem_down_write_failed+0x17/0x30 [205655.284995] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [205655.292015] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [205655.299592] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [205655.306792] [] llog_process_thread+0x82f/0x18e0 [obdclass] [205655.314110] [] llog_process_or_fork+0xbc/0x450 [obdclass] [205655.321317] [] llog_cat_process_cb+0x239/0x250 [obdclass] [205655.328536] [] llog_process_thread+0x82f/0x18e0 [obdclass] [205655.335855] [] llog_process_or_fork+0xbc/0x450 [obdclass] [205655.343083] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [205655.350716] [] llog_cat_process+0x2e/0x30 [obdclass] [205655.357512] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [205655.365079] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [205655.372314] [] mdd_changelog_clear+0x503/0x690 [mdd] [205655.379076] [] mdd_iocontrol+0x163/0x540 [mdd] [205655.385335] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [205655.391581] [] mdt_set_info+0x484/0x490 [mdt] [205655.397736] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [205655.404836] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [205655.412687] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [205655.419159] [] kthread+0xd1/0xe0 [205655.424185] [] ret_from_fork_nospec_begin+0xe/0x21 [205655.430747] [] 0xffffffffffffffff [205655.435865] LustreError: dumping log to /tmp/lustre-log.1584757653.22479 [205761.799630] LNet: Service thread pid 22479 completed after 354.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [208659.157134] Lustre: fir-MDT0001: haven't heard from client 8d994856-6641-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af800afd800, cur 1584760657 expire 1584760507 last 1584760430 [208659.177250] Lustre: Skipped 1 previous similar message [208706.763640] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [208731.785263] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [210539.207357] Lustre: fir-MDT0001: haven't heard from client 83ad7733-be39-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b09da258c00, cur 1584762537 expire 1584762387 last 1584762310 [210539.227348] Lustre: Skipped 1 previous similar message [210647.522507] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [210672.610768] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [215350.935028] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.8.58@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [217924.409098] Lustre: fir-MDT0001: haven't heard from client e312085b-9063-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae833ef5400, cur 1584769922 expire 1584769772 last 1584769695 [217924.429169] Lustre: Skipped 1 previous similar message [219085.527643] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [219110.582895] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [225559.620819] Lustre: fir-MDT0001: haven't heard from client 3e58d365-2c0a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b085538dc00, cur 1584777557 expire 1584777407 last 1584777330 [225559.640795] Lustre: Skipped 1 previous similar message [225598.126339] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [225623.151172] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [242552.787868] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [242577.845520] Lustre: fir-MDT0000: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [242612.108282] Lustre: fir-MDT0000: haven't heard from client e71e196d-00dc-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed79ffec00, cur 1584794609 expire 1584794459 last 1584794382 [242612.128255] Lustre: Skipped 1 previous similar message [252468.392627] Lustre: fir-MDT0000: haven't heard from client 55eb8d67-6a00-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd10f5b400, cur 1584804465 expire 1584804315 last 1584804238 [252468.412597] Lustre: Skipped 1 previous similar message [252503.392738] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [252528.460568] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [287323.532979] Lustre: fir-MDT0001: Connection restored to a47ccd42-c337-4 (at 10.50.17.43@o2ib2) [287325.367466] Lustre: fir-MDT0001: haven't heard from client a47ccd42-c337-4 (at 10.50.17.43@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d4c0aa000, cur 1584839321 expire 1584839171 last 1584839094 [287325.387526] Lustre: Skipped 1 previous similar message [287348.574772] Lustre: fir-MDT0000: Connection restored to a47ccd42-c337-4 (at 10.50.17.43@o2ib2) [290076.447185] Lustre: fir-MDT0000: haven't heard from client 59748e26-ab50-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acda6535800, cur 1584842072 expire 1584841922 last 1584841845 [290076.467170] Lustre: Skipped 1 previous similar message [290108.872649] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [290133.933930] Lustre: fir-MDT0000: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [347587.078885] Lustre: fir-MDT0000: haven't heard from client 28cde8ea-b100-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec53ebe800, cur 1584899581 expire 1584899431 last 1584899354 [347587.098854] Lustre: Skipped 1 previous similar message [347626.648590] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [347651.583601] Lustre: fir-MDT0000: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [353114.575496] Lustre: 22467:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353128.308708] Lustre: 22443:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353151.339161] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353180.033547] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353207.880467] Lustre: 67111:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353238.220475] Lustre: 22417:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353264.172224] Lustre: 22490:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353303.013220] Lustre: 22463:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353379.766443] Lustre: 22447:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353379.777957] Lustre: 22447:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2 previous similar messages [353511.220420] Lustre: 22432:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353511.231912] Lustre: 22432:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 4 previous similar messages [353769.122747] Lustre: 22501:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [353769.134295] Lustre: 22501:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 8 previous similar messages [354279.087929] Lustre: fir-MDT0000: Client 9072f876-fcd1-4 (at 10.50.1.22@o2ib2) reconnecting [354279.096596] Lustre: fir-MDT0000: Connection restored to 9072f876-fcd1-4 (at 10.50.1.22@o2ib2) [354285.981116] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [354285.992702] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 18 previous similar messages [354816.894126] Lustre: fir-MDT0000: Client b21bbe85-153e-4 (at 10.50.1.24@o2ib2) reconnecting [354816.904330] Lustre: fir-MDT0000: Connection restored to b21bbe85-153e-4 (at 10.50.1.24@o2ib2) [354892.166280] Lustre: 22409:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [354892.177890] Lustre: 22409:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 24 previous similar messages [354996.205137] Lustre: fir-MDT0000: Client c3fa2b6b-5cac-4 (at 10.50.8.48@o2ib2) reconnecting [354996.214262] Lustre: fir-MDT0000: Connection restored to c3fa2b6b-5cac-4 (at 10.50.8.48@o2ib2) [355496.938318] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [355496.949865] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 19 previous similar messages [356098.065624] Lustre: 22425:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [356098.077159] Lustre: 22425:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 20 previous similar messages [356714.768949] Lustre: 22389:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [356714.780469] Lustre: 22389:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 19 previous similar messages [356951.257678] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [356951.266754] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357061.274570] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [357061.282944] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357071.190326] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [357071.198803] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357117.378925] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [357117.387300] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357163.139425] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [357163.147812] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357242.020595] Lustre: fir-MDT0001: Client db23ff47-83b4-4 (at 10.50.2.7@o2ib2) reconnecting [357242.028999] Lustre: fir-MDT0001: Connection restored to db23ff47-83b4-4 (at 10.50.2.7@o2ib2) [357357.833097] Lustre: 22445:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [357357.844674] Lustre: 22445:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 19 previous similar messages [357813.851537] Lustre: fir-MDT0000: Client c8932097-dc54-4 (at 10.50.10.41@o2ib2) reconnecting [357813.860041] Lustre: Skipped 1 previous similar message [357813.865383] Lustre: fir-MDT0000: Connection restored to c8932097-dc54-4 (at 10.50.10.41@o2ib2) [357813.874148] Lustre: Skipped 1 previous similar message [357967.072237] Lustre: 67111:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [357967.083789] Lustre: 67111:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 21 previous similar messages [358379.731383] Lustre: fir-MDT0000: Client ab9422cb-105a-4 (at 10.50.1.48@o2ib2) reconnecting [358379.740626] Lustre: fir-MDT0000: Connection restored to ab9422cb-105a-4 (at 10.50.1.48@o2ib2) [358596.013831] Lustre: 22302:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [358596.025361] Lustre: 22302:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 19 previous similar messages [358659.502369] Lustre: fir-MDT0000: Client 98e25bb0-7d9e-4 (at 10.50.8.55@o2ib2) reconnecting [358659.510821] Lustre: fir-MDT0000: Connection restored to 98e25bb0-7d9e-4 (at 10.50.8.55@o2ib2) [358886.384989] Lustre: fir-MDT0000: Client 04a8c3ea-d6d0-4 (at 10.50.10.43@o2ib2) reconnecting [358886.393534] Lustre: fir-MDT0000: Connection restored to 04a8c3ea-d6d0-4 (at 10.50.10.43@o2ib2) [359223.572393] Lustre: 22383:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [359223.583926] Lustre: 22383:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 10 previous similar messages [359882.473079] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [359882.484642] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 12 previous similar messages [360445.986307] Lustre: fir-MDT0000: Client 0aee55bb-3015-4 (at 10.50.8.37@o2ib2) reconnecting [360445.994680] Lustre: Skipped 1 previous similar message [360445.999947] Lustre: fir-MDT0000: Connection restored to 0aee55bb-3015-4 (at 10.50.8.37@o2ib2) [360446.008585] Lustre: Skipped 1 previous similar message [360490.083780] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [360490.095318] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 10 previous similar messages [361110.546140] Lustre: 22365:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [361110.557710] Lustre: 22365:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 9 previous similar messages [361221.184869] Lustre: fir-MDT0000: Client c3fa2b6b-5cac-4 (at 10.50.8.48@o2ib2) reconnecting [361221.193360] Lustre: fir-MDT0000: Connection restored to c3fa2b6b-5cac-4 (at 10.50.8.48@o2ib2) [361730.705033] Lustre: 22420:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [361730.716532] Lustre: 22420:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 17 previous similar messages [361772.332584] Lustre: fir-MDT0000: Client fab83063-bce6-4 (at 10.50.8.42@o2ib2) reconnecting [361772.341106] Lustre: fir-MDT0000: Connection restored to fab83063-bce6-4 (at 10.50.8.42@o2ib2) [362366.910571] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [362366.922063] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 8 previous similar messages [362974.608582] Lustre: 22412:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [362974.620088] Lustre: 22412:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 6 previous similar messages [363612.849497] Lustre: 22449:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [363612.861029] Lustre: 22449:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 14 previous similar messages [363716.473204] Lustre: fir-MDT0000: Client 499df69a-831f-4 (at 10.50.8.56@o2ib2) reconnecting [363716.481681] Lustre: fir-MDT0000: Connection restored to 499df69a-831f-4 (at 10.50.8.56@o2ib2) [364055.090789] LNet: Service thread pid 22389 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [364055.107910] Pid: 22389, comm: mdt00_019 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [364055.118253] Call Trace: [364055.120854] [] call_rwsem_down_write_failed+0x17/0x30 [364055.127719] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [364055.134779] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [364055.142406] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [364055.149646] [] llog_process_thread+0x82f/0x18e0 [obdclass] [364055.156948] [] llog_process_or_fork+0xbc/0x450 [obdclass] [364055.164176] [] llog_cat_process_cb+0x239/0x250 [obdclass] [364055.171416] [] llog_process_thread+0x82f/0x18e0 [obdclass] [364055.178712] [] llog_process_or_fork+0xbc/0x450 [obdclass] [364055.185959] [] llog_cat_process_or_fork+0x17e/0x360 [obdclass] [364055.193612] [] llog_cat_process+0x2e/0x30 [obdclass] [364055.200426] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [364055.208042] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [364055.215297] [] mdd_changelog_clear+0x503/0x690 [mdd] [364055.222073] [] mdd_iocontrol+0x163/0x540 [mdd] [364055.228339] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [364055.234660] [] mdt_set_info+0x484/0x490 [mdt] [364055.240871] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [364055.248047] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [364055.255915] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [364055.262393] [] kthread+0xd1/0xe0 [364055.267466] [] ret_from_fork_nospec_begin+0xe/0x21 [364055.274100] [] 0xffffffffffffffff [364055.279296] LustreError: dumping log to /tmp/lustre-log.1584916048.22389 [364239.812486] Lustre: fir-MDT0000: Client 9072f876-fcd1-4 (at 10.50.1.22@o2ib2) reconnecting [364239.821002] Lustre: fir-MDT0000: Connection restored to 9072f876-fcd1-4 (at 10.50.1.22@o2ib2) [364241.779119] LNet: Service thread pid 22389 completed after 386.68s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [364241.829311] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [364241.840864] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 4 previous similar messages [364960.026394] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [364960.037936] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 4 previous similar messages [365310.455019] Lustre: fir-MDT0000: Client b21bbe85-153e-4 (at 10.50.1.24@o2ib2) reconnecting [365310.464165] Lustre: fir-MDT0000: Connection restored to b21bbe85-153e-4 (at 10.50.1.24@o2ib2) [365354.797708] Lustre: fir-MDT0000: Client b21bbe85-153e-4 (at 10.50.1.24@o2ib2) reconnecting [365354.807542] Lustre: fir-MDT0000: Connection restored to b21bbe85-153e-4 (at 10.50.1.24@o2ib2) [365621.702967] Lustre: 22288:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [365621.714563] Lustre: 22288:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 11 previous similar messages [366275.208942] Lustre: 67107:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [366275.220508] Lustre: 67107:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 8 previous similar messages [367001.218763] LNet: Service thread pid 28971 was inactive for 260.66s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [367001.235927] Pid: 28971, comm: mdt00_087 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [367001.246237] Call Trace: [367001.248822] [] call_rwsem_down_write_failed+0x17/0x30 [367001.255699] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [367001.262813] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [367001.270411] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [367001.277606] [] llog_process_thread+0x82f/0x18e0 [obdclass] [367001.284971] [] llog_process_or_fork+0xbc/0x450 [obdclass] [367001.292223] [] llog_cat_process_cb+0x239/0x250 [obdclass] [367001.299459] [] llog_process_thread+0x82f/0x18e0 [obdclass] [367001.306787] [] llog_process_or_fork+0xbc/0x450 [obdclass] [367001.314081] [] llog_cat_process_or_fork+0x17e/0x360 [obdclass] [367001.321741] [] llog_cat_process+0x2e/0x30 [obdclass] [367001.328594] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [367001.336154] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [367001.343354] [] mdd_changelog_clear+0x503/0x690 [mdd] [367001.350158] [] mdd_iocontrol+0x163/0x540 [mdd] [367001.356397] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [367001.362672] [] mdt_set_info+0x484/0x490 [mdt] [367001.368851] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [367001.376029] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [367001.383914] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [367001.390439] [] kthread+0xd1/0xe0 [367001.395477] [] ret_from_fork_nospec_begin+0xe/0x21 [367001.402103] [] 0xffffffffffffffff [367001.407293] LustreError: dumping log to /tmp/lustre-log.1584918994.28971 [367074.929761] LNet: Service thread pid 28971 completed after 334.37s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [367075.004966] Lustre: 22302:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [367075.016619] Lustre: 22302:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 7 previous similar messages [367717.091063] Lustre: 22467:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [367717.102625] Lustre: 22467:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 17 previous similar messages [367777.102826] Lustre: fir-MDT0000: Client fab83063-bce6-4 (at 10.50.8.42@o2ib2) reconnecting [367777.111295] Lustre: fir-MDT0000: Connection restored to fab83063-bce6-4 (at 10.50.8.42@o2ib2) [367777.810583] Lustre: fir-MDT0000: Client f5c72d15-474b-4 (at 10.50.1.8@o2ib2) reconnecting [367777.818914] Lustre: fir-MDT0000: Connection restored to f5c72d15-474b-4 (at 10.50.1.8@o2ib2) [368329.338995] Lustre: 22343:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [368329.350519] Lustre: 22343:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 28 previous similar messages [368938.796347] Lustre: 22483:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [368938.807890] Lustre: 22483:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 32 previous similar messages [369565.056087] Lustre: 21981:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [369565.067622] Lustre: 21981:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 20 previous similar messages [369966.124624] Lustre: fir-MDT0001: Client 66901ab6-bc8e-4 (at 10.50.17.41@o2ib2) reconnecting [369966.133603] Lustre: fir-MDT0001: Connection restored to 66901ab6-bc8e-4 (at 10.50.17.41@o2ib2) [370177.932766] Lustre: 22497:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [370177.944262] Lustre: 22497:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 36 previous similar messages [370557.666188] LNet: Service thread pid 22445 was inactive for 200.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [370557.683260] Pid: 22445, comm: mdt00_043 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [370557.693533] Call Trace: [370557.696081] [] 0xffffffffffffffff [370557.701201] LustreError: dumping log to /tmp/lustre-log.1584922551.22445 [370951.803661] Lustre: 22490:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8add39c02400 x1661808110765440/t0(0) o46->3dab8abe-e790-3878-3898-4444ee422524@10.0.10.3@o2ib7:525/0 lens 264/224 e 21 to 0 dl 1584922950 ref 2 fl Interpret:/0/0 rc 0/0 [370958.650985] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [370958.661104] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [371059.041442] LNet: Service thread pid 22445 completed after 701.43s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [375784.977038] Lustre: 22478:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584927771/real 1584927771] req@ffff8ad65775ad00 x1661537362492864/t0(0) o104->fir-MDT0001@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584927778 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [375792.004230] Lustre: 22478:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584927778/real 1584927778] req@ffff8ad65775ad00 x1661537362492864/t0(0) o104->fir-MDT0001@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584927785 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [375799.031416] Lustre: 22478:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584927785/real 1584927785] req@ffff8ad65775ad00 x1661537362492864/t0(0) o104->fir-MDT0001@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584927792 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [375806.058600] Lustre: 22478:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584927792/real 1584927792] req@ffff8ad65775ad00 x1661537362492864/t0(0) o104->fir-MDT0001@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584927799 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [375813.085792] Lustre: 22478:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584927799/real 1584927799] req@ffff8ad65775ad00 x1661537362492864/t0(0) o104->fir-MDT0001@10.50.9.37@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584927806 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [375815.843913] Lustre: fir-MDT0001: haven't heard from client 1a7a3e3a-e746-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aebff623000, cur 1584927809 expire 1584927659 last 1584927582 [375815.863883] Lustre: Skipped 1 previous similar message [375854.560826] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [375879.597409] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [376345.858090] Lustre: fir-MDT0001: haven't heard from client bc12991d-b630-4 (at 10.50.6.22@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d75ff5c00, cur 1584928339 expire 1584928189 last 1584928112 [376345.878063] Lustre: Skipped 1 previous similar message [407367.698405] Lustre: 22302:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407368.201588] Lustre: 22365:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407368.213330] Lustre: 22365:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 50 previous similar messages [407369.215227] Lustre: 22365:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407369.226963] Lustre: 22365:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 91 previous similar messages [407371.227595] Lustre: 22377:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407371.239340] Lustre: 22377:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 115 previous similar messages [407375.290966] Lustre: 22434:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407375.302715] Lustre: 22434:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 299 previous similar messages [407383.364758] Lustre: 22386:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407383.376501] Lustre: 22386:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 665 previous similar messages [407399.391039] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [407399.402783] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1192 previous similar messages [410729.822550] Lustre: fir-MDT0000: haven't heard from client c6acd997-7594-4 (at 10.50.9.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add4cf08400, cur 1584962722 expire 1584962572 last 1584962495 [410729.842518] Lustre: Skipped 1 previous similar message [410964.016055] Lustre: fir-MDT0000: Connection restored to c6acd997-7594-4 (at 10.50.9.72@o2ib2) [411190.837148] Lustre: fir-MDT0000: haven't heard from client c6acd997-7594-4 (at 10.50.9.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc5c7c7800, cur 1584963183 expire 1584963033 last 1584962956 [411190.857120] Lustre: Skipped 1 previous similar message [425921.784481] Lustre: fir-MDT0001: Connection restored to fd4f78ff-b4bc-4 (at 10.50.2.23@o2ib2) [425930.226292] Lustre: fir-MDT0001: Connection restored to 360f9d52-0c81-4 (at 10.50.2.36@o2ib2) [425946.811480] Lustre: fir-MDT0000: Connection restored to fd4f78ff-b4bc-4 (at 10.50.2.23@o2ib2) [425955.324260] Lustre: fir-MDT0000: Connection restored to 360f9d52-0c81-4 (at 10.50.2.36@o2ib2) [425955.332882] Lustre: Skipped 1 previous similar message [425973.175004] Lustre: fir-MDT0000: Connection restored to 14fe6180-7b80-4 (at 10.49.27.34@o2ib1) [425986.250686] Lustre: fir-MDT0001: haven't heard from client 144964dd-491d-4 (at 10.50.4.19@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d654e1400, cur 1584977978 expire 1584977828 last 1584977751 [425988.519105] Lustre: fir-MDT0001: Connection restored to c11f79a1-0fcf-4 (at 10.50.5.1@o2ib2) [425988.527638] Lustre: Skipped 1 previous similar message [426012.911522] Lustre: fir-MDT0001: Connection restored to 05b2fcf3-a8de-4 (at 10.50.8.9@o2ib2) [426012.920055] Lustre: Skipped 4 previous similar messages [426045.315573] Lustre: fir-MDT0001: Connection restored to 72a94371-b86b-4 (at 10.50.9.67@o2ib2) [426045.324193] Lustre: Skipped 8 previous similar messages [426109.783480] Lustre: fir-MDT0000: Connection restored to bc12991d-b630-4 (at 10.50.6.22@o2ib2) [426109.792100] Lustre: Skipped 7 previous similar messages [427160.284844] Lustre: 28972:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [427160.296593] Lustre: 28972:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1967 previous similar messages [427164.291534] Lustre: 67111:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [427164.303276] Lustre: 67111:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 516 previous similar messages [427172.302189] Lustre: 22346:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [427172.313936] Lustre: 22346:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1053 previous similar messages [429913.128056] Lustre: fir-MDT0001: Connection restored to 7afa46a0-f8e2-4 (at 10.50.17.42@o2ib2) [429913.136760] Lustre: Skipped 2 previous similar messages [429914.356342] Lustre: fir-MDT0001: haven't heard from client 7afa46a0-f8e2-4 (at 10.50.17.42@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6dc5bc00, cur 1584981906 expire 1584981756 last 1584981679 [429914.376411] Lustre: Skipped 23 previous similar messages [429938.113787] Lustre: fir-MDT0000: Connection restored to 7afa46a0-f8e2-4 (at 10.50.17.42@o2ib2) [443767.733997] Lustre: fir-MDT0001: haven't heard from client ce67c0db-bb4c-4 (at 10.50.15.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd77527c00, cur 1584995759 expire 1584995609 last 1584995532 [443767.753993] Lustre: Skipped 1 previous similar message [445041.347306] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997025/real 1584997025] req@ffff8b0cfc6a1680 x1661542389328128/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997032 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [445048.374639] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997032/real 1584997032] req@ffff8b0cfc6a1680 x1661542389328128/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997039 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445055.401704] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997039/real 1584997039] req@ffff8b0cfc6a1680 x1661542389328128/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997046 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445062.428944] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997046/real 1584997046] req@ffff8b0cfc6a1680 x1661542389328128/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997053 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445076.456263] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997060/real 1584997060] req@ffff8b0cfc6a1680 x1661542389328128/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997067 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445076.483646] Lustre: 53494:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [445096.004791] Lustre: 67111:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997080/real 1584997080] req@ffff8ad3daed1680 x1661542392586304/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997087 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445096.032140] Lustre: 67111:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [445128.769697] Lustre: fir-MDT0001: haven't heard from client 2e6641b9-e38a-4 (at 10.50.12.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add7961f800, cur 1584997120 expire 1584996970 last 1584996893 [445128.789741] Lustre: Skipped 5 previous similar messages [445131.042741] Lustre: 67111:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1584997115/real 1584997115] req@ffff8ad3daed1680 x1661542392586304/t0(0) o104->fir-MDT0000@10.50.12.9@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1584997122 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [445131.070153] Lustre: 67111:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [445188.496315] LustreError: 53494:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.12.9@o2ib2) failed to reply to blocking AST (req@ffff8b0cfc6a1680 x1661542389328128 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8b0d563fca40/0xbe1cb112564ea6ba lrc: 4/0,0 mode: PR/PR res: [0x200040a0d:0x15e6:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.50.12.9@o2ib2 remote: 0x780a4a19742800ed expref: 3211 pid: 23129 timeout: 445317 lvb_type: 0 [445188.539284] LustreError: 138-a: fir-MDT0000: A client on nid 10.50.12.9@o2ib2 was evicted due to a lock blocking callback time out: rc -110 [445188.552015] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.50.12.9@o2ib2 ns: mdt-fir-MDT0000_UUID lock: ffff8b0d563fca40/0xbe1cb112564ea6ba lrc: 3/0,0 mode: PR/PR res: [0x200040a0d:0x15e6:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.50.12.9@o2ib2 remote: 0x780a4a19742800ed expref: 3212 pid: 23129 timeout: 0 lvb_type: 0 [445204.771735] Lustre: fir-MDT0001: haven't heard from client 6d03d9a7-b1f2-4 (at 10.50.12.9@o2ib2) in 196 seconds. I think it's dead, and I am evicting it. exp ffff8add5c65ac00, cur 1584997196 expire 1584997046 last 1584997000 [445204.791717] Lustre: Skipped 1 previous similar message [445215.526974] Lustre: fir-MDT0001: Connection restored to 6d03d9a7-b1f2-4 (at 10.50.12.9@o2ib2) [445240.620406] Lustre: fir-MDT0000: Connection restored to 6d03d9a7-b1f2-4 (at 10.50.12.9@o2ib2) [445542.507248] Lustre: fir-MDT0001: Connection restored to 4304d775-beb4-4 (at 10.50.13.7@o2ib2) [445567.591415] Lustre: fir-MDT0000: Connection restored to 4304d775-beb4-4 (at 10.50.13.7@o2ib2) [445721.786842] Lustre: fir-MDT0001: haven't heard from client 645377df-0a0a-4 (at 10.50.14.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6f894400, cur 1584997713 expire 1584997563 last 1584997486 [445733.787740] Lustre: fir-MDT0000: haven't heard from client 645377df-0a0a-4 (at 10.50.14.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add79f5bc00, cur 1584997725 expire 1584997575 last 1584997498 [446186.808685] Lustre: fir-MDT0001: Connection restored to f8bb9b7d-b961-4 (at 10.50.14.9@o2ib2) [446270.801694] Lustre: fir-MDT0000: haven't heard from client f8bb9b7d-b961-4 (at 10.50.14.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add4e020000, cur 1584998262 expire 1584998112 last 1584998035 [446383.277514] Lustre: fir-MDT0001: Connection restored to 2e6641b9-e38a-4 (at 10.50.12.8@o2ib2) [446383.286141] Lustre: Skipped 1 previous similar message [446746.784478] Lustre: fir-MDT0001: Connection restored to c05f4f30-e5ee-4 (at 10.50.12.10@o2ib2) [446746.793196] Lustre: Skipped 1 previous similar message [446758.814621] Lustre: fir-MDT0001: haven't heard from client c05f4f30-e5ee-4 (at 10.50.12.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5c65a800, cur 1584998750 expire 1584998600 last 1584998523 [446758.834729] Lustre: Skipped 1 previous similar message [447024.946555] Lustre: fir-MDT0000: Connection restored to 645377df-0a0a-4 (at 10.50.14.10@o2ib2) [447024.955270] Lustre: Skipped 2 previous similar messages [447749.302796] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [447749.312331] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [447805.473975] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [447807.140576] Lustre: fir-MDT0001: Client b41c0c6f-84ba-4 (at 10.50.9.4@o2ib2) reconnecting [447856.581416] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [447901.835682] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [447945.963902] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [447990.793180] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [448037.053506] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [448133.327310] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [448133.335679] Lustre: Skipped 2 previous similar messages [448277.888681] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [448277.897072] Lustre: Skipped 3 previous similar messages [448366.788084] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [448366.796733] Lustre: Skipped 15 previous similar messages [448385.837060] LustreError: 22458:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8aed6c68d850 x1659351959244288/t0(0) o37->dbf6b7b0-ca0f-4@10.50.6.71@o2ib2:192/0 lens 448/440 e 0 to 0 dl 1585000382 ref 1 fl Interpret:/0/0 rc 0/0 [448563.356447] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [448563.364828] Lustre: Skipped 7 previous similar messages [448899.872371] Lustre: fir-MDT0000: haven't heard from client 50a6df69-dfc3-4 (at 10.50.12.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add795be800, cur 1585000891 expire 1585000741 last 1585000664 [448899.892370] Lustre: Skipped 1 previous similar message [448986.144013] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [448986.152641] Lustre: Skipped 16 previous similar messages [449108.699343] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [449108.707722] Lustre: Skipped 12 previous similar messages [449241.880508] Lustre: fir-MDT0001: haven't heard from client 2d9a52b2-e42c-4 (at 10.50.12.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786ba800, cur 1585001233 expire 1585001083 last 1585001006 [449241.900575] Lustre: Skipped 1 previous similar message [449740.582037] Lustre: fir-MDT0001: Client 51e8b540-d9f8-4 (at 10.50.2.41@o2ib2) reconnecting [449740.590468] Lustre: Skipped 10 previous similar messages [449740.595912] Lustre: fir-MDT0001: Connection restored to 51e8b540-d9f8-4 (at 10.50.2.41@o2ib2) [449740.604559] Lustre: Skipped 16 previous similar messages [450034.902298] Lustre: fir-MDT0000: haven't heard from client 2422a71a-3ef8-4 (at 10.50.14.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add4ca21c00, cur 1585002026 expire 1585001876 last 1585001799 [450034.922407] Lustre: Skipped 1 previous similar message [450041.902141] Lustre: fir-MDT0001: haven't heard from client e63c4438-c6e3-4 (at 10.50.15.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d12d000, cur 1585002033 expire 1585001883 last 1585001806 [450041.922139] Lustre: Skipped 2 previous similar messages [450510.472349] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [450510.480741] Lustre: Skipped 1 previous similar message [450510.486286] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [450510.495001] Lustre: Skipped 3 previous similar messages [450736.921893] Lustre: fir-MDT0001: haven't heard from client 38135a83-ec5c-4 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5c65e000, cur 1585002728 expire 1585002578 last 1585002501 [450736.941887] Lustre: Skipped 2 previous similar messages [451162.274717] Lustre: fir-MDT0001: Connection restored to e63c4438-c6e3-4 (at 10.50.15.8@o2ib2) [451162.283414] Lustre: Skipped 3 previous similar messages [451343.937137] Lustre: fir-MDT0000: haven't heard from client b2463005-ec38-4 (at 10.50.16.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acda6532000, cur 1585003335 expire 1585003185 last 1585003108 [451343.957150] Lustre: Skipped 1 previous similar message [451346.937202] Lustre: fir-MDT0001: haven't heard from client 0f689ff7-6991-4 (at 10.50.14.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d3b3d8400, cur 1585003338 expire 1585003188 last 1585003111 [451346.957306] Lustre: Skipped 11 previous similar messages [452236.206450] Lustre: fir-MDT0001: Connection restored to 0c02c6b3-fbbf-4 (at 10.50.13.14@o2ib2) [452236.215180] Lustre: Skipped 25 previous similar messages [452257.142808] Lustre: fir-MDT0000: Client 51387ec6-ed83-4 (at 10.50.10.48@o2ib2) reconnecting [452257.151380] Lustre: Skipped 1 previous similar message [452443.411879] Lustre: fir-MDT0000: Client 3b5165e5-30ac-4 (at 10.50.8.39@o2ib2) reconnecting [453231.917449] Lustre: fir-MDT0000: Client dba6cbec-f37c-4 (at 10.50.10.44@o2ib2) reconnecting [453231.925944] Lustre: Skipped 1 previous similar message [453231.931604] Lustre: fir-MDT0000: Connection restored to dba6cbec-f37c-4 (at 10.50.10.44@o2ib2) [453231.940354] Lustre: Skipped 20 previous similar messages [453627.870326] Lustre: fir-MDT0001: Client 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) reconnecting [453946.348250] Lustre: fir-MDT0001: Connection restored to 0f689ff7-6991-4 (at 10.50.14.4@o2ib2) [453946.356889] Lustre: Skipped 4 previous similar messages [455231.498678] Lustre: fir-MDT0000: Client 40019701-810a-4 (at 10.50.1.26@o2ib2) reconnecting [455231.507057] Lustre: Skipped 1 previous similar message [455231.512555] Lustre: fir-MDT0000: Connection restored to 40019701-810a-4 (at 10.50.1.26@o2ib2) [455231.521201] Lustre: Skipped 3 previous similar messages [455306.044279] Lustre: fir-MDT0001: haven't heard from client 482fd27b-a473-4 (at 10.50.13.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d562fd400, cur 1585007297 expire 1585007147 last 1585007070 [455306.064375] Lustre: Skipped 11 previous similar messages [455988.698572] Lustre: fir-MDT0000: Client 04a8c3ea-d6d0-4 (at 10.50.10.43@o2ib2) reconnecting [455988.707394] Lustre: fir-MDT0000: Connection restored to 04a8c3ea-d6d0-4 (at 10.50.10.43@o2ib2) [455988.716148] Lustre: Skipped 2 previous similar messages [456043.064169] Lustre: fir-MDT0001: haven't heard from client e1991612-ee0d-4 (at 10.50.14.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af8aea69800, cur 1585008034 expire 1585007884 last 1585007807 [456043.084291] Lustre: Skipped 1 previous similar message [456229.070219] Lustre: fir-MDT0001: haven't heard from client fbd4143c-cb04-4 (at 10.50.13.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b337800, cur 1585008220 expire 1585008070 last 1585007993 [456229.090266] Lustre: Skipped 1 previous similar message [456437.077575] Lustre: fir-MDT0001: haven't heard from client e4a1516e-1b3e-4 (at 10.50.12.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd77523000, cur 1585008428 expire 1585008278 last 1585008201 [456437.097671] Lustre: Skipped 1 previous similar message [456757.084484] Lustre: fir-MDT0001: haven't heard from client 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6f894c00, cur 1585008748 expire 1585008598 last 1585008521 [456757.104493] Lustre: Skipped 1 previous similar message [457178.101058] Lustre: fir-MDT0000: haven't heard from client 2ac9ee52-f29e-4 (at 10.49.29.1@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d98b400, cur 1585009169 expire 1585009019 last 1585008942 [457178.121038] Lustre: Skipped 1 previous similar message [457253.260972] Lustre: fir-MDT0001: Connection restored to 8530b1fc-5bf1-4 (at 10.50.14.13@o2ib2) [457278.338187] Lustre: fir-MDT0000: Connection restored to 8530b1fc-5bf1-4 (at 10.50.14.13@o2ib2) [457636.530004] Lustre: fir-MDT0001: Connection restored to e4a1516e-1b3e-4 (at 10.50.12.15@o2ib2) [457637.108533] Lustre: fir-MDT0001: haven't heard from client 4c86e85e-ac4f-4 (at 10.49.28.1@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d562ff800, cur 1585009628 expire 1585009478 last 1585009401 [457637.128617] Lustre: Skipped 1 previous similar message [457661.532159] Lustre: fir-MDT0000: Connection restored to e4a1516e-1b3e-4 (at 10.50.12.15@o2ib2) [458091.052883] Lustre: fir-MDT0001: Connection restored to 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) [458116.119657] Lustre: fir-MDT0000: Connection restored to 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) [458154.307858] Lustre: fir-MDT0001: Connection restored to fbd4143c-cb04-4 (at 10.50.13.8@o2ib2) [458287.486986] Lustre: fir-MDT0001: Connection restored to 2672fd12-19fb-4 (at 10.50.12.7@o2ib2) [458287.495611] Lustre: Skipped 1 previous similar message [458455.223509] Lustre: fir-MDT0001: Client 7ff64d49-f30c-4 (at 10.50.2.32@o2ib2) reconnecting [458455.231963] Lustre: fir-MDT0001: Connection restored to 7ff64d49-f30c-4 (at 10.50.2.32@o2ib2) [458455.240601] Lustre: Skipped 1 previous similar message [458578.971407] Lustre: fir-MDT0000: Client 6611c29b-6891-4 (at 10.50.8.53@o2ib2) reconnecting [458991.412218] Lustre: fir-MDT0001: Connection restored to 4c86e85e-ac4f-4 (at 10.49.28.1@o2ib1) [458991.420861] Lustre: Skipped 3 previous similar messages [460476.573203] Lustre: fir-MDT0001: Client dbf6b7b0-ca0f-4 (at 10.50.6.71@o2ib2) reconnecting [460476.581688] Lustre: fir-MDT0001: Connection restored to dbf6b7b0-ca0f-4 (at 10.50.6.71@o2ib2) [460476.590348] Lustre: Skipped 3 previous similar messages [461244.540110] Lustre: fir-MDT0001: Connection restored to ab87f0a5-0357-4 (at 10.49.29.5@o2ib1) [461269.561812] Lustre: fir-MDT0000: Connection restored to ab87f0a5-0357-4 (at 10.49.29.5@o2ib1) [461285.664562] Lustre: fir-MDT0001: Client bf9c0359-a266-4 (at 10.50.0.1@o2ib2) reconnecting [462036.634005] Lustre: fir-MDT0000: Client f9de01a6-d260-4 (at 10.50.8.54@o2ib2) reconnecting [462036.643868] Lustre: fir-MDT0000: Connection restored to f9de01a6-d260-4 (at 10.50.8.54@o2ib2) [462036.652490] Lustre: Skipped 1 previous similar message [462082.767737] Lustre: fir-MDT0000: Client f9de01a6-d260-4 (at 10.50.8.54@o2ib2) reconnecting [462082.776344] Lustre: fir-MDT0000: Connection restored to f9de01a6-d260-4 (at 10.50.8.54@o2ib2) [462308.980568] Lustre: fir-MDT0001: Connection restored to b8ff2e89-6825-4 (at 10.50.13.2@o2ib2) [462333.983107] Lustre: fir-MDT0000: Connection restored to b8ff2e89-6825-4 (at 10.50.13.2@o2ib2) [462495.240477] Lustre: fir-MDT0001: haven't heard from client 3e79a8aa-e20a-4 (at 10.50.12.16@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786bfc00, cur 1585014486 expire 1585014336 last 1585014259 [462495.260628] Lustre: Skipped 1 previous similar message [463685.092148] Lustre: fir-MDT0001: Connection restored to 3e79a8aa-e20a-4 (at 10.50.12.16@o2ib2) [463710.147706] Lustre: fir-MDT0000: Connection restored to 3e79a8aa-e20a-4 (at 10.50.12.16@o2ib2) [463925.691005] Lustre: fir-MDT0000: Client 4c25e1ac-3e03-4 (at 10.50.1.15@o2ib2) reconnecting [463925.699464] Lustre: fir-MDT0000: Connection restored to a52f1005-c8d6-4 (at 10.50.1.15@o2ib2) [463955.861747] Lustre: fir-MDT0000: Client 35751781-79b0-4 (at 10.50.8.40@o2ib2) reconnecting [463955.871988] Lustre: fir-MDT0000: Connection restored to 35751781-79b0-4 (at 10.50.8.40@o2ib2) [463962.280431] Lustre: fir-MDT0001: haven't heard from client 5d22cf6d-6c39-4 (at 10.49.29.7@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6f891800, cur 1585015953 expire 1585015803 last 1585015726 [463962.300453] Lustre: Skipped 1 previous similar message [464877.626458] Lustre: fir-MDT0001: Client ec565b6c-3dc6-4 (at 10.50.9.43@o2ib2) reconnecting [464877.635159] Lustre: fir-MDT0001: Connection restored to ec565b6c-3dc6-4 (at 10.50.9.43@o2ib2) [464877.643859] Lustre: Skipped 1 previous similar message [465039.147593] Lustre: fir-MDT0000: Client c754ebbc-d707-4 (at 10.50.8.45@o2ib2) reconnecting [465039.156020] Lustre: Skipped 1 previous similar message [465039.161285] Lustre: fir-MDT0000: Connection restored to c754ebbc-d707-4 (at 10.50.8.45@o2ib2) [465087.819224] Lustre: fir-MDT0000: Client dba6cbec-f37c-4 (at 10.50.10.44@o2ib2) reconnecting [465087.827794] Lustre: fir-MDT0000: Connection restored to dba6cbec-f37c-4 (at 10.50.10.44@o2ib2) [465134.896717] Lustre: fir-MDT0000: Client 38137f29-3a97-4 (at 10.50.1.16@o2ib2) reconnecting [465134.905428] Lustre: fir-MDT0000: Connection restored to 38137f29-3a97-4 (at 10.50.1.16@o2ib2) [465299.001694] Lustre: fir-MDT0001: Connection restored to 5d22cf6d-6c39-4 (at 10.49.29.7@o2ib1) [466514.044639] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [466514.053113] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [466514.061738] Lustre: Skipped 1 previous similar message [466558.286750] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [466558.295203] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [466882.828522] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [466882.837024] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [466926.916605] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [466926.925468] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [466948.362315] Lustre: fir-MDT0001: haven't heard from client b4f8cb5a-edfb-4 (at 10.50.13.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d70465800, cur 1585018939 expire 1585018789 last 1585018712 [466948.382443] Lustre: Skipped 1 previous similar message [466972.616910] Lustre: fir-MDT0001: Client 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) reconnecting [466972.625643] Lustre: fir-MDT0001: Connection restored to 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) [467017.559983] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [467017.568372] Lustre: Skipped 1 previous similar message [467051.628366] Lustre: fir-MDT0001: Client 76ce6a34-97f8-4 (at 10.50.5.57@o2ib2) reconnecting [467051.636808] Lustre: fir-MDT0001: Connection restored to 76ce6a34-97f8-4 (at 10.50.5.57@o2ib2) [467051.645455] Lustre: Skipped 2 previous similar messages [467086.971156] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [467175.804391] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [467175.812787] Lustre: Skipped 1 previous similar message [467219.910616] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [467219.919263] Lustre: Skipped 3 previous similar messages [467309.558876] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [467309.567323] Lustre: Skipped 4 previous similar messages [467542.993929] Lustre: fir-MDT0001: Connection restored to 64442903-03be-4 (at 10.50.0.61@o2ib2) [467543.002590] Lustre: Skipped 14 previous similar messages [467584.803981] Lustre: fir-MDT0001: Client 60f30658-9a15-4 (at 10.50.8.26@o2ib2) reconnecting [467584.812393] Lustre: Skipped 13 previous similar messages [468104.321041] Lustre: fir-MDT0001: Client 64442903-03be-4 (at 10.50.0.61@o2ib2) reconnecting [468104.329461] Lustre: Skipped 10 previous similar messages [468163.568325] Lustre: fir-MDT0001: Connection restored to b4f8cb5a-edfb-4 (at 10.50.13.3@o2ib2) [468163.576978] Lustre: Skipped 15 previous similar messages [468724.772324] Lustre: fir-MDT0000: Client c754ebbc-d707-4 (at 10.50.8.45@o2ib2) reconnecting [468724.780716] Lustre: Skipped 1 previous similar message [468762.410600] Lustre: fir-MDT0001: haven't heard from client 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd7742dc00, cur 1585020753 expire 1585020603 last 1585020526 [468762.430721] Lustre: Skipped 1 previous similar message [469487.697440] Lustre: fir-MDT0001: Client 76ce6a34-97f8-4 (at 10.50.5.57@o2ib2) reconnecting [469487.705895] Lustre: fir-MDT0001: Connection restored to 76ce6a34-97f8-4 (at 10.50.5.57@o2ib2) [469487.714566] Lustre: Skipped 3 previous similar messages [469832.440862] Lustre: fir-MDT0001: haven't heard from client 13489a82-b9be-4 (at 10.50.13.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73e27000, cur 1585021823 expire 1585021673 last 1585021596 [469832.460888] Lustre: Skipped 1 previous similar message [469873.300030] Lustre: fir-MDT0001: Connection restored to 93a83cb1-bf44-4 (at 10.50.7.51@o2ib2) [469873.308677] Lustre: Skipped 1 previous similar message [470779.465829] Lustre: fir-MDT0001: haven't heard from client 37a9513e-0a78-4 (at 10.50.12.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d562f9000, cur 1585022770 expire 1585022620 last 1585022543 [470779.485964] Lustre: Skipped 1 previous similar message [471004.302386] Lustre: fir-MDT0001: Connection restored to 13489a82-b9be-4 (at 10.50.13.1@o2ib2) [471004.311050] Lustre: Skipped 2 previous similar messages [471005.460837] Lustre: fir-MDT0000: Client dbf6b7b0-ca0f-4 (at 10.50.6.71@o2ib2) reconnecting [471005.469223] Lustre: Skipped 2 previous similar messages [471008.472147] Lustre: fir-MDT0001: haven't heard from client 8db8300c-7159-4 (at 10.50.14.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d12ac00, cur 1585022999 expire 1585022849 last 1585022772 [471008.492231] Lustre: Skipped 5 previous similar messages [471029.382702] Lustre: fir-MDT0000: Connection restored to 13489a82-b9be-4 (at 10.50.13.1@o2ib2) [471029.391360] Lustre: Skipped 1 previous similar message [471298.635202] Lustre: fir-MDT0000: Client 40019701-810a-4 (at 10.50.1.26@o2ib2) reconnecting [471298.643656] Lustre: fir-MDT0000: Connection restored to 40019701-810a-4 (at 10.50.1.26@o2ib2) [471690.490982] Lustre: fir-MDT0001: haven't heard from client efbc6add-d7ba-4 (at 10.50.12.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b336c00, cur 1585023681 expire 1585023531 last 1585023454 [471690.510995] Lustre: Skipped 1 previous similar message [471854.432406] Lustre: fir-MDT0000: Client f3d4dc47-07cf-4 (at 10.50.10.42@o2ib2) reconnecting [471854.441256] Lustre: fir-MDT0000: Connection restored to f3d4dc47-07cf-4 (at 10.50.10.42@o2ib2) [472086.931365] Lustre: fir-MDT0001: Connection restored to 91cab27d-d429-4 (at 10.49.29.2@o2ib1) [472086.940017] Lustre: Skipped 2 previous similar messages [472124.502898] Lustre: fir-MDT0001: haven't heard from client 90f2fa35-5a19-4 (at 10.49.29.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d128400, cur 1585024115 expire 1585023965 last 1585023888 [472124.522928] Lustre: Skipped 1 previous similar message [472964.532908] Lustre: fir-MDT0001: Client 22fe86ac-45a8-4 (at 10.50.17.19@o2ib2) reconnecting [472964.541375] Lustre: Skipped 1 previous similar message [472964.546664] Lustre: fir-MDT0001: Connection restored to 22fe86ac-45a8-4 (at 10.50.17.19@o2ib2) [472964.555408] Lustre: Skipped 6 previous similar messages [473082.602329] Lustre: fir-MDT0001: Client 632a5ec1-0567-4 (at 10.50.17.12@o2ib2) reconnecting [473082.610779] Lustre: Skipped 12 previous similar messages [473486.132093] Lustre: fir-MDT0001: Client 124e16c3-5aff-4 (at 10.50.17.4@o2ib2) reconnecting [473486.140479] Lustre: Skipped 1 previous similar message [473582.132755] Lustre: fir-MDT0001: Connection restored to 59f3e55a-b3e8-4 (at 10.50.17.9@o2ib2) [473582.141443] Lustre: Skipped 24 previous similar messages [474301.734019] Lustre: fir-MDT0001: Connection restored to efbc6add-d7ba-4 (at 10.50.12.4@o2ib2) [474301.742646] Lustre: Skipped 2 previous similar messages [474922.696623] Lustre: fir-MDT0001: Client 72c268c0-e9ae-4 (at 10.50.17.20@o2ib2) reconnecting [474922.698476] Lustre: fir-MDT0001: Connection restored to 03a67404-05e1-4 (at 10.50.17.30@o2ib2) [474922.698479] Lustre: Skipped 1 previous similar message [474922.718999] Lustre: Skipped 14 previous similar messages [474959.699397] Lustre: fir-MDT0001: Client 632a5ec1-0567-4 (at 10.50.17.12@o2ib2) reconnecting [474959.707846] Lustre: Skipped 3 previous similar messages [475158.703557] Lustre: fir-MDT0001: Client 76472dfd-6d79-4 (at 10.50.17.13@o2ib2) reconnecting [475158.712041] Lustre: Skipped 5 previous similar messages [475246.182103] Lustre: fir-MDT0001: Client 3ebd75af-84f9-4 (at 10.50.7.36@o2ib2) reconnecting [475246.190478] Lustre: Skipped 2 previous similar messages [475680.601646] Lustre: fir-MDT0001: haven't heard from client 17ee7eca-4216-4 (at 10.50.13.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d12bc00, cur 1585027671 expire 1585027521 last 1585027444 [475680.621640] Lustre: Skipped 1 previous similar message [475691.601830] Lustre: fir-MDT0000: haven't heard from client 17ee7eca-4216-4 (at 10.50.13.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5b914800, cur 1585027682 expire 1585027532 last 1585027455 [476964.912945] Lustre: fir-MDT0001: Connection restored to 17ee7eca-4216-4 (at 10.50.13.5@o2ib2) [476964.921567] Lustre: Skipped 17 previous similar messages [477120.641417] Lustre: fir-MDT0001: haven't heard from client f7fe261e-a413-4 (at 10.49.28.2@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6dc3cc00, cur 1585029111 expire 1585028961 last 1585028884 [477229.344082] Lustre: fir-MDT0001: Connection restored to d2f2b8c6-0661-4 (at 10.50.16.5@o2ib2) [477229.352744] Lustre: Skipped 1 previous similar message [477792.881891] LNet: Service thread pid 22482 was inactive for 200.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [477792.898937] Pid: 22482, comm: mdt00_064 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [477792.909204] Call Trace: [477792.911766] [] call_rwsem_down_write_failed+0x17/0x30 [477792.918627] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [477792.925663] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [477792.933230] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [477792.940486] [] llog_process_thread+0x82f/0x18e0 [obdclass] [477792.947779] [] llog_process_or_fork+0xbc/0x450 [obdclass] [477792.955045] [] llog_cat_process_cb+0x239/0x250 [obdclass] [477792.962263] [] llog_process_thread+0x82f/0x18e0 [obdclass] [477792.969569] [] llog_process_or_fork+0xbc/0x450 [obdclass] [477792.976789] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [477792.984460] [] llog_cat_process+0x2e/0x30 [obdclass] [477792.991227] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [477792.998784] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [477793.005961] [] mdd_changelog_clear+0x503/0x690 [mdd] [477793.012736] [] mdd_iocontrol+0x163/0x540 [mdd] [477793.018978] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [477793.025219] [] mdt_set_info+0x484/0x490 [mdt] [477793.031393] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [477793.038480] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [477793.046355] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [477793.052806] [] kthread+0xd1/0xe0 [477793.057834] [] ret_from_fork_nospec_begin+0xe/0x21 [477793.064428] [] 0xffffffffffffffff [477793.069554] LustreError: dumping log to /tmp/lustre-log.1585029783.22482 [477930.377747] LNet: Service thread pid 22482 completed after 337.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [481067.975700] Lustre: fir-MDT0001: Connection restored to 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) [481067.984334] Lustre: Skipped 1 previous similar message [481093.065834] Lustre: fir-MDT0000: Connection restored to 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) [481787.768450] Lustre: fir-MDT0001: haven't heard from client c4422e40-3cfa-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a7c674000, cur 1585033778 expire 1585033628 last 1585033551 [481787.788458] Lustre: Skipped 3 previous similar messages [481807.041554] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [481832.040106] Lustre: fir-MDT0000: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [484890.853789] Lustre: fir-MDT0001: haven't heard from client 6e31591f-913a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae792246c00, cur 1585036881 expire 1585036731 last 1585036654 [484890.873768] Lustre: Skipped 1 previous similar message [484945.778388] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [484970.873646] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [486538.951678] Lustre: fir-MDT0001: Connection restored to 33701c5f-e220-4 (at 10.50.13.12@o2ib2) [486564.040384] Lustre: fir-MDT0000: Connection restored to 33701c5f-e220-4 (at 10.50.13.12@o2ib2) [486577.898856] Lustre: fir-MDT0001: haven't heard from client bd12e5c8-ece9-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae781accc00, cur 1585038568 expire 1585038418 last 1585038341 [486577.918855] Lustre: Skipped 1 previous similar message [486800.121087] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [486825.172664] Lustre: fir-MDT0000: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [491801.045103] Lustre: fir-MDT0001: haven't heard from client 0fe03f2f-af46-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af71beca000, cur 1585043791 expire 1585043641 last 1585043564 [491801.065134] Lustre: Skipped 1 previous similar message [491830.826192] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [491855.863064] Lustre: fir-MDT0000: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [503276.731877] Lustre: fir-MDT0001: Client 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) reconnecting [503276.741107] Lustre: fir-MDT0001: Connection restored to 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) [503849.376429] Lustre: fir-MDT0001: haven't heard from client ee56fbe2-040d-4 (at 10.49.25.17@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b694c00, cur 1585055839 expire 1585055689 last 1585055612 [503849.396576] Lustre: Skipped 1 previous similar message [504439.059345] Lustre: fir-MDT0001: Client 26a06b5e-ae7c-4 (at 10.50.9.2@o2ib2) reconnecting [504439.067660] Lustre: fir-MDT0001: Connection restored to 26a06b5e-ae7c-4 (at 10.50.9.2@o2ib2) [504503.046965] Lustre: fir-MDT0001: Client 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) reconnecting [504503.055326] Lustre: fir-MDT0001: Connection restored to 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) [505330.415971] Lustre: fir-MDT0001: haven't heard from client d8c1b053-4ffb-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad46e770800, cur 1585057320 expire 1585057170 last 1585057093 [505330.435963] Lustre: Skipped 1 previous similar message [505380.200329] LNet: Service thread pid 22333 was inactive for 202.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [505380.217389] Pid: 22333, comm: mdt00_007 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [505380.227717] Call Trace: [505380.230295] [] call_rwsem_down_write_failed+0x17/0x30 [505380.237157] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [505380.244202] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [505380.251798] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [505380.259015] [] llog_process_thread+0x82f/0x18e0 [obdclass] [505380.266349] [] llog_process_or_fork+0xbc/0x450 [obdclass] [505380.273655] [] llog_cat_process_cb+0x239/0x250 [obdclass] [505380.280892] [] llog_process_thread+0x82f/0x18e0 [obdclass] [505380.288267] [] llog_process_or_fork+0xbc/0x450 [obdclass] [505380.295561] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [505380.303255] [] llog_cat_process+0x2e/0x30 [obdclass] [505380.310042] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [505380.317614] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [505380.324869] [] mdd_changelog_clear+0x503/0x690 [mdd] [505380.331685] [] mdd_iocontrol+0x163/0x540 [mdd] [505380.337983] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [505380.344261] [] mdt_set_info+0x484/0x490 [mdt] [505380.350502] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [505380.357662] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [505380.365521] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [505380.372064] [] kthread+0xd1/0xe0 [505380.377135] [] ret_from_fork_nospec_begin+0xe/0x21 [505380.383734] [] 0xffffffffffffffff [505380.388890] LustreError: dumping log to /tmp/lustre-log.1585057369.22333 [505389.855758] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [505411.291117] LNet: Service thread pid 22333 completed after 233.43s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [505414.982485] Lustre: fir-MDT0000: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [508436.436328] Lustre: fir-MDT0001: Client 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) reconnecting [508436.444779] Lustre: fir-MDT0001: Connection restored to 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) [509158.666700] Lustre: fir-MDT0001: Client 26a06b5e-ae7c-4 (at 10.50.9.2@o2ib2) reconnecting [509158.675087] Lustre: fir-MDT0001: Connection restored to 26a06b5e-ae7c-4 (at 10.50.9.2@o2ib2) [509274.989460] Lustre: fir-MDT0001: Client b8323df2-a6a7-4 (at 10.50.9.3@o2ib2) reconnecting [509274.997872] Lustre: fir-MDT0001: Connection restored to b8323df2-a6a7-4 (at 10.50.9.3@o2ib2) [510616.123921] Lustre: fir-MDT0001: Client bf9c0359-a266-4 (at 10.50.0.1@o2ib2) reconnecting [510616.132270] Lustre: fir-MDT0001: Connection restored to bf9c0359-a266-4 (at 10.50.0.1@o2ib2) [510625.042048] Lustre: fir-MDT0001: Client 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) reconnecting [510625.050419] Lustre: fir-MDT0001: Connection restored to 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) [513037.665110] Lustre: 22427:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [513037.676659] Lustre: 22427:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 45 previous similar messages [513114.266680] Lustre: 28226:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [513284.726648] Lustre: 68051:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [513284.738186] Lustre: 68051:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 4 previous similar messages [513369.638554] Lustre: fir-MDT0000: haven't heard from client 89912c83-1f62-4 (at 10.50.14.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed0e6ca400, cur 1585065359 expire 1585065209 last 1585065132 [513369.658595] Lustre: Skipped 1 previous similar message [513612.583533] Lustre: 22493:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [513612.595093] Lustre: 22493:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 7 previous similar messages [514226.701756] Lustre: 22473:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [514226.713301] Lustre: 22473:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 16 previous similar messages [514849.814725] Lustre: 22412:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [514849.826279] Lustre: 22412:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 16 previous similar messages [515288.192017] Lustre: fir-MDT0001: Connection restored to 89912c83-1f62-4 (at 10.50.14.5@o2ib2) [515313.199167] Lustre: fir-MDT0000: Connection restored to 89912c83-1f62-4 (at 10.50.14.5@o2ib2) [515456.852644] Lustre: 21888:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585067439/real 1585067439] req@ffff8aea027da400 x1661550917706880/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1585067446 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [515456.880775] Lustre: 21888:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [515456.890763] LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail [515456.904721] LustreError: Skipped 2 previous similar messages [515462.479847] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds [515462.490032] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 3 previous similar messages [515462.499478] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [515465.829004] Lustre: 28970:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [515465.840523] Lustre: 28970:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 16 previous similar messages [515483.532455] Lustre: Failing over fir-MDT0000 [515483.675790] Lustre: fir-MDT0000: Not available for connect from 10.50.14.13@o2ib2 (stopping) [515483.952139] LustreError: 55111:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.109@o2ib7 arrived at 1585067473 with bad export cookie 13699018700024891964 [515483.952531] LustreError: 51249:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffff8af69ae4ad00/0xbe1cb12709f79dea lrc: 3/0,0 mode: CR/CR res: [0x200000006:0x20000:0x4fe1d].0x0 rrc: 62 type: PLN flags: 0x40000000000000 nid: 10.0.10.109@o2ib7 remote: 0xf3793efb0ea6f57b expref: 236 pid: 21996 timeout: 0 lvb_type: 2 [515483.952539] LustreError: 51249:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) Skipped 1 previous similar message [515484.008846] LustreError: 55111:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 21 previous similar messages [515484.209377] Lustre: fir-MDT0000: Not available for connect from 10.50.7.53@o2ib2 (stopping) [515484.217860] Lustre: Skipped 34 previous similar messages [515484.473289] LustreError: 82944:0:(osp_object.c:594:osp_attr_get()) fir-MDT0001-osp-MDT0000:osp_attr_get update error [0x20000000a:0x1:0x0]: rc = -108 [515484.541547] LustreError: 82944:0:(llog_cat.c:444:llog_cat_close()) fir-MDT0001-osp-MDT0000: failure destroying log during cleanup: rc = -108 [515484.830012] LustreError: 11-0: fir-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [515484.839880] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [515485.257754] Lustre: fir-MDT0000: Not available for connect from 10.50.17.10@o2ib2 (stopping) [515485.266289] Lustre: Skipped 135 previous similar messages [515487.281218] Lustre: fir-MDT0000: Not available for connect from 10.50.8.57@o2ib2 (stopping) [515487.289709] Lustre: Skipped 124 previous similar messages [515488.066102] LustreError: 55112:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.108@o2ib7 arrived at 1585067477 with bad export cookie 13699018700024394719 [515488.067178] LustreError: 21972:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffff8b0acf71d7c0/0xbe1cb126c71a0450 lrc: 3/0,0 mode: CR/CR res: [0x200000006:0x2020000:0x4d919].0x0 rrc: 60 type: PLN flags: 0x40000000000000 nid: 10.0.10.108@o2ib7 remote: 0xfb673a99cd801fef expref: 1580 pid: 22525 timeout: 0 lvb_type: 2 [515488.067181] LustreError: 21972:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) Skipped 22 previous similar messages [515488.123423] LustreError: 55112:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 42 previous similar messages [515489.738947] LustreError: 51361:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.102@o2ib7 arrived at 1585067479 with bad export cookie 13699018700024308556 [515489.739032] LustreError: 52773:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) ### ### ns: mdt-fir-MDT0000_UUID lock: ffff8acd85dba640/0xbe1cb12694d4880f lrc: 3/0,0 mode: CR/CR res: [0x200000006:0x2020000:0x50d78].0x0 rrc: 55 type: PLN flags: 0x40000000000000 nid: 10.0.10.102@o2ib7 remote: 0xd896fb54a5490072 expref: 1584 pid: 21992 timeout: 0 lvb_type: 2 [515489.739035] LustreError: 52773:0:(ldlm_lock.c:2713:ldlm_lock_dump_handle()) Skipped 20 previous similar messages [515489.796673] LustreError: 51361:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 21 previous similar messages [515491.307351] Lustre: fir-MDT0000: Not available for connect from 10.50.6.49@o2ib2 (stopping) [515491.315822] Lustre: Skipped 170 previous similar messages [515498.398539] LustreError: 22557:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.104@o2ib7 arrived at 1585067487 with bad export cookie 13699018700024435606 [515498.414363] LustreError: 22557:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 20 previous similar messages [515499.319070] Lustre: fir-MDT0000: Not available for connect from 10.50.1.31@o2ib2 (stopping) [515499.327581] Lustre: Skipped 306 previous similar messages [515505.044170] LustreError: 51250:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.0.10.113@o2ib7 arrived at 1585067494 with bad export cookie 13699018700025194441 [515505.059994] LustreError: 51250:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 82 previous similar messages [515507.071112] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [515515.628799] Lustre: fir-MDT0000: Not available for connect from 10.50.10.59@o2ib2 (stopping) [515515.637329] Lustre: Skipped 506 previous similar messages [515546.776241] Lustre: server umount fir-MDT0000 complete [515579.787370] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.35@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [515579.804917] LustreError: Skipped 2 previous similar messages [515580.655799] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.24.36@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [515580.673314] LustreError: Skipped 1 previous similar message [515582.040490] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.1.43@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [515582.057923] LustreError: Skipped 7 previous similar messages [515584.060804] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.9@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [515584.078198] LustreError: Skipped 27 previous similar messages [515588.062932] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.30.3@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [515588.080401] LustreError: Skipped 183 previous similar messages [515596.099499] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.3.8@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [515596.116886] LustreError: Skipped 361 previous similar messages [515612.207421] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.7.56@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [515612.224834] LustreError: Skipped 752 previous similar messages [515661.302991] Lustre: fir-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [515680.141946] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.22.35@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [515680.159463] LustreError: Skipped 42 previous similar messages [515690.114835] LNet: Service thread pid 53729 was inactive for 200.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [515690.131883] Pid: 53729, comm: mdt01_084 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [515690.142179] Call Trace: [515690.144735] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [515690.151431] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [515690.158244] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [515690.165036] [] osp_md_object_lock+0x162/0x2d0 [osp] [515690.171727] [] lod_object_lock+0xf4/0x780 [lod] [515690.178072] [] mdd_object_lock+0x3e/0xe0 [mdd] [515690.184335] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [515690.191715] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [515690.198584] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [515690.204900] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [515690.211491] [] mdt_reint_rec+0x83/0x210 [mdt] [515690.217664] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [515690.224327] [] mdt_reint+0x67/0x140 [mdt] [515690.230147] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [515690.237206] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [515690.245042] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [515690.251516] [] kthread+0xd1/0xe0 [515690.256531] [] ret_from_fork_nospec_begin+0xe/0x21 [515690.263136] [] 0xffffffffffffffff [515690.268258] LustreError: dumping log to /tmp/lustre-log.1585067679.53729 [515698.307080] LNet: Service thread pid 23099 was inactive for 200.14s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [515698.324155] Pid: 23099, comm: mdt01_017 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [515698.334463] Call Trace: [515698.337055] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [515698.343777] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [515698.350586] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [515698.357423] [] osp_md_object_lock+0x162/0x2d0 [osp] [515698.364086] [] lod_object_lock+0xf4/0x780 [lod] [515698.370425] [] mdd_object_lock+0x3e/0xe0 [mdd] [515698.376672] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [515698.384064] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [515698.390932] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [515698.397274] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [515698.403870] [] mdt_reint_rec+0x83/0x210 [mdt] [515698.410060] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [515698.416725] [] mdt_reint+0x67/0x140 [mdt] [515698.422565] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [515698.429656] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [515698.437498] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [515698.443973] [] kthread+0xd1/0xe0 [515698.448999] [] ret_from_fork_nospec_begin+0xe/0x21 [515698.455596] [] 0xffffffffffffffff [515698.460777] LustreError: dumping log to /tmp/lustre-log.1585067687.23099 [515725.443781] LNet: Service thread pid 68053 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [515725.460810] Pid: 68053, comm: mdt00_102 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [515725.471088] Call Trace: [515725.473652] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [515725.480385] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [515725.487187] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [515725.494007] [] osp_md_object_lock+0x162/0x2d0 [osp] [515725.500699] [] lod_object_lock+0xf4/0x780 [lod] [515725.507067] [] mdd_object_lock+0x3e/0xe0 [mdd] [515725.513321] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [515725.520695] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [515725.527587] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [515725.533914] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [515725.540496] [] mdt_reint_rec+0x83/0x210 [mdt] [515725.546648] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [515725.553326] [] mdt_reint+0x67/0x140 [mdt] [515725.559131] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [515725.566214] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [515725.574060] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [515725.580527] [] kthread+0xd1/0xe0 [515725.585556] [] ret_from_fork_nospec_begin+0xe/0x21 [515725.592150] [] 0xffffffffffffffff [515725.597278] LustreError: dumping log to /tmp/lustre-log.1585067714.68053 [515731.075926] LNet: Service thread pid 78088 was inactive for 200.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [515731.092970] Pid: 78088, comm: mdt03_096 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [515731.103265] Call Trace: [515731.105831] [] ptlrpc_set_wait+0x480/0x790 [ptlrpc] [515731.112559] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [515731.119344] [] ldlm_cli_enqueue+0x3d2/0x920 [ptlrpc] [515731.126162] [] osp_md_object_lock+0x162/0x2d0 [osp] [515731.132845] [] lod_object_lock+0xf4/0x780 [lod] [515731.139190] [] mdd_object_lock+0x3e/0xe0 [mdd] [515731.145431] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [515731.152795] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [515731.159646] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [515731.165974] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [515731.172558] [] mdt_reint_rec+0x83/0x210 [mdt] [515731.178721] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [515731.185413] [] mdt_reint+0x67/0x140 [mdt] [515731.191260] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [515731.198352] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [515731.206220] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [515731.212669] [] kthread+0xd1/0xe0 [515731.217699] [] ret_from_fork_nospec_begin+0xe/0x21 [515731.224275] [] 0xffffffffffffffff [515731.229405] LustreError: dumping log to /tmp/lustre-log.1585067720.78088 [515732.868197] LustreError: 167-0: fir-MDT0000-lwp-MDT0001: This client was evicted by fir-MDT0000; in progress operations using this service will fail. [515732.891457] Lustre: fir-MDT0000-lwp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [515733.355109] LustreError: 11-0: fir-MDT0000-lwp-MDT0001: operation quota_acquire to node 10.0.10.51@o2ib7 failed: rc = -11 [515733.366160] LustreError: Skipped 1 previous similar message [515753.005429] Lustre: fir-MDT0000-osp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [515753.048891] LNet: Service thread pid 53729 completed after 263.13s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [515753.065206] LNet: Skipped 3 previous similar messages [515757.956966] Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0xae6d2c631bd3f37c to 0x764984320c7ad780 [515830.011690] Lustre: fir-MDT0001: Connection restored to f7fe261e-a413-4 (at 10.49.28.2@o2ib1) [515830.020322] Lustre: Skipped 1 previous similar message [516025.710231] Lustre: fir-MDT0001: haven't heard from client 08262875-245e-4 (at 10.50.15.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6dc3c000, cur 1585068015 expire 1585067865 last 1585067788 [516025.730368] Lustre: Skipped 1 previous similar message [516076.791318] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [516076.802831] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 12 previous similar messages [516681.004017] Lustre: 30979:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [516681.015534] Lustre: 30979:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 19 previous similar messages [517234.942042] Lustre: fir-MDT0001: Connection restored to 08262875-245e-4 (at 10.50.15.13@o2ib2) [517234.950751] Lustre: Skipped 2 previous similar messages [517305.649334] Lustre: 67829:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [517305.660861] Lustre: 67829:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 18 previous similar messages [517905.681803] Lustre: 22463:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [517905.693282] Lustre: 22463:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 570 previous similar messages [518463.370228] Lustre: fir-MDT0001: Connection restored to ce67c0db-bb4c-4 (at 10.50.15.5@o2ib2) [518505.985307] Lustre: 22463:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [518505.996796] Lustre: 22463:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1228 previous similar messages [518507.972765] Lustre: fir-MDT0001: Connection restored to 6f426f61-5639-4 (at 10.50.13.10@o2ib2) [519106.125460] Lustre: 22462:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [519106.136941] Lustre: 22462:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1404 previous similar messages [519706.164369] Lustre: 62242:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [519706.175849] Lustre: 62242:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1939 previous similar messages [520310.219568] Lustre: 22479:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [520310.231060] Lustre: 22479:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2208 previous similar messages [520325.830298] Lustre: fir-MDT0001: haven't heard from client c4e4a9fa-0416-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afce77dfc00, cur 1585072315 expire 1585072165 last 1585072088 [520386.659492] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [520910.290645] Lustre: 68053:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [520910.302137] Lustre: 68053:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1044 previous similar messages [521441.862047] Lustre: fir-MDT0001: haven't heard from client 82c5fb51-6200-4 (at 10.50.16.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d67ad0c00, cur 1585073431 expire 1585073281 last 1585073204 [521691.387993] Lustre: 22421:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [521691.399481] Lustre: 22421:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1465 previous similar messages [522291.430040] Lustre: 83152:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [522291.441519] Lustre: 83152:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2094 previous similar messages [522892.234788] Lustre: 22417:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [522892.246273] Lustre: 22417:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1700 previous similar messages [523184.302187] Lustre: fir-MDT0001: Connection restored to 82c5fb51-6200-4 (at 10.50.16.9@o2ib2) [523492.330616] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [523492.342102] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1580 previous similar messages [524092.452121] Lustre: 22421:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [524092.463604] Lustre: 22421:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2784 previous similar messages [524692.554865] Lustre: 22418:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [524692.566344] Lustre: 22418:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2618 previous similar messages [524827.956194] Lustre: fir-MDT0001: haven't heard from client d4aafd71-b333-4 (at 10.50.14.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd77526800, cur 1585076817 expire 1585076667 last 1585076590 [525292.579000] Lustre: 22490:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [525292.590502] Lustre: 22490:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2372 previous similar messages [525895.143716] Lustre: 22333:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [525895.155195] Lustre: 22333:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2167 previous similar messages [525999.263183] Lustre: fir-MDT0001: Connection restored to d4aafd71-b333-4 (at 10.50.14.15@o2ib2) [526495.475660] Lustre: 22399:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [526495.487144] Lustre: 22399:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2151 previous similar messages [527104.173488] Lustre: 22449:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [527104.184966] Lustre: 22449:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2310 previous similar messages [527704.204375] Lustre: 22359:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [527704.215857] Lustre: 22359:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2221 previous similar messages [528304.466219] Lustre: 22469:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [528304.477695] Lustre: 22469:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1955 previous similar messages [528904.546502] Lustre: 83153:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [528904.557979] Lustre: 83153:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1729 previous similar messages [529504.713662] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [529504.725140] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 332 previous similar messages [530105.049416] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [530105.060895] Lustre: 21982:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 2029 previous similar messages [530652.121555] Lustre: fir-MDT0001: haven't heard from client abed66c3-995c-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d02fc0400, cur 1585082641 expire 1585082491 last 1585082414 [530681.771575] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [530705.853301] Lustre: 68053:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [530705.864782] Lustre: 68053:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 457 previous similar messages [530899.130632] Lustre: fir-MDT0001: haven't heard from client 9f02a5e3-9696-4 (at 10.50.13.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d71d36000, cur 1585082888 expire 1585082738 last 1585082661 [531224.631478] LNet: Service thread pid 22359 was inactive for 200.42s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [531224.648501] Pid: 22359, comm: mdt00_012 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [531224.658761] Call Trace: [531224.661319] [] __wait_on_buffer+0x2a/0x30 [531224.667103] [] ldiskfs_bread+0x7c/0xc0 [ldiskfs] [531224.673515] [] osd_ldiskfs_read+0xf4/0x2d0 [osd_ldiskfs] [531224.680605] [] osd_read+0x95/0xc0 [osd_ldiskfs] [531224.686905] [] dt_read+0x1a/0x50 [obdclass] [531224.692911] [] llog_osd_next_block+0x36a/0xbc0 [obdclass] [531224.700106] [] llog_process_thread+0x330/0x18e0 [obdclass] [531224.707383] [] llog_process_or_fork+0xbc/0x450 [obdclass] [531224.714589] [] llog_cat_process_cb+0x239/0x250 [obdclass] [531224.721764] [] llog_process_thread+0x82f/0x18e0 [obdclass] [531224.729026] [] llog_process_or_fork+0xbc/0x450 [obdclass] [531224.736204] [] llog_cat_process_or_fork+0x17e/0x360 [obdclass] [531224.743811] [] llog_cat_process+0x2e/0x30 [obdclass] [531224.750588] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [531224.758106] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [531224.765290] [] mdd_changelog_clear+0x503/0x690 [mdd] [531224.772024] [] mdd_iocontrol+0x163/0x540 [mdd] [531224.778266] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [531224.784513] [] mdt_set_info+0x484/0x490 [mdt] [531224.790647] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [531224.797711] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [531224.805511] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [531224.811941] [] kthread+0xd1/0xe0 [531224.816956] [] ret_from_fork_nospec_begin+0xe/0x21 [531224.823523] [] 0xffffffffffffffff [531224.828628] LustreError: dumping log to /tmp/lustre-log.1585083213.22359 [531619.686750] Lustre: 22490:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8add3be00d80 x1661846488675264/t0(0) o46->3dab8abe-e790-3878-3898-4444ee422524@10.0.10.3@o2ib7:373/0 lens 264/224 e 24 to 0 dl 1585083613 ref 2 fl Interpret:/0/0 rc 0/0 [531625.266122] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [531625.276241] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [532226.360046] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [532226.370162] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [532827.454146] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [532827.464261] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [533428.548355] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [533428.558473] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [533428.568823] Lustre: Skipped 1 previous similar message [534029.647365] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [534029.657490] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [534262.222886] Lustre: fir-MDT0001: haven't heard from client 72866633-325f-4 (at 10.50.15.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d67ad3800, cur 1585086251 expire 1585086101 last 1585086024 [534630.741461] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [534630.751572] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [535231.835390] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [535231.845503] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [535832.929286] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [535832.939400] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [535972.175333] Lustre: fir-MDT0001: Connection restored to 72866633-325f-4 (at 10.50.15.9@o2ib2) [536434.023214] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [536434.033324] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [537035.117227] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [537035.127348] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [537636.211129] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [537636.221271] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [538237.305881] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [538237.316000] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [538838.399691] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [538838.409809] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [538953.354886] Lustre: fir-MDT0001: haven't heard from client e9949bf5-ceeb-4 (at 10.50.12.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d562f8c00, cur 1585090942 expire 1585090792 last 1585090715 [539439.493394] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [539439.503515] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [540040.586981] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [540040.597103] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [540641.680677] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [540641.690798] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [540641.701177] Lustre: Skipped 1 previous similar message [541242.779187] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [541242.789297] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [541843.873886] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [541843.884016] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [542444.967510] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [542444.977644] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [543046.061418] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [543046.071545] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [543647.155482] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [543647.165659] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [544248.249743] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [544248.259869] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [544849.344219] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [544849.354338] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [545450.438825] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [545450.448946] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [545450.459307] Lustre: Skipped 1 previous similar message [546051.538062] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [546051.548176] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [546652.632009] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [546652.642142] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [547253.725674] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [547253.735795] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [547854.819424] Lustre: fir-MDT0001: Client 3dab8abe-e790-3878-3898-4444ee422524 (at 10.0.10.3@o2ib7) reconnecting [547854.829533] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [548402.278451] LNet: Service thread pid 22359 completed after 17377.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [560355.959459] Lustre: fir-MDT0001: haven't heard from client d95d81db-621a-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0aa221bc00, cur 1585112344 expire 1585112194 last 1585112117 [560403.488912] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [561011.490682] Lustre: fir-MDT0001: Connection restored to 88c27327-e888-4 (at 10.49.27.21@o2ib1) [570291.240298] Lustre: fir-MDT0001: haven't heard from client 9cf8861f-40bc-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd440f7000, cur 1585122279 expire 1585122129 last 1585122052 [570337.739885] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [572573.306297] Lustre: fir-MDT0001: haven't heard from client 140a8048-bada-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec82f7dc00, cur 1585124561 expire 1585124411 last 1585124334 [572651.652056] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [574159.351448] Lustre: fir-MDT0001: haven't heard from client 69ede8ab-5c22-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acda61af800, cur 1585126147 expire 1585125997 last 1585125920 [574198.837120] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [580824.540582] Lustre: fir-MDT0001: haven't heard from client 53fb438a-2a45-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6f59b800, cur 1585132812 expire 1585132662 last 1585132585 [580880.066216] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [584168.638871] Lustre: fir-MDT0001: haven't heard from client 568515af-b94a-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af67502cc00, cur 1585136156 expire 1585136006 last 1585135929 [584219.706034] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [587080.724904] Lustre: fir-MDT0001: haven't heard from client 51e206d5-3327-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b04e47a4000, cur 1585139068 expire 1585138918 last 1585138841 [587151.275104] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [590615.824494] Lustre: fir-MDT0001: haven't heard from client e685b13a-a4f9-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed11e2e000, cur 1585142603 expire 1585142453 last 1585142376 [590665.583285] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [591364.850637] Lustre: fir-MDT0001: haven't heard from client f32d3637-39fe-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a86dc800, cur 1585143352 expire 1585143202 last 1585143125 [591406.339626] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [596878.005116] Lustre: fir-MDT0001: haven't heard from client dadffa87-4553-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd2bdedc00, cur 1585148865 expire 1585148715 last 1585148638 [596931.964087] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [598615.053188] Lustre: fir-MDT0001: haven't heard from client 2d5a47ed-4b4c-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae953399000, cur 1585150602 expire 1585150452 last 1585150375 [598664.034906] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [602078.131777] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [602078.143519] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2086 previous similar messages [602080.185060] Lustre: 22443:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [602080.196793] Lustre: 22443:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 141 previous similar messages [602084.389310] Lustre: 22333:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [602084.401053] Lustre: 22333:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 270 previous similar messages [602092.420432] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [602092.432174] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 550 previous similar messages [602108.443072] Lustre: 83152:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [602108.454811] Lustre: 83152:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1066 previous similar messages [607028.606820] Lustre: fir-MDT0001: Connection restored to 51620f49-4677-4 (at 10.50.15.2@o2ib2) [609445.160564] Lustre: fir-MDT0001: Client c8932097-dc54-4 (at 10.50.10.41@o2ib2) reconnecting [609445.169032] Lustre: fir-MDT0001: Connection restored to c8932097-dc54-4 (at 10.50.10.41@o2ib2) [618133.600035] Lustre: fir-MDT0001: haven't heard from client d7618323-c1e8-4 (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed795bf000, cur 1585170120 expire 1585169970 last 1585169893 [620039.588175] Lustre: fir-MDT0001: Connection restored to 430e4894-d38d-4 (at 10.50.14.11@o2ib2) [636179.097772] Lustre: fir-MDT0001: haven't heard from client 68edeab9-3169-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aee7d2c3400, cur 1585188165 expire 1585188015 last 1585187938 [636222.568046] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [638030.148772] Lustre: fir-MDT0001: haven't heard from client 1015a07c-5c4c-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed0e752400, cur 1585190016 expire 1585189866 last 1585189789 [638091.658064] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [639925.203368] Lustre: fir-MDT0001: haven't heard from client af897907-9595-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd70913000, cur 1585191911 expire 1585191761 last 1585191684 [639964.940326] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [689021.618806] Lustre: fir-MDT0001: haven't heard from client 8795bae7-a3eb-4 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add0c700400, cur 1585241006 expire 1585240856 last 1585240779 [690968.214055] Lustre: fir-MDT0001: Connection restored to 38135a83-ec5c-4 (at 10.50.12.5@o2ib2) [706546.909542] Lustre: fir-MDT0001: Connection restored to 6dcf0e67-1d32-4 (at 10.50.7.9@o2ib2) [706553.121430] Lustre: fir-MDT0001: haven't heard from client 6dcf0e67-1d32-4 (at 10.50.7.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d65424400, cur 1585258537 expire 1585258387 last 1585258310 [707714.290555] Lustre: fir-MDT0001: Connection restored to 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) [770955.446252] Lustre: fir-MDT0001: Connection restored to cb22fe63-8523-4 (at 10.49.19.4@o2ib1) [771792.012150] Lustre: fir-MDT0001: haven't heard from client 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add7961a400, cur 1585323774 expire 1585323624 last 1585323547 [773037.354354] Lustre: fir-MDT0001: Connection restored to cb22fe63-8523-4 (at 10.49.19.4@o2ib1) [773069.126968] Lustre: fir-MDT0001: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [773237.673271] Lustre: fir-MDT0001: Connection restored to 6eb34c30-3804-4 (at 10.49.19.6@o2ib1) [773680.402763] Lustre: fir-MDT0001: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [774828.304140] Lustre: fir-MDT0001: Connection restored to 6f68bfc7-8d0d-4 (at 10.49.19.1@o2ib1) [774828.312755] Lustre: Skipped 1 previous similar message [774856.065276] Lustre: fir-MDT0001: Connection restored to 565e39cd-4898-4 (at 10.49.19.8@o2ib1) [774892.138668] Lustre: fir-MDT0001: Connection restored to 0b2cd2d4-168b-4 (at 10.49.19.7@o2ib1) [774899.483853] Lustre: fir-MDT0001: Connection restored to 9939e999-d857-4 (at 10.49.19.5@o2ib1) [775842.755970] Lustre: fir-MDT0001: Connection restored to 6eb34c30-3804-4 (at 10.49.19.6@o2ib1) [775872.955265] Lustre: fir-MDT0001: Connection restored to 3ab8a3cd-4496-4 (at 10.49.19.3@o2ib1) [775872.963878] Lustre: Skipped 1 previous similar message [775892.374055] Lustre: fir-MDT0001: Connection restored to 6f68bfc7-8d0d-4 (at 10.49.19.1@o2ib1) [786535.240619] Lustre: fir-MDT0001: Connection restored to 0b2cd2d4-168b-4 (at 10.49.19.7@o2ib1) [786542.349664] Lustre: fir-MDT0001: Connection restored to 9939e999-d857-4 (at 10.49.19.5@o2ib1) [794442.619555] Lustre: fir-MDT0001: haven't heard from client 661e22b8-e9f5-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5b916800, cur 1585346424 expire 1585346274 last 1585346197 [794442.639532] Lustre: Skipped 7 previous similar messages [794493.978690] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [797378.703621] Lustre: fir-MDT0001: haven't heard from client 144bfe18-3352-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed0bab8c00, cur 1585349360 expire 1585349210 last 1585349133 [797425.085289] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [798147.607681] Lustre: fir-MDT0001: Connection restored to f3e067b1-f2a3-4 (at 10.49.19.2@o2ib1) [820718.889867] Lustre: fir-MDT0001: Connection restored to 35cd2c2f-e02c-4 (at 10.50.12.13@o2ib2) [855760.355880] Lustre: fir-MDT0001: haven't heard from client a93d4107-2edc-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc1f67ac00, cur 1585407740 expire 1585407590 last 1585407513 [855860.213103] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [873374.845223] Lustre: fir-MDT0001: haven't heard from client 85b52166-baef-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8affec7c0800, cur 1585425354 expire 1585425204 last 1585425127 [873418.733295] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [883484.136579] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [883534.122597] Lustre: fir-MDT0001: haven't heard from client 780d3450-c327-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a86dc400, cur 1585435513 expire 1585435363 last 1585435286 [887922.237971] Lustre: fir-MDT0001: haven't heard from client add63273-c3b3-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acda61ac000, cur 1585439901 expire 1585439751 last 1585439674 [887962.894508] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [889123.268137] Lustre: fir-MDT0001: haven't heard from client 9525333d-e5b4-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd2bdef800, cur 1585441102 expire 1585440952 last 1585440875 [889159.558689] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [908551.812287] Lustre: fir-MDT0001: haven't heard from client ecf003cb-b7be-4 (at 10.50.16.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d75ff7800, cur 1585460530 expire 1585460380 last 1585460303 [910306.271084] Lustre: fir-MDT0001: Connection restored to ecf003cb-b7be-4 (at 10.50.16.8@o2ib2) [938185.213394] Lustre: fir-MDT0001: Client ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) reconnecting [938185.221700] Lustre: fir-MDT0001: Connection restored to ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) [938186.084244] LustreError: 21994:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8aec31aaf500 x1661974834042624/t0(0) o37->0b484496-0c22-4@10.50.5.1@o2ib2:20/0 lens 448/440 e 0 to 0 dl 1585490205 ref 1 fl Interpret:/0/0 rc 0/0 [938186.377877] LustreError: 21866:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8aed106f3c00 [938186.479937] LustreError: 21866:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ae99c3b1800 [938187.671203] Lustre: fir-MDT0001: Connection restored to aefc1bcc-c816-4 (at 10.50.2.25@o2ib2) [938187.679829] Lustre: Skipped 20 previous similar messages [938189.707939] Lustre: fir-MDT0001: Connection restored to 5b9787ef-1145-4 (at 10.50.2.22@o2ib2) [938191.438395] LustreError: 22533:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b0a1fe5a400 x1659597242516096/t0(0) o37->d87117b6-a627-4@10.50.2.11@o2ib2:23/0 lens 448/440 e 0 to 0 dl 1585490208 ref 1 fl Interpret:/0/0 rc 0/0 [938191.460626] LustreError: 22533:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 1 previous similar message [938192.818434] LustreError: 21870:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8b0bd369bc00 [938194.220992] Lustre: fir-MDT0001: Connection restored to 2e706dc4-6d42-4 (at 10.50.4.22@o2ib2) [938194.229636] Lustre: Skipped 13 previous similar messages [938194.725871] Lustre: fir-MDT0001: Client e47fbe1a-9f59-4 (at 10.50.4.17@o2ib2) reconnecting [938194.734235] Lustre: Skipped 37 previous similar messages [938196.901545] LustreError: 42023:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8ad1a24dec00 x1659459689736512/t0(0) o37->932170e3-8d55-4@10.50.5.8@o2ib2:10/0 lens 448/440 e 1 to 0 dl 1585490195 ref 1 fl Interpret:/2/0 rc 0/0 [938198.370266] Lustre: fir-MDT0001: Connection restored to 4a774151-5198-4 (at 10.50.5.15@o2ib2) [938198.378890] Lustre: Skipped 17 previous similar messages [938219.779845] Lustre: fir-MDT0001: Client 727fc66d-0b96-4 (at 10.50.1.10@o2ib2) reconnecting [938219.788237] Lustre: Skipped 19 previous similar messages [938219.793737] Lustre: fir-MDT0001: Connection restored to 727fc66d-0b96-4 (at 10.50.1.10@o2ib2) [938219.802361] Lustre: Skipped 3 previous similar messages [938237.321219] Lustre: fir-MDT0001: Connection restored to 480f04f5-a607-4 (at 10.50.4.63@o2ib2) [938237.329866] Lustre: Skipped 2 previous similar messages [938238.389668] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.226@o2ib7 added to recovery queue. Health = 900 [938238.389677] LustreError: 42105:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffff8b03cc28c050 x1661974834042624/t0(0) o37->0b484496-0c22-4@10.50.5.1@o2ib2:47/0 lens 448/440 e 1 to 0 dl 1585490232 ref 1 fl Interpret:/2/0 rc 0/0 [938263.646893] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.224@o2ib7 added to recovery queue. Health = 900 [938263.646906] LustreError: 22545:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4096) req@ffff8afb76a53050 x1659597242516096/t0(0) o37->d87117b6-a627-4@10.50.2.11@o2ib2:79/0 lens 448/440 e 2 to 0 dl 1585490264 ref 1 fl Interpret:/2/0 rc 0/0 [940633.697802] Lustre: fir-MDT0001: haven't heard from client bdc0155b-ddc0-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d65dfc400, cur 1585492611 expire 1585492461 last 1585492384 [940676.572572] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [940676.581193] Lustre: Skipped 4 previous similar messages [946147.850111] Lustre: fir-MDT0001: haven't heard from client 144229db-845e-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad27c212800, cur 1585498125 expire 1585497975 last 1585497898 [946195.071354] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [947523.823350] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [949057.929505] Lustre: fir-MDT0001: haven't heard from client 6e79d69f-c682-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec04af1000, cur 1585501035 expire 1585500885 last 1585500808 [949094.960266] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [949437.479243] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501407/real 1585501407] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501414 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [949444.506425] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501414/real 1585501414] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501421 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949451.535603] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501421/real 1585501421] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501428 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949458.562780] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501428/real 1585501428] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501435 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949472.591143] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501442/real 1585501442] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501449 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949472.618522] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [949493.628684] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501463/real 1585501463] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501470 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949493.656034] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [949528.666587] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585501498/real 1585501498] req@ffff8aec9ff20d80 x1661571629139264/t0(0) o104->fir-MDT0001@10.50.9.27@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585501505 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [949528.693960] Lustre: 42011:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [949584.705073] LustreError: 42011:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.9.27@o2ib2) failed to reply to blocking AST (req@ffff8aec9ff20d80 x1661571629139264 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8b0c7ca70240/0xbe1cb13d1282ed6b lrc: 4/0,0 mode: PR/PR res: [0x240048a7a:0x690f:0x0].0x0 bits 0x13/0x0 rrc: 62 type: IBT flags: 0x60200400000020 nid: 10.50.9.27@o2ib2 remote: 0x5b6dd27f81349b8f expref: 20 pid: 82461 timeout: 949699 lvb_type: 0 [949584.747941] LustreError: 42011:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message [949584.758065] LustreError: 138-a: fir-MDT0001: A client on nid 10.50.9.27@o2ib2 was evicted due to a lock blocking callback time out: rc -110 [949584.770672] LustreError: Skipped 1 previous similar message [949584.776373] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.50.9.27@o2ib2 ns: mdt-fir-MDT0001_UUID lock: ffff8b0c7ca70240/0xbe1cb13d1282ed6b lrc: 3/0,0 mode: PR/PR res: [0x240048a7a:0x690f:0x0].0x0 bits 0x13/0x0 rrc: 62 type: IBT flags: 0x60200400000020 nid: 10.50.9.27@o2ib2 remote: 0x5b6dd27f81349b8f expref: 21 pid: 82461 timeout: 0 lvb_type: 0 [949660.613526] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [956792.918879] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [956792.929141] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 7, oc: 0, rc: 8 [956945.716314] Lustre: fir-MDT0001: Connection restored to f82c1c78-5289-8d3f-c213-fe2a9908b1d9 (at 10.0.10.3@o2ib7) [958004.166316] Lustre: fir-MDT0001: haven't heard from client 0885c11b-bcdc-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd7eb51400, cur 1585509981 expire 1585509831 last 1585509754 [958061.224440] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [967366.424857] Lustre: fir-MDT0001: haven't heard from client 8c7a79ff-f053-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0961b64400, cur 1585519343 expire 1585519193 last 1585519116 [967456.973825] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [990282.067935] Lustre: fir-MDT0001: haven't heard from client 141984c0-0bb7-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acda6056800, cur 1585542258 expire 1585542108 last 1585542031 [990325.116515] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [992008.113683] Lustre: fir-MDT0001: haven't heard from client 3ca91764-a501-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad78da02400, cur 1585543984 expire 1585543834 last 1585543757 [992055.170101] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [992640.493448] Lustre: fir-MDT0001: Client 8f8b1101-4272-4 (at 10.50.17.38@o2ib2) reconnecting [992640.501928] Lustre: Skipped 7 previous similar messages [992640.507307] Lustre: fir-MDT0001: Connection restored to 8f8b1101-4272-4 (at 10.50.17.38@o2ib2) [992650.186822] Lustre: fir-MDT0001: Client 6f03212a-4363-4 (at 10.50.9.10@o2ib2) reconnecting [992650.195315] Lustre: fir-MDT0001: Connection restored to 6f03212a-4363-4 (at 10.50.9.10@o2ib2) [992722.186468] Lustre: fir-MDT0001: Client b1197e35-6879-4 (at 10.50.9.61@o2ib2) reconnecting [992722.195056] Lustre: fir-MDT0001: Connection restored to b1197e35-6879-4 (at 10.50.9.61@o2ib2) [992840.979351] Lustre: fir-MDT0001: Client 8f687c46-c0de-4 (at 10.50.5.16@o2ib2) reconnecting [992840.987773] Lustre: fir-MDT0001: Connection restored to 8f687c46-c0de-4 (at 10.50.5.16@o2ib2) [993081.369841] Lustre: fir-MDT0001: Client e3776b2c-e1e3-4 (at 10.50.10.33@o2ib2) reconnecting [993081.378381] Lustre: fir-MDT0001: Connection restored to e3776b2c-e1e3-4 (at 10.50.10.33@o2ib2) [993155.594643] Lustre: fir-MDT0001: Connection restored to a47ccd42-c337-4 (at 10.50.17.43@o2ib2) [993814.547533] Lustre: fir-MDT0001: Client e3776b2c-e1e3-4 (at 10.50.10.33@o2ib2) reconnecting [993814.556002] Lustre: Skipped 1 previous similar message [993814.561299] Lustre: fir-MDT0001: Connection restored to e3776b2c-e1e3-4 (at 10.50.10.33@o2ib2) [1000486.341636] Lustre: fir-MDT0001: haven't heard from client 76521fe8-386d-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae8ce6bc800, cur 1585552462 expire 1585552312 last 1585552235 [1000533.887668] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1031717.803623] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1031788.214574] Lustre: fir-MDT0001: haven't heard from client fdd6ec41-5cf5-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b054c638400, cur 1585583763 expire 1585583613 last 1585583536 [1058980.969301] Lustre: fir-MDT0001: haven't heard from client e47349d1-c447-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd22a43800, cur 1585610955 expire 1585610805 last 1585610728 [1061571.431467] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1061710.172158] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.7.12@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1061710.189633] LustreError: Skipped 1381 previous similar messages [1061718.174971] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.22.32@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1061718.192513] LustreError: Skipped 67 previous similar messages [1061742.344597] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.7.14@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1061742.362050] LustreError: Skipped 230 previous similar messages [1061774.787386] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.8.32@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1061774.804842] LustreError: Skipped 106 previous similar messages [1061777.986391] Lustre: fir-MDT0001: Client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) reconnecting [1061777.994849] Lustre: fir-MDT0001: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [1061937.565011] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1061937.582464] LustreError: Skipped 9 previous similar messages [1062069.086703] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.8.18@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1062249.063241] Lustre: fir-MDT0001: haven't heard from client c0c00923-1be8-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6fee20c00, cur 1585614223 expire 1585614073 last 1585613996 [1062290.995568] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1062355.752816] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1062355.770271] LustreError: Skipped 7 previous similar messages [1074699.763951] Lustre: 68056:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1074699.775781] Lustre: 68056:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2397 previous similar messages [1074703.765193] Lustre: 22463:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1074703.777019] Lustre: 22463:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2203 previous similar messages [1074711.769214] Lustre: 22383:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1074711.781037] Lustre: 22383:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2589 previous similar messages [1089758.007544] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1089758.019374] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 6 previous similar messages [1089760.011569] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1089760.023395] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 789 previous similar messages [1089764.082814] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1089764.094633] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 753 previous similar messages [1106350.325608] Lustre: fir-MDT0001: haven't heard from client e15458cc-5a0f-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae6c9a82c00, cur 1585658323 expire 1585658173 last 1585658096 [1106394.822590] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1119367.689538] Lustre: fir-MDT0001: haven't heard from client 540b2ede-974e-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d3e6a6c00, cur 1585671340 expire 1585671190 last 1585671113 [1119455.572079] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1129979.626199] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585681944/real 1585681944] req@ffff8afd6b523600 x1661582890786880/t0(0) o104->fir-MDT0001@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585681951 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1129979.653655] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [1129993.663588] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585681958/real 1585681958] req@ffff8afd6b523600 x1661582890786880/t0(0) o104->fir-MDT0001@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585681965 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1129993.691048] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message [1130014.701164] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585681979/real 1585681979] req@ffff8afd6b523600 x1661582890786880/t0(0) o104->fir-MDT0001@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585681986 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1130014.728635] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1130049.739100] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585682014/real 1585682014] req@ffff8afd6b523600 x1661582890786880/t0(0) o104->fir-MDT0001@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585682021 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1130049.766571] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [1130119.778004] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1585682084/real 1585682084] req@ffff8afd6b523600 x1661582890786880/t0(0) o104->fir-MDT0001@10.50.6.54@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1585682091 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [1130119.805438] Lustre: 41923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [1130120.975031] Lustre: fir-MDT0001: haven't heard from client 71c512fc-f1b5-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af68345e400, cur 1585682093 expire 1585681943 last 1585681866 [1130120.995165] LustreError: 41923:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.6.54@o2ib2) failed to reply to blocking AST (req@ffff8afd6b523600 x1661582890786880 status 0 rc -5), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8ae80a260fc0/0xbe1cb14baaca6712 lrc: 4/0,0 mode: PR/PR res: [0x240048a7a:0x690f:0x0].0x0 bits 0x13/0x0 rrc: 97 type: IBT flags: 0x60200400000020 nid: 10.50.6.54@o2ib2 remote: 0x128f3bf1c21eba1c expref: 48 pid: 23159 timeout: 1130236 lvb_type: 0 [1130121.038051] LustreError: 138-a: fir-MDT0001: A client on nid 10.50.6.54@o2ib2 was evicted due to a lock blocking callback time out: rc -5 [1130170.521146] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1152538.600288] Lustre: fir-MDT0001: haven't heard from client 21c8c585-5f1e-4 (at 10.50.8.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d65422c00, cur 1585704510 expire 1585704360 last 1585704283 [1157197.958650] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1157197.970481] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 628 previous similar messages [1157198.958574] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1157198.970395] Lustre: 22466:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 536 previous similar messages [1157200.960363] Lustre: 22288:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1157200.972193] Lustre: 22288:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1113 previous similar messages [1157205.034892] Lustre: 68055:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1157205.046713] Lustre: 68055:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1829 previous similar messages [1182346.489723] Lustre: fir-MDT0001: haven't heard from client 51a2b347-1573-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afce77db000, cur 1585734317 expire 1585734167 last 1585734090 [1182393.655565] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1209361.220049] Lustre: fir-MDT0001: haven't heard from client 39fa7b7d-0126-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae789afb400, cur 1585761331 expire 1585761181 last 1585761104 [1209406.922372] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1215429.266889] Lustre: fir-MDT0001: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1215429.275359] Lustre: fir-MDT0001: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1215454.519425] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.0.71@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1215454.536883] LustreError: Skipped 3 previous similar messages [1215529.784372] Lustre: fir-MDT0001: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1215529.792835] Lustre: fir-MDT0001: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1215597.763540] Lustre: fir-MDT0001: Client b0013499-7d1a-4 (at 10.50.10.62@o2ib2) reconnecting [1215597.772093] Lustre: fir-MDT0001: Connection restored to b0013499-7d1a-4 (at 10.50.10.62@o2ib2) [1215597.780881] Lustre: Skipped 1 previous similar message [1215606.503640] Lustre: fir-MDT0001: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1215606.512077] Lustre: Skipped 1 previous similar message [1215606.517419] Lustre: fir-MDT0001: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1215613.793339] Lustre: fir-MDT0001: Client 158ae5ee-63a7-4 (at 10.50.10.72@o2ib2) reconnecting [1215613.801870] Lustre: Skipped 4 previous similar messages [1215613.807299] Lustre: fir-MDT0001: Connection restored to 158ae5ee-63a7-4 (at 10.50.10.72@o2ib2) [1215613.816089] Lustre: Skipped 4 previous similar messages [1215638.098996] Lustre: fir-MDT0001: Client 933e4cd4-a6c0-4 (at 10.50.10.63@o2ib2) reconnecting [1215638.107529] Lustre: Skipped 1 previous similar message [1215638.112870] Lustre: fir-MDT0001: Connection restored to 933e4cd4-a6c0-4 (at 10.50.10.63@o2ib2) [1215638.121664] Lustre: Skipped 1 previous similar message [1217812.900820] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1217812.909607] Lustre: Skipped 3 previous similar messages [1217859.469602] Lustre: fir-MDT0001: haven't heard from client b449688d-0d11-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6780ac00, cur 1585769829 expire 1585769679 last 1585769602 [1218293.819190] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1218342.522187] Lustre: fir-MDT0001: haven't heard from client f23e2d13-b3ae-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0c14283000, cur 1585770312 expire 1585770162 last 1585770085 [1218898.508454] Lustre: fir-MDT0001: haven't heard from client f0da1f04-da4c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aee7d2c1c00, cur 1585770868 expire 1585770718 last 1585770641 [1218901.917461] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1234058.574072] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1234111.008001] Lustre: fir-MDT0001: haven't heard from client 1b6de69e-1c49-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec7eb2b800, cur 1585786080 expire 1585785930 last 1585785853 [1264156.805183] Lustre: fir-MDT0001: haven't heard from client 479a23d1-e582-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add3b90b800, cur 1585816125 expire 1585815975 last 1585815898 [1264223.613458] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1277214.472694] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1277274.183632] Lustre: fir-MDT0001: haven't heard from client 7fa477f6-65a5-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad6371b0800, cur 1585829242 expire 1585829092 last 1585829015 [1277799.779468] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1277855.192371] Lustre: fir-MDT0001: haven't heard from client b34989b7-841a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d79987400, cur 1585829823 expire 1585829673 last 1585829596 [1278036.198272] Lustre: fir-MDT0001: haven't heard from client aaf569c5-b770-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc65e91800, cur 1585830004 expire 1585829854 last 1585829777 [1278252.211421] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1278502.669515] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1278581.326019] Lustre: fir-MDT0001: haven't heard from client 0bcab18e-ca9e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb51affc00, cur 1585830549 expire 1585830399 last 1585830322 [1278896.616552] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1278932.243371] Lustre: fir-MDT0001: haven't heard from client 23290f18-a3b6-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd78a3bc00, cur 1585830900 expire 1585830750 last 1585830673 [1279401.241606] Lustre: fir-MDT0001: haven't heard from client 4c28cfb2-f28b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea41e3f000, cur 1585831369 expire 1585831219 last 1585831142 [1281702.968356] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1281762.309953] Lustre: fir-MDT0001: haven't heard from client df593509-e01a-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad022f15c00, cur 1585833730 expire 1585833580 last 1585833503 [1285668.673819] Lustre: fir-MDT0001: Client 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) reconnecting [1285668.682200] Lustre: Skipped 3 previous similar messages [1285668.687643] Lustre: fir-MDT0001: Connection restored to 547acdb4-58e5-4 (at 10.50.9.1@o2ib2) [1286922.813142] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1287874.366350] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1287903.609795] Lustre: fir-MDT0001: haven't heard from client 58a1fac0-9465-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afbe3778400, cur 1585839871 expire 1585839721 last 1585839644 [1287914.124264] Lustre: fir-MDT0001: Connection restored to 21c8c585-5f1e-4 (at 10.50.8.18@o2ib2) [1288819.531772] Lustre: fir-MDT0001: haven't heard from client 0b982012-b4a2-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0cfa204000, cur 1585840787 expire 1585840637 last 1585840560 [1288907.083982] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1301395.877478] Lustre: fir-MDT0001: haven't heard from client 74a22577-dbc5-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af52e75ec00, cur 1585853363 expire 1585853213 last 1585853136 [1301425.406085] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1301794.423049] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1301854.883083] Lustre: fir-MDT0001: haven't heard from client 34f0ba71-4d3e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6f59f800, cur 1585853822 expire 1585853672 last 1585853595 [1302348.899041] Lustre: fir-MDT0001: haven't heard from client a6f39580-441f-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb167ca800, cur 1585854316 expire 1585854166 last 1585854089 [1302415.229816] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1303249.370129] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1303295.938119] Lustre: fir-MDT0001: haven't heard from client ab887596-2654-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed4f54a800, cur 1585855263 expire 1585855113 last 1585855036 [1304183.402669] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1304205.952372] Lustre: fir-MDT0001: haven't heard from client 7800f108-edbf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed7e508000, cur 1585856173 expire 1585856023 last 1585855946 [1304711.964800] Lustre: fir-MDT0001: haven't heard from client 25db87c4-882a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd3a9fdc00, cur 1585856679 expire 1585856529 last 1585856452 [1304962.660472] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1305283.373355] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1305341.986302] Lustre: fir-MDT0001: haven't heard from client b4ff50cd-f32a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0837765c00, cur 1585857309 expire 1585857159 last 1585857082 [1306691.020915] Lustre: fir-MDT0001: haven't heard from client ddb48759-c809-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6f59c800, cur 1585858658 expire 1585858508 last 1585858431 [1306717.771546] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1307435.241633] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1307464.044406] Lustre: fir-MDT0001: haven't heard from client 4e2e9ff3-63e1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6289c8c00, cur 1585859431 expire 1585859281 last 1585859204 [1308376.499558] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1308391.074987] Lustre: fir-MDT0001: haven't heard from client 40c9ab89-a8eb-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb51affc00, cur 1585860358 expire 1585860208 last 1585860131 [1309232.092776] Lustre: fir-MDT0001: haven't heard from client 0d59efc3-4437-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea41e3a400, cur 1585861199 expire 1585861049 last 1585860972 [1309258.850777] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1310190.121677] Lustre: fir-MDT0001: haven't heard from client 74c0ee2c-8abf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af18864f400, cur 1585862157 expire 1585862007 last 1585861930 [1310214.489816] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1311020.171680] Lustre: fir-MDT0001: haven't heard from client 7f24d805-4b6c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8affec7c0400, cur 1585862987 expire 1585862837 last 1585862760 [1311062.850700] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1312091.766045] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1312094.180590] Lustre: fir-MDT0001: haven't heard from client dae56af1-c8a1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af4ce64d000, cur 1585864061 expire 1585863911 last 1585863834 [1312972.200438] Lustre: fir-MDT0001: haven't heard from client 5582702c-e427-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aefd7aea400, cur 1585864939 expire 1585864789 last 1585864712 [1312991.705982] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1313673.001969] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1313722.222687] Lustre: fir-MDT0001: haven't heard from client a5eeac10-067e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af731278000, cur 1585865689 expire 1585865539 last 1585865462 [1314395.672522] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1314428.244106] Lustre: fir-MDT0001: haven't heard from client e22e23c2-e0bf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d4fb49400, cur 1585866395 expire 1585866245 last 1585866168 [1315389.271069] Lustre: fir-MDT0001: haven't heard from client 8dc15a3d-db81-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afb68aeb400, cur 1585867356 expire 1585867206 last 1585867129 [1315437.014919] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1316275.297556] Lustre: fir-MDT0001: haven't heard from client 5124239a-c213-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed7e50c000, cur 1585868242 expire 1585868092 last 1585868015 [1316311.835453] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1317076.061615] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1317091.331810] Lustre: fir-MDT0001: haven't heard from client c867a4b7-7672-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73cbc000, cur 1585869058 expire 1585868908 last 1585868831 [1317841.261712] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1317880.345138] Lustre: fir-MDT0001: haven't heard from client 08103f19-de06-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd3501dc00, cur 1585869847 expire 1585869697 last 1585869620 [1318725.556635] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1318772.369600] Lustre: fir-MDT0001: haven't heard from client 69fe9fc9-3326-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6524e8800, cur 1585870739 expire 1585870589 last 1585870512 [1319591.559691] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1319631.394935] Lustre: fir-MDT0001: haven't heard from client 926cf9a9-13e7-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc8b2f2c00, cur 1585871598 expire 1585871448 last 1585871371 [1319824.399664] Lustre: fir-MDT0001: haven't heard from client 9fafd4c0-0b7b-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b096fedac00, cur 1585871791 expire 1585871641 last 1585871564 [1319863.948775] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1320353.781900] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1320372.418049] Lustre: fir-MDT0001: haven't heard from client c65fab57-35bd-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af69251bc00, cur 1585872339 expire 1585872189 last 1585872112 [1321140.515719] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1321159.446926] Lustre: fir-MDT0001: haven't heard from client 2aa88a3c-b748-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae681e28000, cur 1585873126 expire 1585872976 last 1585872899 [1321739.790473] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1321745.457940] Lustre: fir-MDT0001: haven't heard from client 5fea1caa-a7bb-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6524ee000, cur 1585873712 expire 1585873562 last 1585873485 [1322334.859616] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1322369.475582] Lustre: fir-MDT0001: haven't heard from client 4fc7f937-8bf2-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a86d9c00, cur 1585874336 expire 1585874186 last 1585874109 [1323240.528457] Lustre: fir-MDT0001: haven't heard from client 3abc36c6-05e4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a7f27800, cur 1585875207 expire 1585875057 last 1585874980 [1323342.652909] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1323925.965345] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1323947.525685] Lustre: fir-MDT0001: haven't heard from client 457a238b-523a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a7f22000, cur 1585875914 expire 1585875764 last 1585875687 [1324555.539507] Lustre: fir-MDT0001: haven't heard from client f8ebc65b-1d76-4 (at 10.49.21.13@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d67ad7000, cur 1585876522 expire 1585876372 last 1585876295 [1324631.915511] Lustre: fir-MDT0001: Connection restored to f8ebc65b-1d76-4 (at 10.49.21.13@o2ib1) [1324955.555904] Lustre: fir-MDT0001: haven't heard from client cb3fb692-e931-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0c14286c00, cur 1585876922 expire 1585876772 last 1585876695 [1325037.481651] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1327938.399211] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1327938.411037] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 41 previous similar messages [1327939.399994] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1327939.411823] Lustre: 22417:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 606 previous similar messages [1331111.737872] Lustre: fir-MDT0001: haven't heard from client 0317481a-dbc1-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0cffa47400, cur 1585883078 expire 1585882928 last 1585882851 [1331177.616055] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1349682.715964] Lustre: fir-MDT0001: Connection restored to 06f6e79a-f25a-4 (at 10.49.23.21@o2ib1) [1349743.267632] Lustre: fir-MDT0001: haven't heard from client 06f6e79a-f25a-4 (at 10.49.23.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d7ef74400, cur 1585901709 expire 1585901559 last 1585901482 [1368632.978651] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1368662.815263] Lustre: fir-MDT0001: haven't heard from client e11b8b3f-72d0-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aecb8719000, cur 1585920628 expire 1585920478 last 1585920401 [1369367.499360] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1369412.840375] Lustre: fir-MDT0001: haven't heard from client d474ebd5-a9fc-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6657e3800, cur 1585921378 expire 1585921228 last 1585921151 [1370323.920215] Lustre: fir-MDT0001: haven't heard from client e2f38f60-659f-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae78ce15400, cur 1585922289 expire 1585922139 last 1585922062 [1370335.574655] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1371040.310406] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1371090.888694] Lustre: fir-MDT0001: haven't heard from client 46b016f2-d1f4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8affec7c7000, cur 1585923056 expire 1585922906 last 1585922829 [1371166.889553] Lustre: fir-MDT0001: haven't heard from client 681deff3-eb13-4 (at 10.50.6.54@o2ib2) in 204 seconds. I think it's dead, and I am evicting it. exp ffff8b0d44f82000, cur 1585923132 expire 1585922982 last 1585922928 [1371239.728394] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1371970.914502] Lustre: fir-MDT0001: haven't heard from client 7d2ba658-0baa-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd473bc400, cur 1585923936 expire 1585923786 last 1585923709 [1372033.687881] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1372813.977982] Lustre: fir-MDT0001: haven't heard from client f0184f56-3555-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad6371b6000, cur 1585924779 expire 1585924629 last 1585924552 [1372846.461487] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1373721.599748] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1373725.965997] Lustre: fir-MDT0001: haven't heard from client 658cd963-8a05-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0ce7fbd800, cur 1585925691 expire 1585925541 last 1585925464 [1374464.323578] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1374526.989884] Lustre: fir-MDT0001: haven't heard from client e0d2a6f9-90c8-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b054c63e000, cur 1585926492 expire 1585926342 last 1585926265 [1375316.379856] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1375345.009707] Lustre: fir-MDT0001: haven't heard from client 67ebe6f5-5aae-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d3e6a0000, cur 1585927310 expire 1585927160 last 1585927083 [1376322.039833] Lustre: fir-MDT0001: haven't heard from client 996bbcc9-9596-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed77c54800, cur 1585928287 expire 1585928137 last 1585928060 [1376344.581045] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1377123.414297] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1377150.065658] Lustre: fir-MDT0001: haven't heard from client 62b6f27c-0c27-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af5a6faa800, cur 1585929115 expire 1585928965 last 1585928888 [1378029.788804] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1378054.087925] Lustre: fir-MDT0001: haven't heard from client 315777c7-4f4b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd21ad7000, cur 1585930019 expire 1585929869 last 1585929792 [1378853.727688] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1378885.120346] Lustre: fir-MDT0001: haven't heard from client 1adc3b55-8749-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea02692c00, cur 1585930850 expire 1585930700 last 1585930623 [1378993.697391] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1379702.803948] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1379734.190771] Lustre: fir-MDT0001: haven't heard from client a9261f64-2c5e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a17330000, cur 1585931699 expire 1585931549 last 1585931472 [1379734.210913] Lustre: Skipped 1 previous similar message [1380583.166532] Lustre: fir-MDT0001: haven't heard from client a93e6deb-131a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd22a45000, cur 1585932548 expire 1585932398 last 1585932321 [1380607.906778] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1380640.275519] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1381338.694346] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1381388.184030] Lustre: fir-MDT0001: haven't heard from client f414c76d-7b01-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af958ebac00, cur 1585933353 expire 1585933203 last 1585933126 [1381388.204172] Lustre: Skipped 1 previous similar message [1382298.208040] Lustre: fir-MDT0001: haven't heard from client 0ec25327-8576-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc3dbec400, cur 1585934263 expire 1585934113 last 1585934036 [1382345.210675] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1382450.324361] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1382495.213113] Lustre: fir-MDT0001: haven't heard from client a0f2210c-95b3-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af958ebec00, cur 1585934460 expire 1585934310 last 1585934233 [1383623.713885] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1383682.252923] Lustre: fir-MDT0001: haven't heard from client 8eae559f-8716-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af20dfe2c00, cur 1585935647 expire 1585935497 last 1585935420 [1384537.908991] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1384554.288224] Lustre: fir-MDT0001: haven't heard from client 48846965-459c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed6b648800, cur 1585936519 expire 1585936369 last 1585936292 [1385007.283848] Lustre: fir-MDT0001: haven't heard from client c1177a89-5194-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6977bc00, cur 1585936972 expire 1585936822 last 1585936745 [1385044.235065] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1385166.342816] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1385218.290260] Lustre: fir-MDT0001: haven't heard from client 304e6b7a-f74b-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6a747000, cur 1585937183 expire 1585937033 last 1585936956 [1386122.668011] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1386172.329902] Lustre: fir-MDT0001: haven't heard from client e26a0b8d-df3e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5a26c800, cur 1585938137 expire 1585937987 last 1585937910 [1386929.337774] Lustre: fir-MDT0001: haven't heard from client 539d87e3-6b19-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af69935ec00, cur 1585938894 expire 1585938744 last 1585938667 [1386971.283343] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1389005.395470] Lustre: fir-MDT0001: haven't heard from client 835afd83-dfec-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afccba6c800, cur 1585940970 expire 1585940820 last 1585940743 [1389050.324385] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1392013.480862] Lustre: fir-MDT0001: haven't heard from client 5ab99ad1-8b26-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a8bf3f000, cur 1585943978 expire 1585943828 last 1585943751 [1392239.291492] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1392272.495950] Lustre: fir-MDT0001: haven't heard from client 7f196257-4021-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d67830c00, cur 1585944237 expire 1585944087 last 1585944010 [1392277.553330] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1392448.888003] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1392504.493534] Lustre: fir-MDT0001: haven't heard from client 3f36eb03-79bd-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6d419cc00, cur 1585944469 expire 1585944319 last 1585944242 [1392623.877959] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1392686.498851] Lustre: fir-MDT0001: haven't heard from client 274a7547-b7aa-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec5ffd6400, cur 1585944651 expire 1585944501 last 1585944424 [1394374.555165] Lustre: fir-MDT0001: haven't heard from client 73f503d3-884e-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0b63a61c00, cur 1585946339 expire 1585946189 last 1585946112 [1394423.747916] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1396132.601853] Lustre: fir-MDT0001: haven't heard from client 75373d48-fe8f-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad7db337c00, cur 1585948097 expire 1585947947 last 1585947870 [1396173.276780] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1399763.702917] Lustre: fir-MDT0001: haven't heard from client 69cc2ce3-9c83-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a13aba400, cur 1585951728 expire 1585951578 last 1585951501 [1399821.408170] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1400455.288261] Lustre: 22447:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1400455.300084] Lustre: 22447:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 710 previous similar messages [1400455.790427] Lustre: 22288:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1400455.802253] Lustre: 22288:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 264 previous similar messages [1400456.793700] Lustre: 22389:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1400456.805528] Lustre: 22389:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 584 previous similar messages [1400458.797175] Lustre: 22479:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1400458.809002] Lustre: 22479:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1145 previous similar messages [1400462.810181] Lustre: 22479:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1400462.822006] Lustre: 22479:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1875 previous similar messages [1426217.467702] Lustre: fir-MDT0001: haven't heard from client 037ca474-0477-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af8ff68a400, cur 1585978181 expire 1585978031 last 1585977954 [1426265.900218] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1438857.828215] Lustre: fir-MDT0001: haven't heard from client cc32a056-ed4a-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad75bb44000, cur 1585990821 expire 1585990671 last 1585990594 [1438887.702180] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1440344.871451] Lustre: fir-MDT0001: haven't heard from client e4687612-d8e5-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add0f6d3800, cur 1585992308 expire 1585992158 last 1585992081 [1440382.831841] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1447760.976514] Lustre: fir-MDT0001: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1447783.081374] Lustre: fir-MDT0001: haven't heard from client bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add59ba8000, cur 1585999746 expire 1585999596 last 1585999519 [1452357.552060] Lustre: fir-MDT0001: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1452402.212790] Lustre: fir-MDT0001: haven't heard from client 9b7d199c-0123-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae5373aa800, cur 1586004365 expire 1586004215 last 1586004138 [1454235.267212] Lustre: fir-MDT0001: haven't heard from client 934ddb45-94a2-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5998f000, cur 1586006198 expire 1586006048 last 1586005971 [1454271.341924] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1454979.952488] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1455041.289843] Lustre: fir-MDT0001: haven't heard from client 68109525-701f-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad559cccc00, cur 1586007004 expire 1586006854 last 1586006777 [1458933.400672] Lustre: fir-MDT0001: haven't heard from client 3dd153d6-2947-4 (at 10.50.4.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d18213000, cur 1586010896 expire 1586010746 last 1586010669 [1459322.511477] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1459361.412086] Lustre: fir-MDT0001: haven't heard from client 0cb0db66-c8ed-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af75f399400, cur 1586011324 expire 1586011174 last 1586011097 [1459902.429940] Lustre: fir-MDT0001: haven't heard from client 9b0c99d2-345c-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b004a792800, cur 1586011865 expire 1586011715 last 1586011638 [1459950.695044] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1460288.209944] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1460326.440719] Lustre: fir-MDT0001: haven't heard from client ca6aa2c0-a51a-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af1a925c000, cur 1586012289 expire 1586012139 last 1586012062 [1460374.720151] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1460402.447027] Lustre: fir-MDT0001: haven't heard from client 33c56caf-f05e-4 (at 10.49.21.21@o2ib1) in 199 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73cb8400, cur 1586012365 expire 1586012215 last 1586012166 [1462613.051440] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1462635.504298] Lustre: fir-MDT0001: haven't heard from client f072dd11-76e6-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6e2db800, cur 1586014598 expire 1586014448 last 1586014371 [1462711.507722] Lustre: fir-MDT0001: haven't heard from client 5a471ce2-27f1-4 (at 10.50.6.54@o2ib2) in 176 seconds. I think it's dead, and I am evicting it. exp ffff8ad35c46d000, cur 1586014674 expire 1586014524 last 1586014498 [1462858.064988] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [1465249.579308] Lustre: fir-MDT0001: haven't heard from client d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786b8800, cur 1586017212 expire 1586017062 last 1586016985 [1465315.841846] Lustre: fir-MDT0001: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [1466126.604588] Lustre: fir-MDT0001: haven't heard from client 97017afb-9fdb-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b04e47a1800, cur 1586018089 expire 1586017939 last 1586017862 [1466153.739528] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1477027.781980] Lustre: fir-MDT0001: Connection restored to ee56fbe2-040d-4 (at 10.49.25.17@o2ib1) [1481443.033535] Lustre: fir-MDT0001: haven't heard from client ff3d8c7f-ee13-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af52e75fc00, cur 1586033405 expire 1586033255 last 1586033178 [1481477.356148] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1491971.336435] Lustre: fir-MDT0001: haven't heard from client dfbfebbc-fbbb-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8adaf2b41400, cur 1586043933 expire 1586043783 last 1586043706 [1492007.285270] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1506527.749473] Lustre: fir-MDT0001: haven't heard from client 03fdb3f2-9fe5-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af17738c000, cur 1586058489 expire 1586058339 last 1586058262 [1506594.683424] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1506833.750777] Lustre: fir-MDT0001: haven't heard from client af8b075a-d960-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0961b61000, cur 1586058795 expire 1586058645 last 1586058568 [1506858.741792] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1507043.771209] Lustre: fir-MDT0001: haven't heard from client 05c68320-d699-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af936720800, cur 1586059005 expire 1586058855 last 1586058778 [1507080.432862] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1509141.817462] Lustre: fir-MDT0001: haven't heard from client ce295894-cf8f-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0cfdb11000, cur 1586061103 expire 1586060953 last 1586060876 [1509251.424494] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1511408.890795] Lustre: fir-MDT0001: haven't heard from client 87e3819d-30dd-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0cea2a8c00, cur 1586063370 expire 1586063220 last 1586063143 [1511445.082325] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1513780.945683] Lustre: fir-MDT0001: haven't heard from client d497665e-1a35-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add66ef1c00, cur 1586065742 expire 1586065592 last 1586065515 [1513828.679639] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1520610.136807] Lustre: fir-MDT0001: haven't heard from client 8d22f512-aa17-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add62e4b400, cur 1586072571 expire 1586072421 last 1586072344 [1520651.951452] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1521006.148672] Lustre: fir-MDT0001: haven't heard from client e51c3087-0e32-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af6fee21000, cur 1586072967 expire 1586072817 last 1586072740 [1521074.621724] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1523747.226444] Lustre: fir-MDT0001: haven't heard from client 04237cd7-b89c-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af690c26400, cur 1586075708 expire 1586075558 last 1586075481 [1523770.342902] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1526030.290359] Lustre: fir-MDT0001: haven't heard from client a734e045-6ae0-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc5c7c1800, cur 1586077991 expire 1586077841 last 1586077764 [1526052.312137] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1531877.452937] Lustre: fir-MDT0001: haven't heard from client 3ac90f8e-5504-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0cfa202000, cur 1586083838 expire 1586083688 last 1586083611 [1535778.563497] Lustre: fir-MDT0001: haven't heard from client 71bbaa66-52fd-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad37fce9800, cur 1586087739 expire 1586087589 last 1586087512 [1535837.413731] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1539437.668489] Lustre: fir-MDT0001: haven't heard from client 5c81c5e2-1ff2-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae9da777400, cur 1586091398 expire 1586091248 last 1586091171 [1539802.931327] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [1546552.866838] Lustre: fir-MDT0001: haven't heard from client 29f566cd-115e-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73cb9400, cur 1586098513 expire 1586098363 last 1586098286 [1546596.942838] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1564989.380920] Lustre: fir-MDT0001: haven't heard from client 90b6f92d-d059-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af71bec8000, cur 1586116949 expire 1586116799 last 1586116722 [1565231.302806] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1569607.171044] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [1569607.182613] Lustre: 22386:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 1251 previous similar messages [1569682.381299] Lustre: 22457:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [1569682.392882] Lustre: 22457:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 160 previous similar messages [1569832.672514] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [1569832.684107] Lustre: 22434:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 400 previous similar messages [1570133.989719] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [1570134.001294] Lustre: 22346:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 639 previous similar messages [1571312.918128] Lustre: 22412:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586123265/real 1586123265] req@ffff8ad5754a7500 x1661593095704576/t0(0) o104->fir-MDT0001@10.49.0.64@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1586123272 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1571326.530022] Lustre: fir-MDT0001: Client 2f954eed-4b6c-4 (at 10.49.27.9@o2ib1) reconnecting [1571326.538513] Lustre: fir-MDT0001: Connection restored to 2f954eed-4b6c-4 (at 10.49.27.9@o2ib1) [1571327.067316] Lustre: fir-MDT0001: Client ef304413-5f16-4 (at 10.49.24.25@o2ib1) reconnecting [1571327.075871] Lustre: Skipped 2 previous similar messages [1571327.081301] Lustre: fir-MDT0001: Connection restored to ef304413-5f16-4 (at 10.49.24.25@o2ib1) [1571327.090091] Lustre: Skipped 2 previous similar messages [1571328.127553] Lustre: fir-MDT0001: Client aa34587f-505a-4 (at 10.49.18.30@o2ib1) reconnecting [1571328.136109] Lustre: Skipped 12 previous similar messages [1571328.141664] Lustre: fir-MDT0001: Connection restored to aa34587f-505a-4 (at 10.49.18.30@o2ib1) [1571328.150459] Lustre: Skipped 12 previous similar messages [1571337.570723] Lustre: fir-MDT0001: Client 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) reconnecting [1571337.579174] Lustre: Skipped 12 previous similar messages [1571337.584823] Lustre: fir-MDT0001: Connection restored to 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) [1571337.593531] Lustre: Skipped 12 previous similar messages [1571352.083289] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.30.14@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1571360.312675] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.7.20@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1571360.330139] LustreError: Skipped 273 previous similar messages [1571404.356330] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.28.1@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1571404.373852] LustreError: Skipped 24 previous similar messages [1589396.059560] Lustre: fir-MDT0001: haven't heard from client 89b8930d-7d49-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d44f85c00, cur 1586141355 expire 1586141205 last 1586141128 [1589433.268505] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1593158.155924] Lustre: fir-MDT0001: haven't heard from client fe9d3211-9fa0-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea656d1800, cur 1586145117 expire 1586144967 last 1586144890 [1593200.563277] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1617090.045071] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1617118.834266] Lustre: fir-MDT0001: haven't heard from client 5b0cb4e2-0c62-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed14ea1c00, cur 1586169077 expire 1586168927 last 1586168850 [1626526.101084] Lustre: fir-MDT0001: haven't heard from client b9dc3cb9-ba1a-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed11258400, cur 1586178484 expire 1586178334 last 1586178257 [1626613.207088] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1628009.211654] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1628046.143426] Lustre: fir-MDT0001: haven't heard from client 5d276014-2b20-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad0e3138c00, cur 1586180004 expire 1586179854 last 1586179777 [1642024.522556] Lustre: fir-MDT0001: haven't heard from client 3367b05d-b386-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad234db6000, cur 1586193982 expire 1586193832 last 1586193755 [1642081.800768] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1643072.249345] Lustre: fir-MDT0001: Client 78da859a-079c-4 (at 10.50.4.27@o2ib2) reconnecting [1643072.257872] Lustre: fir-MDT0001: Connection restored to 78da859a-079c-4 (at 10.50.4.27@o2ib2) [1643072.327558] LustreError: 42053:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b0cef606c00 x1659658758999104/t0(0) o37->2a4e319d-3b9c-4@10.50.10.60@o2ib2:472/0 lens 448/440 e 0 to 0 dl 1586195072 ref 1 fl Interpret:/0/0 rc 0/0 [1643072.772926] Lustre: fir-MDT0001: Client 2b2c7387-9609-4 (at 10.50.4.23@o2ib2) reconnecting [1643072.781400] Lustre: Skipped 8 previous similar messages [1643072.786897] Lustre: fir-MDT0001: Connection restored to 2b2c7387-9609-4 (at 10.50.4.23@o2ib2) [1643072.795629] Lustre: Skipped 8 previous similar messages [1643073.299520] LustreError: 42020:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8ad06ec46300 x1662160510080000/t0(0) o37->a9b23a4c-a92b-4@10.50.15.10@o2ib2:472/0 lens 448/440 e 0 to 0 dl 1586195072 ref 1 fl Interpret:/0/0 rc 0/0 [1643101.281837] Lustre: fir-MDT0001: Client 99ec6091-5082-4 (at 10.50.8.69@o2ib2) reconnecting [1643101.290333] Lustre: fir-MDT0001: Connection restored to 99ec6091-5082-4 (at 10.50.8.69@o2ib2) [1643109.284186] Lustre: fir-MDT0001: Client a8000a9c-9e85-4 (at 10.50.2.18@o2ib2) reconnecting [1643109.292657] Lustre: Skipped 4 previous similar messages [1643109.298111] Lustre: fir-MDT0001: Connection restored to a8000a9c-9e85-4 (at 10.50.2.18@o2ib2) [1643109.306824] Lustre: Skipped 4 previous similar messages [1649881.306835] Lustre: fir-MDT0001: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [1649881.315457] Lustre: Skipped 2 previous similar messages [1656038.435393] Lustre: fir-MDT0001: Connection restored to bb05bfe6-c379-4 (at 10.50.4.28@o2ib2) [1657311.502520] Lustre: fir-MDT0001: Connection restored to 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) [1657363.947462] Lustre: fir-MDT0001: haven't heard from client 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73059400, cur 1586209321 expire 1586209171 last 1586209094 [1717130.140678] Lustre: fir-MDT0001: Connection restored to b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) [1717152.644917] Lustre: fir-MDT0001: haven't heard from client b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d71783400, cur 1586269108 expire 1586268958 last 1586268881 [1724736.387561] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [1724753.854328] Lustre: fir-MDT0001: haven't heard from client 19ab4ca3-4eb4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af91c3ef800, cur 1586276709 expire 1586276559 last 1586276482 [1728263.373973] Lustre: fir-MDT0001: Connection restored to b3a54002-f8c5-4 (at 10.50.12.2@o2ib2) [1730384.007717] Lustre: fir-MDT0001: haven't heard from client 7c15878b-7f74-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a86da800, cur 1586282339 expire 1586282189 last 1586282112 [1730417.198379] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1748657.517358] Lustre: fir-MDT0001: haven't heard from client eeb6444d-4e51-4 (at 10.50.16.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec37fc9c00, cur 1586300612 expire 1586300462 last 1586300385 [1748678.820024] Lustre: fir-MDT0001: Connection restored to 6c45c03c-4b15-4 (at 10.50.16.6@o2ib2) [1760537.857291] Lustre: fir-MDT0001: haven't heard from client 2ad6fa60-c8a2-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb8e2de000, cur 1586312492 expire 1586312342 last 1586312265 [1760586.918851] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1763439.140616] Lustre: 21913:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586314792/real 1586314792] req@ffff8b0d59bcc800 x1661612394054208/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586315393 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1763439.168994] Lustre: 21913:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1763439.178914] Lustre: fir-OST0034-osc-MDT0001: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1763439.195415] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1763693.062224] INFO: task mdt03_016:22391 blocked for more than 120 seconds. [1763693.069193] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763693.077206] mdt03_016 D ffff8afd598b2080 0 22391 2 0x00000080 [1763693.084488] Call Trace: [1763693.087147] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763693.094030] [] schedule+0x29/0x70 [1763693.099189] [] rwsem_down_write_failed+0x225/0x3a0 [1763693.105822] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [1763693.112618] [] call_rwsem_down_write_failed+0x17/0x30 [1763693.119514] [] down_write+0x2d/0x3d [1763693.124850] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763693.131736] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763693.138641] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763693.145270] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763693.152598] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763693.160452] [] lod_prepare_create+0x215/0x2e0 [lod] [1763693.167163] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763693.174574] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763693.181753] [] lod_declare_create+0x204/0x590 [lod] [1763693.188469] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763693.196485] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763693.203128] [] mdd_create+0x867/0x14a0 [mdd] [1763693.209238] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763693.215796] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763693.223483] [] mdt_reint_rec+0x83/0x210 [mdt] [1763693.229677] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763693.236387] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763693.243639] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763693.250026] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763693.257344] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763693.263989] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763693.271327] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763693.278214] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763693.285550] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763693.292109] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763693.299368] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763693.307085] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763693.313392] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763693.320502] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763693.328256] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763693.335523] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763693.343386] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763693.350366] [] ? __wake_up+0x44/0x50 [1763693.355802] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763693.362278] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763693.369864] [] kthread+0xd1/0xe0 [1763693.374920] [] ? insert_kthread_work+0x40/0x40 [1763693.381197] [] ret_from_fork_nospec_begin+0xe/0x21 [1763693.387845] [] ? insert_kthread_work+0x40/0x40 [1763693.394130] INFO: task mdt00_053:22466 blocked for more than 120 seconds. [1763693.401093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763693.409116] mdt00_053 D ffff8b0d65de2080 0 22466 2 0x00000080 [1763693.416386] Call Trace: [1763693.419020] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763693.425901] [] schedule+0x29/0x70 [1763693.431080] [] rwsem_down_write_failed+0x225/0x3a0 [1763693.437696] [] call_rwsem_down_write_failed+0x17/0x30 [1763693.444575] [] down_write+0x2d/0x3d [1763693.449933] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763693.456821] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763693.463711] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763693.470357] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763693.477675] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763693.485517] [] lod_prepare_create+0x215/0x2e0 [lod] [1763693.492248] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763693.499650] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763693.506794] [] lod_declare_create+0x204/0x590 [lod] [1763693.513525] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763693.521550] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763693.528173] [] mdd_create+0x867/0x14a0 [mdd] [1763693.534296] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763693.540849] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763693.548521] [] mdt_reint_rec+0x83/0x210 [mdt] [1763693.554728] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763693.561443] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763693.568676] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763693.575074] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763693.582397] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763693.589028] [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] [1763693.596365] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763693.603698] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763693.610604] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763693.617918] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763693.624478] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763693.631748] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763693.639442] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763693.645752] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763693.652854] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763693.660620] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763693.667876] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763693.675750] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763693.682717] [] ? __wake_up+0x44/0x50 [1763693.688154] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763693.694642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763693.702216] [] kthread+0xd1/0xe0 [1763693.707271] [] ? insert_kthread_work+0x40/0x40 [1763693.713559] [] ret_from_fork_nospec_begin+0xe/0x21 [1763693.720175] [] ? insert_kthread_work+0x40/0x40 [1763693.726454] INFO: task mdt00_060:22474 blocked for more than 120 seconds. [1763693.733431] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763693.741435] mdt00_060 D ffff8add79eee180 0 22474 2 0x00000080 [1763693.748728] Call Trace: [1763693.751363] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763693.758241] [] schedule+0x29/0x70 [1763693.763405] [] rwsem_down_write_failed+0x225/0x3a0 [1763693.770032] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [1763693.776744] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [1763693.783554] [] call_rwsem_down_write_failed+0x17/0x30 [1763693.790445] [] down_write+0x2d/0x3d [1763693.795765] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763693.802664] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763693.809544] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763693.816171] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763693.823509] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763693.831343] [] lod_prepare_create+0x215/0x2e0 [lod] [1763693.838052] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763693.845478] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763693.852622] [] lod_declare_create+0x204/0x590 [lod] [1763693.859338] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763693.867371] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763693.873993] [] mdd_create+0x867/0x14a0 [mdd] [1763693.880099] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763693.886673] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763693.894338] [] mdt_reint_rec+0x83/0x210 [mdt] [1763693.900536] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763693.907267] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763693.914503] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763693.920885] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763693.928239] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763693.934864] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763693.942207] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763693.949111] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763693.956432] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763693.962987] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763693.970261] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763693.977953] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763693.984258] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763693.991359] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763693.999122] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763694.006379] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763694.014255] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763694.021222] [] ? __wake_up+0x44/0x50 [1763694.026661] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763694.033149] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763694.040721] [] kthread+0xd1/0xe0 [1763694.045782] [] ? insert_kthread_work+0x40/0x40 [1763694.052072] [] ret_from_fork_nospec_begin+0xe/0x21 [1763694.058687] [] ? insert_kthread_work+0x40/0x40 [1763694.065054] INFO: task mdt03_072:50099 blocked for more than 120 seconds. [1763694.072032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763694.080034] mdt03_072 D ffff8aed662b2080 0 50099 2 0x00000080 [1763694.087328] Call Trace: [1763694.089963] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763694.096841] [] schedule+0x29/0x70 [1763694.102003] [] rwsem_down_write_failed+0x225/0x3a0 [1763694.108623] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [1763694.115415] [] call_rwsem_down_write_failed+0x17/0x30 [1763694.122309] [] down_write+0x2d/0x3d [1763694.127650] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763694.134541] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763694.141445] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763694.148069] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763694.155393] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763694.163251] [] lod_prepare_create+0x215/0x2e0 [lod] [1763694.169957] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763694.177364] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763694.184529] [] lod_declare_create+0x204/0x590 [lod] [1763694.191236] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763694.199245] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763694.205883] [] mdd_create+0x867/0x14a0 [mdd] [1763694.211993] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763694.218540] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763694.226231] [] mdt_reint_rec+0x83/0x210 [mdt] [1763694.232423] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763694.239143] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763694.246395] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763694.252772] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763694.260092] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763694.266748] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763694.274085] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763694.280976] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763694.288314] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763694.294871] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763694.302128] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763694.309842] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763694.316146] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763694.323235] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763694.331005] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763694.338275] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763694.346140] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763694.353122] [] ? __wake_up+0x44/0x50 [1763694.358557] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763694.365033] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763694.372618] [] kthread+0xd1/0xe0 [1763694.377675] [] ? insert_kthread_work+0x40/0x40 [1763694.383950] [] ret_from_fork_nospec_begin+0xe/0x21 [1763694.390587] [] ? insert_kthread_work+0x40/0x40 [1763694.396872] INFO: task mdt01_080:53086 blocked for more than 120 seconds. [1763694.403838] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763694.411861] mdt01_080 D ffff8afaa33a2080 0 53086 2 0x00000080 [1763694.419132] Call Trace: [1763694.421768] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763694.428646] [] schedule+0x29/0x70 [1763694.433809] [] rwsem_down_write_failed+0x225/0x3a0 [1763694.440430] [] call_rwsem_down_write_failed+0x17/0x30 [1763694.447313] [] down_write+0x2d/0x3d [1763694.452658] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763694.459541] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763694.466429] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763694.473069] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763694.480385] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763694.488226] [] lod_prepare_create+0x215/0x2e0 [lod] [1763694.494951] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763694.502353] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763694.509497] [] lod_declare_create+0x204/0x590 [lod] [1763694.516230] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763694.524238] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763694.530870] [] mdd_create+0x867/0x14a0 [mdd] [1763694.537014] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763694.543568] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763694.551239] [] mdt_reint_rec+0x83/0x210 [mdt] [1763694.557450] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763694.564161] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763694.571399] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763694.577791] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763694.585109] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763694.591735] [] ? cfs_hash_bd_add_locked+0x24/0x80 [libcfs] [1763694.599068] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763694.606403] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763694.613320] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763694.620639] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763694.627195] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763694.634467] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763694.642161] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763694.648470] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763694.655575] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763694.663322] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763694.670578] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763694.678463] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763694.685428] [] ? __wake_up+0x44/0x50 [1763694.690865] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763694.697353] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763694.704928] [] kthread+0xd1/0xe0 [1763694.709990] [] ? insert_kthread_work+0x40/0x40 [1763694.716281] [] ret_from_fork_nospec_begin+0xe/0x21 [1763694.722901] [] ? insert_kthread_work+0x40/0x40 [1763694.729185] INFO: task mdt03_084:68611 blocked for more than 120 seconds. [1763694.736159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763694.744162] mdt03_084 D ffff8afd6d425140 0 68611 2 0x00000080 [1763694.751447] Call Trace: [1763694.754081] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763694.760963] [] schedule+0x29/0x70 [1763694.766127] [] rwsem_down_write_failed+0x225/0x3a0 [1763694.772749] [] call_rwsem_down_write_failed+0x17/0x30 [1763694.779630] [] down_write+0x2d/0x3d [1763694.784972] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763694.791855] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763694.798743] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763694.805397] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763694.812719] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763694.820561] [] lod_prepare_create+0x215/0x2e0 [lod] [1763694.827293] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763694.834696] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763694.841837] [] lod_declare_create+0x204/0x590 [lod] [1763694.848569] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763694.856577] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763694.863200] [] mdd_create+0x867/0x14a0 [mdd] [1763694.869323] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763694.875889] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763694.883557] [] mdt_reint_rec+0x83/0x210 [mdt] [1763694.889766] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763694.896476] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763694.903716] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763694.910106] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763694.917424] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763694.924056] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763694.931411] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763694.938298] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763694.945614] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763694.952185] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763694.959434] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763694.967128] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763694.973450] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763694.980534] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763694.988282] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763694.995547] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763695.003414] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763695.010376] [] ? __wake_up+0x44/0x50 [1763695.015842] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763695.022314] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763695.029887] [] kthread+0xd1/0xe0 [1763695.034966] [] ? insert_kthread_work+0x40/0x40 [1763695.041239] [] ret_from_fork_nospec_begin+0xe/0x21 [1763695.047862] [] ? insert_kthread_work+0x40/0x40 [1763695.054168] INFO: task mdt00_088:28972 blocked for more than 120 seconds. [1763695.061129] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763695.069139] mdt00_088 D ffff8add3be530c0 0 28972 2 0x00000080 [1763695.076431] Call Trace: [1763695.079066] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763695.085945] [] schedule+0x29/0x70 [1763695.091108] [] rwsem_down_write_failed+0x225/0x3a0 [1763695.097729] [] call_rwsem_down_write_failed+0x17/0x30 [1763695.104602] [] down_write+0x2d/0x3d [1763695.109949] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763695.116834] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763695.123719] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763695.130360] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763695.137683] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763695.145520] [] lod_prepare_create+0x215/0x2e0 [lod] [1763695.152248] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763695.159652] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763695.166798] [] lod_declare_create+0x204/0x590 [lod] [1763695.173528] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763695.181536] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763695.188174] [] mdd_create+0x867/0x14a0 [mdd] [1763695.194282] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763695.200833] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763695.208520] [] mdt_reint_rec+0x83/0x210 [mdt] [1763695.214732] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763695.221442] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763695.228693] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763695.235065] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763695.242393] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763695.249036] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763695.256369] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763695.263260] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763695.270595] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763695.277152] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763695.284409] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763695.292117] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763695.298423] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763695.305524] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763695.313272] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763695.320543] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763695.328403] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763695.335370] [] ? __wake_up+0x44/0x50 [1763695.340822] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763695.347295] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763695.354870] [] kthread+0xd1/0xe0 [1763695.359948] [] ? insert_kthread_work+0x40/0x40 [1763695.366224] [] ret_from_fork_nospec_begin+0xe/0x21 [1763695.372845] [] ? insert_kthread_work+0x40/0x40 [1763695.379142] INFO: task mdt00_108:80232 blocked for more than 120 seconds. [1763695.386104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1763695.394105] mdt00_108 D ffff8aed677ae180 0 80232 2 0x00000080 [1763695.401398] Call Trace: [1763695.404033] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1763695.410912] [] schedule+0x29/0x70 [1763695.416089] [] rwsem_down_write_failed+0x225/0x3a0 [1763695.422704] [] call_rwsem_down_write_failed+0x17/0x30 [1763695.429583] [] down_write+0x2d/0x3d [1763695.434920] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1763695.441808] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1763695.448708] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1763695.455337] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1763695.462661] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1763695.470517] [] lod_prepare_create+0x215/0x2e0 [lod] [1763695.477226] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1763695.484627] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1763695.491787] [] lod_declare_create+0x204/0x590 [lod] [1763695.498497] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1763695.506513] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1763695.513162] [] mdd_create+0x867/0x14a0 [mdd] [1763695.519275] [] mdt_reint_open+0x224f/0x3240 [mdt] [1763695.525827] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1763695.533513] [] mdt_reint_rec+0x83/0x210 [mdt] [1763695.539710] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1763695.546427] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1763695.553694] [] mdt_intent_open+0x82/0x3a0 [mdt] [1763695.560069] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1763695.567393] [] mdt_intent_policy+0x435/0xd80 [mdt] [1763695.574037] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1763695.581368] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1763695.588251] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1763695.595589] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1763695.602146] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1763695.609403] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1763695.617147] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1763695.623455] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1763695.630551] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1763695.638318] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1763695.645585] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1763695.653451] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1763695.660431] [] ? __wake_up+0x44/0x50 [1763695.665868] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1763695.672343] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1763695.679930] [] kthread+0xd1/0xe0 [1763695.684991] [] ? insert_kthread_work+0x40/0x40 [1763695.691263] [] ret_from_fork_nospec_begin+0xe/0x21 [1763695.697897] [] ? insert_kthread_work+0x40/0x40 [1764040.117321] Lustre: 21914:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315393/real 1586315393] req@ffff8b0d59bc9b00 x1661612394054784/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586315994 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1764040.145872] Lustre: fir-OST0034-osc-MDT0001: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764040.162364] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764195.216875] Lustre: 22155:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315393/real 1586315393] req@ffff8adfc66df080 x1661612396442304/t0(0) o5->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1586316149 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1764195.245245] Lustre: 22155:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1764195.255172] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1764306.173147] Lustre: 21910:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315503/real 1586315503] req@ffff8af06b338900 x1661612396592704/t0(0) o6->fir-OST0032-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1586316259 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1764306.201444] Lustre: fir-OST0032-osc-MDT0001: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764306.217876] Lustre: fir-OST0032-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764602.389917] Lustre: 21898:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315800/real 1586315800] req@ffff8ae4c5606780 x1661612396993216/t0(0) o6->fir-OST0036-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1586316556 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1764602.418206] Lustre: fir-OST0036-osc-MDT0001: Connection to fir-OST0036 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764602.434674] Lustre: fir-OST0036-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764641.087062] Lustre: 21913:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586315994/real 1586315994] req@ffff8b0d59bcc800 x1661612394054208/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586316595 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1764641.115617] Lustre: fir-OST0034-osc-MDT0001: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764641.132098] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764711.866173] Lustre: 21910:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586316008/real 1586316008] req@ffff8af93c3a5a00 x1661612397419328/t0(0) o6->fir-OST003a-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1586316665 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1764711.894470] Lustre: fir-OST003a-osc-MDT0001: Connection to fir-OST003a (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1764711.910879] Lustre: fir-OST003a-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1764952.290313] Lustre: 22155:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586316150/real 1586316150] req@ffff8ae9df3ad580 x1661612397603200/t0(0) o5->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 432/432 e 0 to 1 dl 1586316906 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1764952.318728] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1765062.403563] Lustre: fir-OST0032-osc-MDT0001: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1765062.420083] Lustre: fir-OST0032-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1765173.619585] Lustre: fir-MDT0001: Connection restored to 9ed38912-482b-4 (at 10.50.1.57@o2ib2) [1765224.003367] Lustre: fir-MDT0001: haven't heard from client 9ed38912-482b-4 (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6f890c00, cur 1586317178 expire 1586317028 last 1586316951 [1765242.432815] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586316595/real 1586316595] req@ffff8b0d67d18000 x1661612394053632/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586317196 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1765242.440821] Lustre: fir-OST0034-osc-MDT0001: Connection to fir-OST0034 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1765242.441047] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1765242.487594] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [1765358.836203] Lustre: fir-OST0036-osc-MDT0001: Connection to fir-OST0036 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1765532.139756] Lustre: fir-MDT0001: Connection restored to 78da859a-079c-4 (at 10.50.4.27@o2ib2) [1765532.148457] Lustre: Skipped 2 previous similar messages [1765590.009358] Lustre: fir-MDT0001: haven't heard from client 2147e45e-5aee-4 (at 10.50.4.29@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6688b800, cur 1586317544 expire 1586317394 last 1586317317 [1765709.353392] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1765818.617432] Lustre: 21910:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586317016/real 1586317016] req@ffff8af06b338900 x1661612396592704/t0(0) o6->fir-OST0032-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 0 to 1 dl 1586317772 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1765818.633435] Lustre: fir-OST0032-osc-MDT0001: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1765818.633437] Lustre: Skipped 1 previous similar message [1765818.633656] Lustre: fir-OST0032-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1765818.633657] Lustre: Skipped 2 previous similar messages [1765818.682830] Lustre: 21910:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [1766114.585586] Lustre: fir-OST0036-osc-MDT0001: Connection to fir-OST0036 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1766114.601827] Lustre: Skipped 2 previous similar messages [1766254.035255] Lustre: fir-MDT0001: haven't heard from client 81fc576f-a47a-4 (at 10.50.10.31@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786bcc00, cur 1586318208 expire 1586318058 last 1586317981 [1766254.055399] Lustre: Skipped 2 previous similar messages [1766351.032447] Lustre: fir-OST0038-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1766351.042454] Lustre: Skipped 5 previous similar messages [1766444.066804] Lustre: 21923:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586317797/real 1586317797] req@ffff8b0d59bcd100 x1661612394054016/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586318398 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1766444.095353] Lustre: 21923:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [1766466.387438] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1766682.068392] Lustre: fir-MDT0001: haven't heard from client 376f46a0-8125-4 (at 10.50.5.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73f17400, cur 1586318636 expire 1586318486 last 1586318409 [1766682.321461] Lustre: fir-OST003a-osc-MDT0001: Connection to fir-OST003a (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1766682.337704] Lustre: Skipped 4 previous similar messages [1767045.307446] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586318398/real 1586318398] req@ffff8b0d67d18000 x1661612394053632/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586318999 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1767045.336000] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [1767045.346177] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1767045.356219] Lustre: Skipped 6 previous similar messages [1767223.421355] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1767330.139285] Lustre: fir-OST0032-osc-MDT0001: Connection to fir-OST0032 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1767330.155523] Lustre: Skipped 4 previous similar messages [1767341.075010] Lustre: fir-MDT0001: haven't heard from client d2c2a701-05d0-4 (at 10.50.2.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d4b429800, cur 1586319295 expire 1586319145 last 1586319068 [1767646.116041] Lustre: 21919:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586318999/real 1586318999] req@ffff8b0d67d1a400 x1661612394053568/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586319600 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1767646.144596] Lustre: 21919:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [1767646.154753] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1767646.164769] Lustre: Skipped 7 previous similar messages [1767650.092518] Lustre: fir-MDT0001: haven't heard from client c385efcd-dc92-4 (at 10.50.2.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d54a85800, cur 1586319604 expire 1586319454 last 1586319377 [1767980.455468] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1767996.461914] Lustre: fir-OST003a-osc-MDT0001: Connection to fir-OST003a (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1767996.478157] Lustre: Skipped 5 previous similar messages [1768247.277022] Lustre: 21920:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586319600/real 1586319600] req@ffff8b0d67d1da00 x1661612394053824/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586320201 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1768247.305570] Lustre: 21920:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [1768247.315696] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1768247.325749] Lustre: Skipped 4 previous similar messages [1768619.743566] Lustre: fir-OST0038-osc-MDT0001: Connection to fir-OST0038 (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1768619.759810] Lustre: Skipped 4 previous similar messages [1768737.489892] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1768761.097515] Lustre: fir-MDT0001: haven't heard from client a95eaf77-b4b8-4 (at 10.50.2.59@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d4c0ac800, cur 1586320715 expire 1586320565 last 1586320488 [1768761.117567] Lustre: Skipped 1 previous similar message [1768848.254076] Lustre: 21920:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586320201/real 1586320201] req@ffff8b0d67d1da00 x1661612394053824/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586320802 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1768848.282627] Lustre: 21920:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [1768848.292778] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1768848.302805] Lustre: Skipped 6 previous similar messages [1769138.774367] LustreError: 22159:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0036-osc-MDT0001: cannot cleanup orphans: rc = -11 [1769193.383918] LustreError: 22147:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0030-osc-MDT0001: cannot cleanup orphans: rc = -11 [1769311.371265] Lustre: fir-OST003a-osc-MDT0001: Connection to fir-OST003a (at 10.0.10.109@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1769311.387504] Lustre: Skipped 5 previous similar messages [1769375.786097] LustreError: 22163:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0038-osc-MDT0001: cannot cleanup orphans: rc = -107 [1769409.950067] LustreError: 22167:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST003a-osc-MDT0001: cannot cleanup orphans: rc = -107 [1769449.231178] Lustre: 21919:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586320802/real 1586320802] req@ffff8b0d67d1a400 x1661612394053568/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586321403 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1769449.259724] Lustre: 21919:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [1769449.269862] Lustre: fir-OST0034-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1769449.279889] Lustre: Skipped 4 previous similar messages [1769494.524479] LustreError: 22155:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0034-osc-MDT0001: cannot cleanup orphans: rc = -107 [1769579.901896] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [1769579.912074] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (5): c: 0, oc: 0, rc: 8 [1769579.924518] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769584.789205] Lustre: fir-MDT0001: Client 5ab066c6-3005-4 (at 10.50.13.9@o2ib2) reconnecting [1769584.797645] Lustre: Skipped 2 previous similar messages [1769588.599351] Lustre: fir-MDT0001: Client 5ab066c6-3005-4 (at 10.50.13.9@o2ib2) reconnecting [1769595.617547] Lustre: fir-MDT0001: Client 5ab066c6-3005-4 (at 10.50.13.9@o2ib2) reconnecting [1769598.227854] LNetError: 98876:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769598.237562] Lustre: fir-MDT0001: Client d7c9b82a-5f48-4 (at 10.49.18.25@o2ib1) reconnecting [1769598.251428] LustreError: 22151:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0032-osc-MDT0001: cannot cleanup orphans: rc = -11 [1769606.590231] Lustre: fir-MDT0001: Client d3d661a4-8549-4 (at 10.50.17.23@o2ib2) reconnecting [1769615.362897] Lustre: fir-MDT0001: Client fe333c8b-4ed9-4 (at 10.50.9.66@o2ib2) reconnecting [1769615.371337] Lustre: Skipped 2 previous similar messages [1769631.873616] Lustre: fir-MDT0001: Client 541f81d4-bd4f-4 (at 10.50.7.3@o2ib2) reconnecting [1769631.881969] Lustre: Skipped 5 previous similar messages [1769638.271682] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.25.2@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769638.289141] LustreError: Skipped 18 previous similar messages [1769649.903894] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 10 seconds [1769649.914341] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769650.903927] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 11 seconds [1769650.914354] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 4 previous similar messages [1769652.903977] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 68 seconds [1769652.914403] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 3 previous similar messages [1769653.557494] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.2.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769653.574951] LustreError: Skipped 1 previous similar message [1769656.904087] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 1 seconds [1769656.914428] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 1 previous similar message [1769662.384749] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.17.26@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769662.402297] LustreError: Skipped 2 previous similar messages [1769663.912297] Lustre: fir-MDT0001: Client d0a45662-de66-4 (at 10.50.10.6@o2ib2) reconnecting [1769663.920737] Lustre: Skipped 26 previous similar messages [1769667.904418] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 0 seconds [1769667.914761] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 5 previous similar messages [1769682.955840] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.17.4@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769682.973293] LustreError: Skipped 6 previous similar messages [1769700.905373] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 1 seconds [1769700.915733] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769715.344536] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.7.29@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769715.361997] LustreError: Skipped 44 previous similar messages [1769732.694439] Lustre: fir-MDT0001: Client a7b4e450-17cf-4 (at 10.50.8.18@o2ib2) reconnecting [1769732.702882] Lustre: Skipped 78 previous similar messages [1769742.123910] Lustre: fir-MDT0001: haven't heard from client fir-MDT0001-lwp-OST0038_UUID (at 10.0.10.109@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd20b0a800, cur 1586321696 expire 1586321546 last 1586321469 [1769749.906791] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 9 seconds [1769749.917135] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 17 previous similar messages [1769749.926711] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769749.938807] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [1769779.845296] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.6.22@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1769779.862755] LustreError: Skipped 211 previous similar messages [1769801.908294] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769801.920375] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [1769813.908627] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.109@o2ib7: 0 seconds [1769813.918967] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 32 previous similar messages [1769895.808974] LustreError: 22159:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0036-osc-MDT0001: cannot cleanup orphans: rc = -11 [1769931.911986] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [1769931.922163] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (6): c: 0, oc: 0, rc: 8 [1769931.934597] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1769931.946698] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 7 previous similar messages [1769939.139669] Lustre: fir-MDT0001: haven't heard from client 322961a3-22bb-4 (at 10.50.2.60@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d658be400, cur 1586321893 expire 1586321743 last 1586321666 [1769939.159728] Lustre: Skipped 5 previous similar messages [1770005.914096] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [1770005.924268] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.109@o2ib7 (5): c: 0, oc: 0, rc: 8 [1770050.232355] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586321403/real 1586321403] req@ffff8b0d67d18000 x1661612394053632/t0(0) o6->fir-OST0034-osc-MDT0001@10.0.10.109@o2ib7:28/4 lens 544/432 e 23 to 1 dl 1586322004 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 [1770050.260906] Lustre: 21918:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [1770101.431579] Lustre: fir-MDT0001: Connection restored to fir-MDT0001-lwp-OST0036_UUID (at 10.0.10.109@o2ib7) [1770101.441496] Lustre: Skipped 182 previous similar messages [1770132.820728] LustreError: 22163:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) fir-OST0038-osc-MDT0001: cannot cleanup orphans: rc = -11 [1770132.833854] LustreError: 22163:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [1770295.892498] INFO: task mdt01_002:21985 blocked for more than 120 seconds. [1770295.899462] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1770295.907464] mdt01_002 D ffff8add6026c100 0 21985 2 0x00000080 [1770295.914740] Call Trace: [1770295.917382] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1770295.924278] [] schedule+0x29/0x70 [1770295.929440] [] rwsem_down_write_failed+0x225/0x3a0 [1770295.936058] [] call_rwsem_down_write_failed+0x17/0x30 [1770295.942953] [] down_write+0x2d/0x3d [1770295.948280] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1770295.955164] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1770295.962087] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1770295.968720] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1770295.976045] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1770295.983917] [] lod_prepare_create+0x215/0x2e0 [lod] [1770295.990629] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1770295.998031] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1770296.005189] [] lod_declare_create+0x204/0x590 [lod] [1770296.011902] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1770296.019915] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1770296.026551] [] mdd_create+0x867/0x14a0 [mdd] [1770296.032675] [] mdt_reint_open+0x224f/0x3240 [mdt] [1770296.039236] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1770296.046923] [] mdt_reint_rec+0x83/0x210 [mdt] [1770296.053120] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1770296.059836] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1770296.067089] [] mdt_intent_open+0x82/0x3a0 [mdt] [1770296.073458] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1770296.080776] [] mdt_intent_policy+0x435/0xd80 [mdt] [1770296.087421] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1770296.094758] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1770296.101643] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1770296.108989] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1770296.115550] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1770296.122809] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1770296.130526] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1770296.136834] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1770296.143921] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1770296.151683] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1770296.158940] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1770296.166799] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1770296.173795] [] ? __wake_up+0x44/0x50 [1770296.179228] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1770296.185703] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1770296.193290] [] kthread+0xd1/0xe0 [1770296.198350] [] ? insert_kthread_work+0x40/0x40 [1770296.204619] [] ret_from_fork_nospec_begin+0xe/0x21 [1770296.211253] [] ? insert_kthread_work+0x40/0x40 [1770296.217622] INFO: task mdt01_038:41318 blocked for more than 120 seconds. [1770296.224586] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1770296.232609] mdt01_038 D ffff8aed6ef6b0c0 0 41318 2 0x00000080 [1770296.239896] Call Trace: [1770296.242532] [] ? lquota_disk_read+0xf2/0x390 [lquota] [1770296.249410] [] schedule+0x29/0x70 [1770296.254572] [] rwsem_down_write_failed+0x225/0x3a0 [1770296.261191] [] ? cfs_hash_lookup+0xa2/0xd0 [libcfs] [1770296.267905] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [1770296.274713] [] call_rwsem_down_write_failed+0x17/0x30 [1770296.281589] [] down_write+0x2d/0x3d [1770296.286920] [] lod_qos_statfs_update+0x97/0x2b0 [lod] [1770296.293822] [] lod_qos_prep_create+0x16a/0x1890 [lod] [1770296.300718] [] ? qsd_op_begin+0x262/0x4b0 [lquota] [1770296.307348] [] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs] [1770296.314686] [] ? osd_declare_inode_qid+0x27b/0x440 [osd_ldiskfs] [1770296.322520] [] lod_prepare_create+0x215/0x2e0 [lod] [1770296.329229] [] lod_declare_striped_create+0x1ee/0x980 [lod] [1770296.336652] [] ? lod_sub_declare_create+0xdf/0x210 [lod] [1770296.343796] [] lod_declare_create+0x204/0x590 [lod] [1770296.350507] [] mdd_declare_create_object_internal+0xea/0x360 [mdd] [1770296.358534] [] mdd_declare_create+0x4c/0xdf0 [mdd] [1770296.365158] [] mdd_create+0x867/0x14a0 [mdd] [1770296.371269] [] mdt_reint_open+0x224f/0x3240 [mdt] [1770296.377832] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [1770296.385497] [] mdt_reint_rec+0x83/0x210 [mdt] [1770296.391694] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1770296.398424] [] ? mdt_intent_fixup_resent+0x36/0x220 [mdt] [1770296.405653] [] mdt_intent_open+0x82/0x3a0 [mdt] [1770296.412024] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [1770296.419365] [] mdt_intent_policy+0x435/0xd80 [mdt] [1770296.425985] [] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [1770296.433320] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [1770296.440237] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [1770296.447555] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [1770296.454113] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [1770296.461384] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [1770296.469078] [] tgt_enqueue+0x62/0x210 [ptlrpc] [1770296.475387] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1770296.482482] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1770296.490230] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1770296.497484] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1770296.505359] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1770296.512343] [] ? __wake_up+0x44/0x50 [1770296.517774] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1770296.524265] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1770296.531836] [] kthread+0xd1/0xe0 [1770296.536889] [] ? insert_kthread_work+0x40/0x40 [1770296.543180] [] ret_from_fork_nospec_begin+0xe/0x21 [1770296.549795] [] ? insert_kthread_work+0x40/0x40 [1770352.138432] Lustre: fir-MDT0001: haven't heard from client 605a3fcc-3131-4 (at 10.50.2.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add7961f400, cur 1586322306 expire 1586322156 last 1586322079 [1770976.660567] Lustre: fir-OST003a-osc-MDT0001: Connection restored to 10.0.10.109@o2ib7 (at 10.0.10.109@o2ib7) [1770976.670565] Lustre: Skipped 6 previous similar messages [1771183.162045] Lustre: fir-MDT0001: haven't heard from client c68ee752-f14f-4 (at 10.50.1.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d70108400, cur 1586323137 expire 1586322987 last 1586322910 [1771259.165155] Lustre: fir-MDT0001: haven't heard from client 844e77f6-3f22-4 (at 10.50.2.31@o2ib2) in 163 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6dc59c00, cur 1586323213 expire 1586323063 last 1586323050 [1771568.172915] Lustre: fir-MDT0001: haven't heard from client 89cdc238-a9fc-4 (at 10.50.2.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d75ff4800, cur 1586323522 expire 1586323372 last 1586323295 [1772175.771336] Lustre: fir-MDT0001: Connection restored to 21098921-f7b5-4 (at 10.50.4.6@o2ib2) [1772175.779965] Lustre: Skipped 8 previous similar messages [1773518.334263] Lustre: fir-MDT0001: Connection restored to 522bdb54-ddbf-4 (at 10.50.2.58@o2ib2) [1773570.230830] Lustre: fir-MDT0001: haven't heard from client 522bdb54-ddbf-4 (at 10.50.2.58@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d5b649800, cur 1586325524 expire 1586325374 last 1586325297 [1781701.221752] Lustre: fir-MDT0001: Connection restored to 48a86d34-282f-4 (at 10.50.5.38@o2ib2) [1781763.461644] Lustre: fir-MDT0001: haven't heard from client 18e71bec-a099-4 (at 10.50.5.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d2de3f400, cur 1586333717 expire 1586333567 last 1586333490 [1782757.183275] Lustre: fir-MDT0001: Connection restored to 00c7f158-cc8b-4 (at 10.50.4.48@o2ib2) [1782824.490259] Lustre: fir-MDT0001: haven't heard from client 00c7f158-cc8b-4 (at 10.50.4.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d73e20c00, cur 1586334778 expire 1586334628 last 1586334551 [1783330.629930] Lustre: fir-MDT0001: Connection restored to 9fe5415f-280d-4 (at 10.50.5.48@o2ib2) [1783386.506745] Lustre: fir-MDT0001: haven't heard from client 9fe5415f-280d-4 (at 10.50.5.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786b8400, cur 1586335340 expire 1586335190 last 1586335113 [1789811.624011] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1789855.690163] Lustre: fir-MDT0001: haven't heard from client 9b444a4b-8a4b-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc39e78c00, cur 1586341809 expire 1586341659 last 1586341582 [1790836.669078] Lustre: fir-MDT0001: Connection restored to 632f4069-9f90-4 (at 10.50.2.10@o2ib2) [1790905.720707] Lustre: fir-MDT0001: haven't heard from client 632f4069-9f90-4 (at 10.50.2.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d786bf000, cur 1586342859 expire 1586342709 last 1586342632 [1802688.397001] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [1802740.262394] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1802740.279854] LustreError: Skipped 3 previous similar messages [1802760.056768] Lustre: fir-MDT0001: haven't heard from client f3629f90-ec37-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea02692800, cur 1586354713 expire 1586354563 last 1586354486 [1802840.617272] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.14.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1811487.312798] Lustre: fir-MDT0001: haven't heard from client 14f6955a-e3d1-4 (at 10.49.8.32@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b336400, cur 1586363440 expire 1586363290 last 1586363213 [1812678.034599] Lustre: 21899:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586364622/real 1586364622] req@ffff8afb52697080 x1661613710676288/t0(0) o41->fir-MDT0002-osp-MDT0001@10.0.10.53@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1586364629 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1812678.062889] Lustre: 21899:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [1812678.072899] Lustre: fir-MDT0002-osp-MDT0001: Connection to fir-MDT0002 (at 10.0.10.53@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1812678.089072] Lustre: Skipped 8 previous similar messages [1812700.854048] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.27.11@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812702.478407] Lustre: fir-MDT0001: Client da346c94-b71d-4 (at 10.50.10.27@o2ib2) reconnecting [1812702.486938] Lustre: Skipped 61 previous similar messages [1812702.492457] Lustre: fir-MDT0001: Connection restored to da346c94-b71d-4 (at 10.50.10.27@o2ib2) [1812705.835225] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.5.15@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812705.852681] LustreError: Skipped 2 previous similar messages [1812718.281873] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.10.27@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812718.299419] LustreError: Skipped 11 previous similar messages [1812735.992882] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.6.68@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812736.010342] LustreError: Skipped 11 previous similar messages [1812769.700324] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.9.60@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812769.717777] LustreError: Skipped 19 previous similar messages [1812784.120541] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1812784.130883] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.53@o2ib7 (106): c: 4, oc: 0, rc: 8 [1812784.143380] LNetError: 21869:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.53@o2ib7 added to recovery queue. Health = 900 [1812785.129013] LNetError: 110146:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1812785.141180] LNetError: 110146:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [1812793.544177] Lustre: fir-MDT0001: Client da346c94-b71d-4 (at 10.50.10.27@o2ib2) reconnecting [1812793.552728] Lustre: fir-MDT0001: Connection restored to da346c94-b71d-4 (at 10.50.10.27@o2ib2) [1812830.200540] Lustre: fir-MDT0001: Client 21a490cd-650d-4 (at 10.50.7.10@o2ib2) reconnecting [1812830.209004] Lustre: fir-MDT0001: Connection restored to 21a490cd-650d-4 (at 10.50.7.10@o2ib2) [1812833.735792] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.5.39@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812833.753265] LustreError: Skipped 223 previous similar messages [1812838.134611] LNetError: 110146:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1812838.146781] LNetError: 110146:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [1812849.986167] Lustre: fir-MDT0001: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1812884.123315] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.53@o2ib7: 0 seconds [1812884.133569] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 2 previous similar messages [1812884.143061] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1812940.265216] Lustre: fir-MDT0001: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [1812940.273660] Lustre: Skipped 1 previous similar message [1812940.279004] Lustre: fir-MDT0001: Connection restored to 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) [1812944.124990] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.53@o2ib7: 0 seconds [1812944.135244] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 3 previous similar messages [1812961.855195] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.8.44@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1812961.872649] LustreError: Skipped 1608 previous similar messages [1812985.126105] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.53@o2ib7: 1 seconds [1812985.136378] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1812985.148468] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [1813218.273277] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.8.10@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1813218.290728] LustreError: Skipped 3087 previous similar messages [1813309.059317] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.53@o2ib7, removing former export from same NID [1813309.070373] Lustre: fir-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [1813359.851191] Lustre: fir-MDT0002-osp-MDT0001: Connection restored to 10.0.10.53@o2ib7 (at 10.0.10.53@o2ib7) [1826236.177462] LNet: Service thread pid 42202 was inactive for 200.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [1826236.194573] Pid: 42202, comm: mdt01_045 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [1826236.204919] Call Trace: [1826236.207563] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [1826236.214674] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [1826236.221956] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [1826236.228833] [] osp_md_object_lock+0x162/0x2d0 [osp] [1826236.235577] [] lod_object_lock+0xf4/0x780 [lod] [1826236.241990] [] mdd_object_lock+0x3e/0xe0 [mdd] [1826236.248293] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [1826236.255743] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [1826236.262654] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [1826236.269063] [] mdt_reint_rename+0x2c5/0x2b90 [mdt] [1826236.275714] [] mdt_reint_rec+0x83/0x210 [mdt] [1826236.281930] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [1826236.288662] [] mdt_reint+0x67/0x140 [mdt] [1826236.294560] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [1826236.301681] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1826236.309568] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [1826236.316069] [] kthread+0xd1/0xe0 [1826236.321171] [] ret_from_fork_nospec_begin+0xe/0x21 [1826236.327820] [] 0xffffffffffffffff [1826236.333027] LustreError: dumping log to /tmp/lustre-log.1586378188.42202 [1826238.148617] LNet: Service thread pid 42202 completed after 202.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [1828695.778768] Lustre: fir-MDT0001: haven't heard from client 0ae7c2ea-59d3-4 (at 10.50.7.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5d951c00, cur 1586380648 expire 1586380498 last 1586380421 [1828788.765510] Lustre: fir-MDT0001: Connection restored to 0ae7c2ea-59d3-4 (at 10.50.7.28@o2ib2) [1842166.174452] Lustre: fir-MDT0001: haven't heard from client 83859ea9-b080-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af467739000, cur 1586394118 expire 1586393968 last 1586393891 [1872635.022632] Lustre: fir-MDT0001: haven't heard from client 64e3e3ad-807a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a85228400, cur 1586424586 expire 1586424436 last 1586424359 [1872676.815268] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1872728.540539] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.37@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1872728.557997] LustreError: Skipped 1412 previous similar messages [1872828.894563] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.9.37@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1877175.146734] Lustre: fir-MDT0001: haven't heard from client 4e76023f-67af-4 (at 10.50.9.42@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6a2bb000, cur 1586429126 expire 1586428976 last 1586428899 [1877212.973579] Lustre: fir-MDT0001: Connection restored to 4e76023f-67af-4 (at 10.50.9.42@o2ib2) [1881156.038932] Lustre: 28973:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881156.050759] Lustre: 28973:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1070 previous similar messages [1881157.038358] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881157.050188] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 379 previous similar messages [1881159.039908] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881159.051729] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 654 previous similar messages [1881163.040575] Lustre: 22483:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881163.052402] Lustre: 22483:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1027 previous similar messages [1881171.041841] Lustre: 22343:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881171.053670] Lustre: 22343:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2752 previous similar messages [1881187.045557] Lustre: 22425:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881187.057380] Lustre: 22425:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 5057 previous similar messages [1881219.047386] Lustre: 22425:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881219.059214] Lustre: 22425:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 9381 previous similar messages [1881283.089634] Lustre: 22482:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881283.101464] Lustre: 22482:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 9056 previous similar messages [1881411.115082] Lustre: 22483:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [1881411.126906] Lustre: 22483:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 13606 previous similar messages [1889083.425310] Lustre: 21919:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586441026/real 1586441026] req@ffff8b0cfc6a1200 x1661614146416768/t0(0) o41->fir-MDT0003-osp-MDT0001@10.0.10.54@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1586441033 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [1889083.453607] Lustre: fir-MDT0003-osp-MDT0001: Connection to fir-MDT0003 (at 10.0.10.54@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [1889122.638892] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.4.15@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889142.825525] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.17.25@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889142.843069] LustreError: Skipped 64 previous similar messages [1889180.589966] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.7.37@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889180.607420] LustreError: Skipped 7 previous similar messages [1889189.270237] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1889189.280582] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.54@o2ib7 (106): c: 5, oc: 0, rc: 8 [1889189.293036] LNetError: 21866:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.54@o2ib7 added to recovery queue. Health = 900 [1889189.486643] LNetError: 4196:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1889189.498643] LNetError: 4196:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [1889245.247898] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.13.9@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889245.265356] LustreError: Skipped 162 previous similar messages [1889245.271810] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 0 seconds [1889245.282070] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 3 previous similar messages [1889245.291559] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1889245.303658] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [1889290.273056] LNet: 21855:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.10.54@o2ib7: 1 seconds [1889290.283327] LNetError: 21855:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [1889321.492343] Lustre: fir-MDT0001: haven't heard from client b54a7599-582a-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afada307000, cur 1586441272 expire 1586441122 last 1586441045 [1889373.258320] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.3.54@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889373.275778] LustreError: Skipped 730 previous similar messages [1889389.517849] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1889610.026822] Lustre: fir-MDT0001: Received new LWP connection from 10.0.10.54@o2ib7, removing former export from same NID [1889610.037889] Lustre: fir-MDT0001: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [1889631.373478] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.103@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [1889631.391033] LustreError: Skipped 3614 previous similar messages [1889990.056801] Lustre: fir-MDT0003-osp-MDT0001: Connection restored to 10.0.10.54@o2ib7 (at 10.0.10.54@o2ib7) [1895204.660225] Lustre: fir-MDT0001: haven't heard from client 939da034-317d-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d64680400, cur 1586447155 expire 1586447005 last 1586446928 [1895250.527823] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [1956495.386615] Lustre: fir-MDT0001: haven't heard from client fce251b3-293e-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae71ab64800, cur 1586508444 expire 1586508294 last 1586508217 [1956524.601449] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [1971011.869091] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1971083.798009] Lustre: fir-MDT0001: haven't heard from client c68d5354-8b09-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad4f6dd5800, cur 1586523032 expire 1586522882 last 1586522805 [1972536.670271] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1972594.827679] Lustre: fir-MDT0001: haven't heard from client 9e8b099d-def7-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b086f2a2800, cur 1586524543 expire 1586524393 last 1586524316 [1979709.027221] Lustre: fir-MDT0001: haven't heard from client 552c625f-92e8-4 (at 10.49.29.7@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af69251a000, cur 1586531657 expire 1586531507 last 1586531430 [1979759.034684] Lustre: fir-MDT0001: Connection restored to 5d22cf6d-6c39-4 (at 10.49.29.7@o2ib1) [1985110.742000] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [1985181.177255] Lustre: fir-MDT0001: haven't heard from client 3ebaba32-426d-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acf4a428800, cur 1586537129 expire 1586536979 last 1586536902 [2013285.206456] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.10.1@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2013285.223908] LustreError: Skipped 18 previous similar messages [2013308.500324] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2013308.508793] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2013349.679810] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2013349.697273] LustreError: Skipped 219 previous similar messages [2013399.859120] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2013399.867587] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2013454.753798] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2013454.762257] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2013479.605079] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2013479.622536] LustreError: Skipped 43 previous similar messages [2013554.868905] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2013554.877366] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2013609.984402] Lustre: 21899:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586565550/real 1586565550] req@ffff8aeca56d0480 x1661617143581312/t0(0) o103->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:17/18 lens 328/224 e 0 to 1 dl 1586565557 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [2013610.012873] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [2013611.168435] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586565551/real 1586565552] req@ffff8add62255e80 x1661617143778880/t0(0) o103->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:17/18 lens 328/224 e 0 to 1 dl 1586565558 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [2013611.196898] Lustre: 21878:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 696 previous similar messages [2013611.207586] Lustre: fir-MDT0000-lwp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [2013618.000634] Lustre: 21913:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586565558/real 1586565558] req@ffff8afe546a7500 x1661617143792256/t0(0) o103->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:17/18 lens 328/224 e 0 to 1 dl 1586565565 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [2013618.000641] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [2013618.045242] Lustre: 21913:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 213 previous similar messages [2013625.008832] Lustre: 21898:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586565565/real 1586565565] req@ffff8ae77e3f4380 x1661617143794112/t0(0) o103->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:17/18 lens 328/224 e 0 to 1 dl 1586565572 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [2013625.033840] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [2013625.053451] Lustre: 21898:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1302 previous similar messages [2013627.474592] Lustre: fir-MDT0000-lwp-MDT0001: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) [2013627.484421] Lustre: Skipped 1 previous similar message [2013635.025120] Lustre: 21897:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1586565574/real 1586565574] req@ffff8aec01bad100 x1661617143796800/t0(0) o103->fir-MDT0000-lwp-MDT0001@10.0.10.51@o2ib7:17/18 lens 328/224 e 0 to 1 dl 1586565582 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [2013635.049129] Lustre: fir-MDT0000-lwp-MDT0001: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [2013635.069738] Lustre: 21897:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2057 previous similar messages [2064351.429044] Lustre: fir-MDT0001: haven't heard from client 698e191c-1531-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad559cc8800, cur 1586616297 expire 1586616147 last 1586616070 [2064403.461845] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2064403.470551] Lustre: Skipped 1 previous similar message [2081666.883902] Lustre: fir-MDT0001: haven't heard from client 672e7b7e-01eb-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc98307000, cur 1586633612 expire 1586633462 last 1586633385 [2081719.947476] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2094649.240199] Lustre: fir-MDT0001: haven't heard from client 252c2e21-1c12-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0c1ff58400, cur 1586646594 expire 1586646444 last 1586646367 [2094702.464310] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2111549.462895] Lustre: fir-MDT0001: Client 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) reconnecting [2111549.471383] Lustre: fir-MDT0001: Connection restored to 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) [2111605.924428] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.64@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [2111605.941884] LustreError: Skipped 2 previous similar messages [2111631.013093] Lustre: fir-MDT0001: Client 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) reconnecting [2111631.021567] Lustre: fir-MDT0001: Connection restored to 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) [2140195.499758] Lustre: fir-MDT0001: haven't heard from client e0fef7ff-2654-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8adfc67ac000, cur 1586692139 expire 1586691989 last 1586691912 [2140367.849249] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2141064.941952] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2141123.526016] Lustre: fir-MDT0001: haven't heard from client 1c6a6f4d-068e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd72284400, cur 1586693067 expire 1586692917 last 1586692840 [2141416.762189] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2141469.534843] Lustre: fir-MDT0001: haven't heard from client b415c909-db92-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeca2787c00, cur 1586693413 expire 1586693263 last 1586693186 [2141778.562746] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2141846.548311] Lustre: fir-MDT0001: haven't heard from client b7869070-a0d6-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae95339e000, cur 1586693790 expire 1586693640 last 1586693563 [2142429.427090] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2142458.564188] Lustre: fir-MDT0001: haven't heard from client 058ea76c-4314-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a8522a000, cur 1586694402 expire 1586694252 last 1586694175 [2142658.531045] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2142708.570346] Lustre: fir-MDT0001: haven't heard from client 0258821e-98e2-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af0366ca800, cur 1586694652 expire 1586694502 last 1586694425 [2143088.583717] Lustre: fir-MDT0001: haven't heard from client 1594f2c6-da1e-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed6a83bc00, cur 1586695032 expire 1586694882 last 1586694805 [2143267.184853] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2143860.360993] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2143896.611611] Lustre: fir-MDT0001: haven't heard from client 5f909c98-feb4-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d54c5d800, cur 1586695840 expire 1586695690 last 1586695613 [2144347.709690] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2144389.619534] Lustre: fir-MDT0001: haven't heard from client 39f2741c-70f0-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeba4770400, cur 1586696333 expire 1586696183 last 1586696106 [2144773.863864] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2144852.632049] Lustre: fir-MDT0001: haven't heard from client dd701585-6bc3-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc3ea78400, cur 1586696796 expire 1586696646 last 1586696569 [2145858.645421] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2145905.665333] Lustre: fir-MDT0001: haven't heard from client 9eb4aed3-38f9-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae79a253000, cur 1586697849 expire 1586697699 last 1586697622 [2146309.822244] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2146362.677883] Lustre: fir-MDT0001: haven't heard from client 8da6e638-9ced-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afca289f000, cur 1586698306 expire 1586698156 last 1586698079 [2146911.282468] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2146964.690316] Lustre: fir-MDT0001: haven't heard from client c778e1af-cfbf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6363c000, cur 1586698908 expire 1586698758 last 1586698681 [2147469.160994] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2147515.706642] Lustre: fir-MDT0001: haven't heard from client 9e100a1d-5641-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a7f24000, cur 1586699459 expire 1586699309 last 1586699232 [2148577.330368] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2148625.736575] Lustre: fir-MDT0001: haven't heard from client 208ae758-c4cd-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aee7d2c1400, cur 1586700569 expire 1586700419 last 1586700342 [2149668.466075] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2149708.766555] Lustre: fir-MDT0001: haven't heard from client 76da9219-5db5-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aff43e42800, cur 1586701652 expire 1586701502 last 1586701425 [2150762.300359] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [2150799.797348] Lustre: fir-MDT0001: haven't heard from client 01ee3658-91cf-4 (at 10.49.21.21@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d18212400, cur 1586702743 expire 1586702593 last 1586702516 [2204946.606018] Lustre: 30979:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [2204946.617596] Lustre: 30979:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 153 previous similar messages [2205021.772200] Lustre: 83152:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [2205021.783778] Lustre: 83152:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 129 previous similar messages [2205178.506497] Lustre: 22470:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [2205178.518070] Lustre: 22470:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 238 previous similar messages [2220505.634957] LNet: Service thread pid 22309 was inactive for 226.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [2220505.652077] Pid: 22309, comm: mdt00_006 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [2220505.662447] Call Trace: [2220505.665108] [] call_rwsem_down_write_failed+0x17/0x30 [2220505.672053] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [2220505.679188] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [2220505.686838] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [2220505.694118] [] llog_process_thread+0x82f/0x18e0 [obdclass] [2220505.701502] [] llog_process_or_fork+0xbc/0x450 [obdclass] [2220505.708799] [] llog_cat_process_cb+0x239/0x250 [obdclass] [2220505.716094] [] llog_process_thread+0x82f/0x18e0 [obdclass] [2220505.723465] [] llog_process_or_fork+0xbc/0x450 [obdclass] [2220505.730763] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [2220505.738483] [] llog_cat_process+0x2e/0x30 [obdclass] [2220505.745341] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [2220505.752977] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [2220505.760247] [] mdd_changelog_clear+0x503/0x690 [mdd] [2220505.767093] [] mdd_iocontrol+0x163/0x540 [mdd] [2220505.773418] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [2220505.779740] [] mdt_set_info+0x484/0x490 [mdt] [2220505.785971] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [2220505.793122] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2220505.801051] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [2220505.807586] [] kthread+0xd1/0xe0 [2220505.812689] [] ret_from_fork_nospec_begin+0xe/0x21 [2220505.819338] [] 0xffffffffffffffff [2220505.824554] LustreError: dumping log to /tmp/lustre-log.1586772447.22309 [2220812.221740] LNet: Service thread pid 22309 completed after 532.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [2222506.073309] LNet: Service thread pid 22463 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [2222506.090432] Pid: 22463, comm: mdt00_052 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [2222506.100812] Call Trace: [2222506.103455] [] call_rwsem_down_write_failed+0x17/0x30 [2222506.110428] [] llog_cat_id2handle+0x69/0x5b0 [obdclass] [2222506.117551] [] llog_cat_cancel_records+0x120/0x3c0 [obdclass] [2222506.125222] [] llog_changelog_cancel_cb+0x104/0x2a0 [mdd] [2222506.132502] [] llog_process_thread+0x82f/0x18e0 [obdclass] [2222506.139931] [] llog_process_or_fork+0xbc/0x450 [obdclass] [2222506.147218] [] llog_cat_process_cb+0x239/0x250 [obdclass] [2222506.154506] [] llog_process_thread+0x82f/0x18e0 [obdclass] [2222506.161892] [] llog_process_or_fork+0xbc/0x450 [obdclass] [2222506.169198] [] llog_cat_process_or_fork+0x1e1/0x360 [obdclass] [2222506.176937] [] llog_cat_process+0x2e/0x30 [obdclass] [2222506.183801] [] llog_changelog_cancel.isra.16+0x54/0x1c0 [mdd] [2222506.191428] [] mdd_changelog_llog_cancel+0xd0/0x270 [mdd] [2222506.198722] [] mdd_changelog_clear+0x503/0x690 [mdd] [2222506.205573] [] mdd_iocontrol+0x163/0x540 [mdd] [2222506.211900] [] mdt_iocontrol+0x5ec/0xb00 [mdt] [2222506.218243] [] mdt_set_info+0x484/0x490 [mdt] [2222506.224485] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [2222506.231652] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2222506.239593] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [2222506.246126] [] kthread+0xd1/0xe0 [2222506.251265] [] ret_from_fork_nospec_begin+0xe/0x21 [2222506.257963] [] 0xffffffffffffffff [2222506.263174] LustreError: dumping log to /tmp/lustre-log.1586774447.22463 [2222516.044940] LNet: Service thread pid 22463 completed after 210.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [2274870.838632] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2274943.241339] Lustre: fir-MDT0001: haven't heard from client 0a77aee4-d8bd-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af2c8a4e400, cur 1586826883 expire 1586826733 last 1586826656 [2275147.247000] Lustre: fir-MDT0001: haven't heard from client e02da275-b1c4-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc65e91c00, cur 1586827087 expire 1586826937 last 1586826860 [2275205.361187] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2278227.315546] Lustre: 81300:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2278227.327373] Lustre: 81300:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 19127 previous similar messages [2302476.092912] Lustre: fir-MDT0001: haven't heard from client b8098985-8326-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad2c152c000, cur 1586854415 expire 1586854265 last 1586854188 [2324668.648471] Lustre: fir-MDT0001: haven't heard from client c742d3fe-d253-4 (at 10.49.8.33@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d67ad1000, cur 1586876607 expire 1586876457 last 1586876380 [2325293.062553] Lustre: 22497:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2325293.074383] Lustre: 22497:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 14202 previous similar messages [2335168.274256] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2335208.412785] Lustre: fir-MDT0001: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [2337404.033757] Lustre: fir-MDT0001: haven't heard from client 1620deb5-d90a-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed27a63400, cur 1586889342 expire 1586889192 last 1586889115 [2337457.756990] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2339894.074606] Lustre: fir-MDT0001: haven't heard from client 107c82f8-c563-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6f59f800, cur 1586891832 expire 1586891682 last 1586891605 [2339937.233851] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2345711.247330] Lustre: fir-MDT0001: haven't heard from client 8955f9b5-2c05-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad35c46a000, cur 1586897649 expire 1586897499 last 1586897422 [2345751.342246] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2348965.316606] Lustre: fir-MDT0001: haven't heard from client 8818084b-1959-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad46e773800, cur 1586900903 expire 1586900753 last 1586900676 [2349005.366771] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2351541.406052] Lustre: fir-MDT0001: haven't heard from client 54463c57-4d17-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd4c258000, cur 1586903479 expire 1586903329 last 1586903252 [2351581.986337] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2353675.421886] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2353749.455548] Lustre: fir-MDT0001: haven't heard from client 73d4fdc2-5bc6-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af224368400, cur 1586905687 expire 1586905537 last 1586905460 [2357881.550805] Lustre: fir-MDT0001: haven't heard from client d0ac6bd3-52c8-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d55064c00, cur 1586909819 expire 1586909669 last 1586909592 [2357933.339066] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2378313.134767] Lustre: fir-MDT0001: haven't heard from client 7cc51052-039c-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b054c63f400, cur 1586930250 expire 1586930100 last 1586930023 [2378357.272611] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [2379487.186226] Lustre: fir-MDT0001: haven't heard from client 72a279b9-76fc-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad722769c00, cur 1586931424 expire 1586931274 last 1586931197 [2379719.313846] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2381659.033747] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2381732.228018] Lustre: fir-MDT0001: haven't heard from client 1eda1f50-2242-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad01b074c00, cur 1586933669 expire 1586933519 last 1586933442 [2390494.475936] Lustre: fir-MDT0001: haven't heard from client 36d9a4da-0b63-4 (at 10.50.9.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8affec7c2000, cur 1586942431 expire 1586942281 last 1586942204 [2407869.774089] Lustre: 22500:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407869.785910] Lustre: 22500:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1181 previous similar messages [2407870.275469] Lustre: 22500:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407870.287297] Lustre: 22500:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 549 previous similar messages [2407871.274903] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407871.286728] Lustre: 22409:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 1004 previous similar messages [2407873.274720] Lustre: 68056:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407873.286549] Lustre: 68056:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2082 previous similar messages [2407877.277219] Lustre: 22493:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407877.289042] Lustre: 22493:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 3212 previous similar messages [2407886.007464] Lustre: 22379:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407886.019289] Lustre: 22379:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 7828 previous similar messages [2407902.007666] Lustre: 68057:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407902.019495] Lustre: 68057:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 10170 previous similar messages [2407934.008401] Lustre: 22379:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407934.020225] Lustre: 22379:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 13966 previous similar messages [2407998.010829] Lustre: 22413:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2407998.022648] Lustre: 22413:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 28607 previous similar messages [2420086.684990] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [2420142.311509] Lustre: fir-MDT0001: haven't heard from client 76583765-b356-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d556dec00, cur 1586972078 expire 1586971928 last 1586971851 [2421744.354136] Lustre: fir-MDT0001: haven't heard from client dce6bda0-5113-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aea77e02000, cur 1586973680 expire 1586973530 last 1586973453 [2421790.154660] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2466805.566368] Lustre: fir-MDT0001: haven't heard from client bdf290df-3ae9-4 (at 10.50.2.66@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d799fc400, cur 1587018740 expire 1587018590 last 1587018513 [2481512.978823] Lustre: fir-MDT0001: haven't heard from client de0cac05-64a1-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0a39ea8c00, cur 1587033447 expire 1587033297 last 1587033220 [2497366.298780] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2497560.029162] Lustre: 22477:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2497560.040988] Lustre: 22477:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 32196 previous similar messages [2503891.606615] Lustre: fir-MDT0001: haven't heard from client 8261e53b-f1d4-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad280572400, cur 1587055825 expire 1587055675 last 1587055598 [2503943.254754] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2505024.637905] Lustre: fir-MDT0001: haven't heard from client ff8736f4-8474-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d06bc8c00, cur 1587056958 expire 1587056808 last 1587056731 [2505084.700119] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2507369.700945] Lustre: fir-MDT0001: haven't heard from client 03f69256-011e-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb8e2dc000, cur 1587059303 expire 1587059153 last 1587059076 [2507424.665316] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2526923.066796] Lustre: 22478:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2526923.078623] Lustre: 22478:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 60 previous similar messages [2526925.068202] Lustre: 22478:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2526925.080027] Lustre: 22478:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 2707 previous similar messages [2526929.067862] Lustre: 80232:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2526929.079687] Lustre: 80232:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 4166 previous similar messages [2536891.369975] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2536955.551267] Lustre: fir-MDT0001: haven't heard from client a7f82a61-0ef4-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d3e6a1000, cur 1587088888 expire 1587088738 last 1587088661 [2540203.458610] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2540255.630046] Lustre: fir-MDT0001: haven't heard from client 100bbfda-61da-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afbcaffcc00, cur 1587092188 expire 1587092038 last 1587091961 [2540813.378173] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2540883.643661] Lustre: fir-MDT0001: haven't heard from client 1bd47ab7-bf7d-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aebff626000, cur 1587092816 expire 1587092666 last 1587092589 [2541547.293343] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2541618.663928] Lustre: fir-MDT0001: haven't heard from client d18ea0ef-b0a9-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd436e9c00, cur 1587093551 expire 1587093401 last 1587093324 [2545506.772039] Lustre: fir-MDT0001: haven't heard from client 58701091-a03e-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5a26b800, cur 1587097439 expire 1587097289 last 1587097212 [2545571.397561] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2548558.859043] Lustre: fir-MDT0001: haven't heard from client 70fdd805-10f2-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b06da676000, cur 1587100491 expire 1587100341 last 1587100264 [2548599.437150] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2552966.983159] Lustre: fir-MDT0001: haven't heard from client 3c477ba6-15a5-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8affe5ba0400, cur 1587104899 expire 1587104749 last 1587104672 [2553040.473681] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2555426.053124] Lustre: fir-MDT0001: haven't heard from client ad1e04e7-1b84-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afb5921fc00, cur 1587107358 expire 1587107208 last 1587107131 [2555498.526836] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2556504.083970] Lustre: fir-MDT0001: haven't heard from client 5856b6d8-891c-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af5a6fad800, cur 1587108436 expire 1587108286 last 1587108209 [2556579.830472] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2559066.155842] Lustre: fir-MDT0001: haven't heard from client 6cff9178-04d1-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5b20c400, cur 1587110998 expire 1587110848 last 1587110771 [2559138.566840] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [2560870.991027] LNetError: 21855:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [2560871.001375] LNetError: 21855:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.10.105@o2ib7 (8): c: 2, oc: 0, rc: 8 [2560871.013740] LNetError: 21856:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.105@o2ib7 added to recovery queue. Health = 900 [2560871.278365] LNetError: 79099:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [2560871.290447] LNetError: 79099:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [2560872.278449] Lustre: 21906:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1587112797/real 1587112804] req@ffff8af6106a3180 x1661645979986624/t0(0) o13->fir-OST0022-osc-MDT0001@10.0.10.105@o2ib7:7/4 lens 224/368 e 0 to 1 dl 1587112804 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 [2560872.278459] Lustre: fir-OST001c-osc-MDT0001: Connection to fir-OST001c (at 10.0.10.105@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [2560872.323313] Lustre: 21906:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [2560909.067115] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.7.9@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2560922.406657] Lustre: fir-OST0018-osc-MDT0001: Connection restored to 10.0.10.105@o2ib7 (at 10.0.10.105@o2ib7) [2560929.248443] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.105@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [2560929.265988] LustreError: Skipped 5 previous similar messages [2560954.337108] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.0.10.105@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [2560954.354649] LustreError: Skipped 6 previous similar messages [2560979.425759] Lustre: fir-MDT0001: Client fir-MDT0001-lwp-OST001e_UUID (at 10.0.10.105@o2ib7) reconnecting [2560979.425806] Lustre: fir-MDT0001: Connection restored to fir-MDT0001-lwp-OST0018_UUID (at 10.0.10.105@o2ib7) [2560979.425809] Lustre: Skipped 5 previous similar messages [2560979.450730] Lustre: Skipped 5 previous similar messages [2560984.333128] Lustre: fir-MDT0001: Client 448cf5ef-d74b-4 (at 10.50.7.9@o2ib2) reconnecting [2560984.341518] Lustre: fir-MDT0001: Connection restored to 6dcf0e67-1d32-4 (at 10.50.7.9@o2ib2) [2560984.350131] Lustre: Skipped 5 previous similar messages [2566860.048709] Lustre: fir-MDT0001: Connection restored to e71e196d-00dc-4 (at 10.50.5.33@o2ib2) [2566896.375163] Lustre: fir-MDT0001: haven't heard from client 90a39970-59ef-4 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed6f195000, cur 1587118828 expire 1587118678 last 1587118601 [2600769.344474] Lustre: fir-MDT0001: Client 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) reconnecting [2600769.352936] Lustre: fir-MDT0001: Connection restored to 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) [2609784.560769] Lustre: fir-MDT0001: haven't heard from client eed62e6a-939f-4 (at 10.50.7.35@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d09621c00, cur 1587161715 expire 1587161565 last 1587161488 [2609848.190137] Lustre: fir-MDT0001: Connection restored to eed62e6a-939f-4 (at 10.50.7.35@o2ib2) [2609848.198845] Lustre: Skipped 2 previous similar messages [2612152.974289] Lustre: 22383:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1587164069/real 1587164069] req@ffff8adac9692880 x1661646383784768/t0(0) o104->fir-MDT0001@10.49.0.61@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1587164083 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [2612167.001662] Lustre: 22383:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1587164083/real 1587164083] req@ffff8adac9692880 x1661646383784768/t0(0) o104->fir-MDT0001@10.49.0.61@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1587164097 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [2612208.362206] Lustre: fir-MDT0001: Client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) reconnecting [2612208.370644] Lustre: Skipped 2 previous similar messages [2612208.376076] Lustre: fir-MDT0001: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [2612229.716097] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [2612229.733555] LustreError: Skipped 2 previous similar messages [2612266.961308] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 128s: evicting client at 10.49.0.61@o2ib1 ns: mdt-fir-MDT0001_UUID lock: ffff8af9833a60c0/0xbe1cb198dd2d20bb lrc: 3/0,0 mode: PR/PR res: [0x24005646a:0x410:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.49.0.61@o2ib1 remote: 0x727a59721c6d5b82 expref: 4258351 pid: 23147 timeout: 2612192 lvb_type: 0 [2612273.232141] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [2612308.049122] Lustre: fir-MDT0001: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [2612339.378238] LNet: Service thread pid 22383 was inactive for 200.39s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [2612339.395345] Pid: 22383, comm: mdt00_017 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [2612339.405686] Call Trace: [2612339.408332] [] ldlm_completion_ast+0x430/0x860 [ptlrpc] [2612339.415477] [] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc] [2612339.422843] [] mdt_object_local_lock+0x438/0xb20 [mdt] [2612339.429873] [] mdt_object_lock_internal+0x70/0x360 [mdt] [2612339.437050] [] mdt_object_lock+0x20/0x30 [mdt] [2612339.443363] [] mdt_reint_open+0x106a/0x3240 [mdt] [2612339.449951] [] mdt_reint_rec+0x83/0x210 [mdt] [2612339.456189] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [2612339.462927] [] mdt_intent_open+0x82/0x3a0 [mdt] [2612339.469329] [] mdt_intent_policy+0x435/0xd80 [mdt] [2612339.475981] [] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] [2612339.482927] [] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [2612339.490211] [] tgt_enqueue+0x62/0x210 [ptlrpc] [2612339.496560] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [2612339.503680] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2612339.511582] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [2612339.518098] [] kthread+0xd1/0xe0 [2612339.523188] [] ret_from_fork_nospec_begin+0xe/0x21 [2612339.529842] [] 0xffffffffffffffff [2612339.535041] LustreError: dumping log to /tmp/lustre-log.1587164269.22383 [2612480.004998] LustreError: 22383:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1587164110, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8ada99305c40/0xbe1cb1990eb6b4e0 lrc: 3/0,1 mode: --/CW res: [0x24005646a:0x410:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 22383 timeout: 0 lvb_type: 0 [2612683.219472] LNet: Service thread pid 22383 completed after 544.23s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [2612913.685378] Lustre: fir-MDT0001: haven't heard from client d47ead97-b807-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae90f348800, cur 1587164844 expire 1587164694 last 1587164617 [2612951.128476] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2620944.079049] Lustre: fir-MDT0001: Client 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) reconnecting [2620944.087515] Lustre: fir-MDT0001: Connection restored to 01fdd60b-2214-4 (at 10.50.7.58@o2ib2) [2623787.852284] Lustre: fir-MDT0001: Client ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) reconnecting [2623787.860678] Lustre: fir-MDT0001: Connection restored to ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) [2624115.179615] Lustre: fir-MDT0001: Client ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) reconnecting [2624115.188001] Lustre: fir-MDT0001: Connection restored to ef6c32ab-9321-4 (at 10.50.1.9@o2ib2) [2624116.838378] Lustre: fir-MDT0001: Client 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) reconnecting [2624116.846862] Lustre: fir-MDT0001: Connection restored to 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) [2624141.988909] Lustre: fir-MDT0001: Client 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) reconnecting [2624141.997380] Lustre: fir-MDT0001: Connection restored to 8ba5eec6-ac42-4 (at 10.50.7.57@o2ib2) [2624151.977565] Lustre: fir-MDT0001: Client 4520ec7d-a297-4 (at 10.50.16.5@o2ib2) reconnecting [2624151.986036] Lustre: fir-MDT0001: Connection restored to d2f2b8c6-0661-4 (at 10.50.16.5@o2ib2) [2624198.600718] Lustre: fir-MDT0001: Client 4520ec7d-a297-4 (at 10.50.16.5@o2ib2) reconnecting [2624198.609168] Lustre: Skipped 1 previous similar message [2624198.614519] Lustre: fir-MDT0001: Connection restored to d2f2b8c6-0661-4 (at 10.50.16.5@o2ib2) [2624198.623226] Lustre: Skipped 1 previous similar message [2632449.793338] Lustre: 22399:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2632449.805164] Lustre: 22399:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 5608 previous similar messages [2660740.085516] Lustre: 22377:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2660740.097344] Lustre: 22377:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 555 previous similar messages [2669149.213325] Lustre: fir-MDT0001: haven't heard from client dededc42-b0be-4 (at 10.50.9.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b03a7f25c00, cur 1587221078 expire 1587220928 last 1587220851 [2669181.776779] Lustre: 22493:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2669181.788605] Lustre: 22493:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 339 previous similar messages [2669194.312359] Lustre: fir-MDT0001: Connection restored to f4363950-d6c3-4 (at 10.50.9.37@o2ib2) [2669194.321058] Lustre: Skipped 2 previous similar messages [2676025.978862] Lustre: 67108:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2676025.990692] Lustre: 67108:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 132 previous similar messages [2676026.482869] Lustre: 67108:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [2676026.494703] Lustre: 67108:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 722 previous similar messages [2810331.153962] Lustre: fir-MDT0001: haven't heard from client e6e5cd2b-072e-4 (at 10.49.27.10@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d654e3000, cur 1587362256 expire 1587362106 last 1587362029 [2810912.469808] Lustre: fir-MDT0001: Connection restored to e6e5cd2b-072e-4 (at 10.49.27.10@o2ib1) [2811139.174916] Lustre: fir-MDT0001: haven't heard from client e6e5cd2b-072e-4 (at 10.49.27.10@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0447031000, cur 1587363064 expire 1587362914 last 1587362837 [2833147.787489] Lustre: fir-MDT0001: haven't heard from client 73645170-79e0-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d572fe800, cur 1587385072 expire 1587384922 last 1587384845 [2833206.896145] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2844362.882935] Lustre: fir-MDT0001: Connection restored to 90f2fa35-5a19-4 (at 10.49.29.8@o2ib1) [2844402.778118] Lustre: fir-MDT0001: Connection restored to 2a2adc1f-d7d0-4 (at 10.49.28.12@o2ib1) [2844414.426987] Lustre: fir-MDT0001: Connection restored to 482ab38b-94a8-4 (at 10.49.28.3@o2ib1) [2844427.252301] Lustre: fir-MDT0001: Connection restored to ef345730-39be-4 (at 10.49.28.4@o2ib1) [2844427.261008] Lustre: Skipped 3 previous similar messages [2844448.115159] Lustre: fir-MDT0001: haven't heard from client 16365f08-4eb8-4 (at 10.49.28.6@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d4c0a8000, cur 1587396372 expire 1587396222 last 1587396145 [2844491.278070] Lustre: fir-MDT0001: Connection restored to e8c40140-31a1-4 (at 10.49.28.9@o2ib1) [2844491.286776] Lustre: Skipped 2 previous similar messages [2844518.273659] Lustre: fir-MDT0001: Connection restored to ab87f0a5-0357-4 (at 10.49.29.5@o2ib1) [2844518.282363] Lustre: Skipped 1 previous similar message [2844539.875053] Lustre: fir-MDT0001: Connection restored to 1ce69533-b12b-4 (at 10.49.29.3@o2ib1) [2844539.883751] Lustre: Skipped 1 previous similar message [2844584.084037] Lustre: fir-MDT0001: Connection restored to 4c86e85e-ac4f-4 (at 10.49.28.1@o2ib1) [2844584.092739] Lustre: Skipped 4 previous similar messages [2847122.190840] Lustre: fir-MDT0001: Connection restored to d66e7d02-84db-4 (at 10.49.28.11@o2ib1) [2945480.102949] Lustre: fir-MDT0001: Client d7c9b82a-5f48-4 (at 10.49.18.25@o2ib1) reconnecting [2945480.111482] Lustre: Skipped 2 previous similar messages [2945480.116909] Lustre: fir-MDT0001: Connection restored to d7c9b82a-5f48-4 (at 10.49.18.25@o2ib1) [2946102.756474] Lustre: fir-MDT0001: Connection restored to c6acd997-7594-4 (at 10.50.9.72@o2ib2) [2950813.182766] Lustre: fir-MDT0001: haven't heard from client 4196c68f-076d-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af2c8a4c800, cur 1587502734 expire 1587502584 last 1587502507 [2950813.202824] Lustre: Skipped 16 previous similar messages [2950867.628991] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [2951634.245502] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2951634.253967] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2958683.230512] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2958683.238970] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2958702.747063] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2958702.764521] LustreError: Skipped 1 previous similar message [2958727.835179] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [2958774.044076] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2958774.052534] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2958875.374891] Lustre: fir-MDT0001: Client 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) reconnecting [2958875.383349] Lustre: fir-MDT0001: Connection restored to 8c27500b-ffc7-4 (at 10.50.0.64@o2ib2) [2959112.592061] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3022009.032939] Lustre: fir-MDT0001: Client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) reconnecting [3022009.041398] Lustre: fir-MDT0001: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [3022031.284605] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3039565.597480] Lustre: fir-MDT0001: haven't heard from client 6d071c68-3896-4 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad18437a400, cur 1587591484 expire 1587591334 last 1587591257 [3039624.780154] Lustre: fir-MDT0001: Connection restored to 84be5f83-bbde-4 (at 10.50.8.20@o2ib2) [3048633.248170] Lustre: fir-MDT0001: Client 6a32e96e-045b-4 (at 10.49.0.63@o2ib1) reconnecting [3048633.256644] Lustre: fir-MDT0001: Connection restored to 9823016a-d5f7-4 (at 10.49.0.63@o2ib1) [3108520.821158] LNetError: 36777:0:(o2iblnd_cb.c:2962:kiblnd_rejected()) 10.0.10.217@o2ib7 rejected: o2iblnd fatal error [3108520.831883] LNetError: 36777:0:(o2iblnd_cb.c:2962:kiblnd_rejected()) Skipped 1 previous similar message [3108520.841455] LNetError: 36777:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3108520.853540] LNetError: 36777:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 3 previous similar messages [3108522.032097] Lustre: fir-MDT0001: Client db6de01c-817d-4 (at 10.49.17.26@o2ib1) reconnecting [3108522.040653] Lustre: fir-MDT0001: Connection restored to db6de01c-817d-4 (at 10.49.17.26@o2ib1) [3108523.105858] Lustre: fir-MDT0001: Client da21b453-9110-4 (at 10.49.23.18@o2ib1) reconnecting [3108523.114387] Lustre: Skipped 4 previous similar messages [3108523.119806] Lustre: fir-MDT0001: Connection restored to da21b453-9110-4 (at 10.49.23.18@o2ib1) [3108523.128593] Lustre: Skipped 4 previous similar messages [3108524.145281] LNetError: 36777:0:(o2iblnd_cb.c:2962:kiblnd_rejected()) 10.0.10.219@o2ib7 rejected: o2iblnd fatal error [3108524.155987] LNetError: 36777:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3108524.558781] Lustre: fir-MDT0001: Client 2677fca0-436c-4 (at 10.49.24.30@o2ib1) reconnecting [3108524.567310] Lustre: Skipped 5 previous similar messages [3108524.572738] Lustre: fir-MDT0001: Connection restored to 2677fca0-436c-4 (at 10.49.24.30@o2ib1) [3108524.581540] Lustre: Skipped 5 previous similar messages [3108526.627926] Lustre: fir-MDT0001: Client 1d00195d-6484-4 (at 10.49.23.29@o2ib1) reconnecting [3108526.636459] Lustre: Skipped 14 previous similar messages [3108526.639723] Lustre: fir-MDT0001: Connection restored to ee56fbe2-040d-4 (at 10.49.25.17@o2ib1) [3108526.639724] Lustre: Skipped 14 previous similar messages [3108530.669172] Lustre: fir-MDT0001: Client 49eb42a9-595b-4 (at 10.49.8.18@o2ib1) reconnecting [3108530.677615] Lustre: Skipped 28 previous similar messages [3108530.683125] Lustre: fir-MDT0001: Connection restored to 49eb42a9-595b-4 (at 10.49.8.18@o2ib1) [3108530.691831] Lustre: Skipped 29 previous similar messages [3108538.966305] Lustre: fir-MDT0001: Client 36400b4c-cd69-4 (at 10.49.23.21@o2ib1) reconnecting [3108538.974836] Lustre: Skipped 49 previous similar messages [3108538.980342] Lustre: fir-MDT0001: Connection restored to 06f6e79a-f25a-4 (at 10.49.23.21@o2ib1) [3108538.989127] Lustre: Skipped 49 previous similar messages [3108548.999939] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.23.3@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3108550.467300] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.7.3@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3108554.151784] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.30.21@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3108556.280000] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.24.21@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3108556.297539] LustreError: Skipped 1 previous similar message [3108643.469761] Lustre: fir-MDT0001: Client 5c143de3-c5b2-4 (at 10.49.26.20@o2ib1) reconnecting [3108643.478288] Lustre: Skipped 18 previous similar messages [3108643.483807] Lustre: fir-MDT0001: Connection restored to 5c143de3-c5b2-4 (at 10.49.26.20@o2ib1) [3108643.492589] Lustre: Skipped 18 previous similar messages [3108819.590297] LNetError: 123711:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3109120.612929] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3109120.625101] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3109421.626508] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3109421.638682] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3109722.642775] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3109722.654949] LNetError: 122490:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3110023.657483] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3110023.669651] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3110053.599441] Lustre: fir-MDT0001: Connection restored to bdf290df-3ae9-4 (at 10.50.2.66@o2ib2) [3110053.608140] Lustre: Skipped 3 previous similar messages [3110324.676808] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3110324.688981] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3110625.693367] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3110625.705533] LNetError: 124493:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 2 previous similar messages [3110926.707580] LNetError: 125006:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3110926.719749] LNetError: 125006:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 1 previous similar message [3110986.081326] Lustre: fir-MDT0001: Client 7181ab1a-285e-4 (at 10.49.31.8@o2ib1) reconnecting [3110986.089766] Lustre: Skipped 3 previous similar messages [3110986.095194] Lustre: fir-MDT0001: Connection restored to 7181ab1a-285e-4 (at 10.49.31.8@o2ib1) [3111397.037588] Lustre: fir-MDT0001: Client 2f2088fd-c30d-4 (at 10.49.24.17@o2ib1) reconnecting [3111397.046139] Lustre: fir-MDT0001: Connection restored to 2f2088fd-c30d-4 (at 10.49.24.17@o2ib1) [3111407.452408] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.23.11@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3111407.469952] LustreError: Skipped 3 previous similar messages [3111482.718479] Lustre: fir-MDT0001: Client 21175b94-115a-4 (at 10.49.23.11@o2ib1) reconnecting [3111482.727040] Lustre: fir-MDT0001: Connection restored to 21175b94-115a-4 (at 10.49.23.11@o2ib1) [3112140.341594] LNetError: 36777:0:(o2iblnd_cb.c:2962:kiblnd_rejected()) 10.0.10.219@o2ib7 rejected: o2iblnd fatal error [3112140.352298] LNetError: 36777:0:(lib-msg.c:485:lnet_handle_local_failure()) ni 10.0.10.52@o2ib7 added to recovery queue. Health = 900 [3112140.364385] LNetError: 36777:0:(lib-msg.c:485:lnet_handle_local_failure()) Skipped 6 previous similar messages [3112141.969685] Lustre: fir-MDT0001: Client 56bdeeef-d646-4 (at 10.49.27.24@o2ib1) reconnecting [3112141.978234] Lustre: fir-MDT0001: Connection restored to 56bdeeef-d646-4 (at 10.49.27.24@o2ib1) [3112144.071345] Lustre: fir-MDT0001: Client bbc7752c-d751-4 (at 10.49.30.15@o2ib1) reconnecting [3112144.079897] Lustre: fir-MDT0001: Connection restored to bbc7752c-d751-4 (at 10.49.30.15@o2ib1) [3112146.645373] Lustre: fir-MDT0001: Connection restored to b449688d-0d11-4 (at 10.49.21.21@o2ib1) [3112148.112022] Lustre: fir-MDT0001: Client 4e1c0cee-e6a4-4 (at 10.49.7.2@o2ib1) reconnecting [3112148.120374] Lustre: Skipped 7 previous similar messages [3112149.270046] Lustre: fir-MDT0001: Connection restored to b4d413a8-0b8b-4 (at 10.49.0.71@o2ib1) [3112149.278749] Lustre: Skipped 8 previous similar messages [3112153.822079] Lustre: fir-MDT0001: Connection restored to 203cf276-de8a-4 (at 10.49.7.13@o2ib1) [3112153.830777] Lustre: Skipped 6 previous similar messages [3112156.275831] Lustre: fir-MDT0001: Client 803c3c6b-2f66-4 (at 10.49.18.19@o2ib1) reconnecting [3112156.284356] Lustre: Skipped 11 previous similar messages [3112162.171023] Lustre: fir-MDT0001: Connection restored to 14fe6180-7b80-4 (at 10.49.27.34@o2ib1) [3112162.179814] Lustre: Skipped 10 previous similar messages [3112170.291686] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.26.15@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3112172.696516] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.21.18@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3112172.714077] LustreError: Skipped 1 previous similar message [3112175.338607] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.17.17@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3112185.702307] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.24.19@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3112185.719847] LustreError: Skipped 2 previous similar messages [3112194.323359] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.30.12@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3112194.340905] LustreError: Skipped 1 previous similar message [3112244.696866] Lustre: fir-MDT0001: Client b8b46eec-58d2-4 (at 10.49.28.12@o2ib1) reconnecting [3112244.705398] Lustre: Skipped 16 previous similar messages [3112244.710916] Lustre: fir-MDT0001: Connection restored to 2a2adc1f-d7d0-4 (at 10.49.28.12@o2ib1) [3112244.719710] Lustre: Skipped 8 previous similar messages [3112410.005503] Lustre: fir-MDT0001: Client 4f8d098a-313e-4 (at 10.49.24.16@o2ib1) reconnecting [3112410.014035] Lustre: Skipped 2 previous similar messages [3112410.019492] Lustre: fir-MDT0001: Connection restored to 4f8d098a-313e-4 (at 10.49.24.16@o2ib1) [3112410.028275] Lustre: Skipped 2 previous similar messages [3121619.043068] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121709.287602] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.16.5@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121711.430854] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.7.15@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121711.448312] LustreError: Skipped 10 previous similar messages [3121715.441995] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.8.38@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121715.459467] LustreError: Skipped 11 previous similar messages [3121724.427067] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.10.48@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121724.444616] LustreError: Skipped 9 previous similar messages [3121891.178379] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3121891.195839] LustreError: Skipped 15 previous similar messages [3122007.102346] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.50.2.24@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3122007.119799] LustreError: Skipped 1 previous similar message [3122080.879820] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3122080.897276] LustreError: Skipped 8 previous similar messages [3122215.283913] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.50.0.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3122215.301374] LustreError: Skipped 4 previous similar messages [3139207.481338] Lustre: fir-MDT0001: Connection restored to e6e5cd2b-072e-4 (at 10.49.27.10@o2ib1) [3139207.490122] Lustre: Skipped 10 previous similar messages [3140313.137677] Lustre: fir-MDT0001: Connection restored to 374fbe2d-aad4-4 (at 10.50.9.16@o2ib2) [3141001.401823] Lustre: fir-MDT0001: Connection restored to 1c46010b-e86f-4 (at 10.50.14.3@o2ib2) [3141061.393035] Lustre: fir-MDT0001: haven't heard from client dbaf7e91-ed99-4 (at 10.50.14.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8af067205000, cur 1587692977 expire 1587692827 last 1587692750 [3148529.615575] Lustre: fir-MDT0001: haven't heard from client a18d39c5-c8e2-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0837765400, cur 1587700445 expire 1587700295 last 1587700218 [3148588.391543] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3157087.863911] Lustre: fir-MDT0001: haven't heard from client c2d2e31a-e130-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add5524b800, cur 1587709003 expire 1587708853 last 1587708776 [3157136.608980] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3162985.011696] Lustre: fir-MDT0001: haven't heard from client 7c785842-10d4-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd76275400, cur 1587714900 expire 1587714750 last 1587714673 [3163011.811553] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3173579.301412] Lustre: fir-MDT0001: haven't heard from client 602fd730-17b5-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad8e356b400, cur 1587725494 expire 1587725344 last 1587725267 [3173665.757907] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3188520.716669] Lustre: fir-MDT0001: haven't heard from client d09a0ede-d388-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aecb7209000, cur 1587740435 expire 1587740285 last 1587740208 [3188593.219844] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3196209.920898] Lustre: fir-MDT0001: haven't heard from client 698282da-cb91-4 (at 10.49.30.25@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add7961e000, cur 1587748124 expire 1587747974 last 1587747897 [3196322.597747] Lustre: fir-MDT0001: Connection restored to 3c0f2777-beb3-4 (at 10.50.1.60@o2ib2) [3196330.988894] Lustre: fir-MDT0001: Connection restored to 698282da-cb91-4 (at 10.49.30.25@o2ib1) [3196360.173958] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.30.26@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3196360.191520] LustreError: Skipped 2 previous similar messages [3196398.722258] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.30.27@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3196398.739803] LustreError: Skipped 1 previous similar message [3196435.440050] Lustre: fir-MDT0001: Connection restored to 433c72b8-4873-4 (at 10.49.30.26@o2ib1) [3196448.899438] Lustre: fir-MDT0001: Connection restored to a2e2b1c3-65e8-4 (at 10.49.30.27@o2ib1) [3196486.050709] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.30.28@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3196486.068261] LustreError: Skipped 2 previous similar messages [3196536.228272] Lustre: fir-MDT0001: Connection restored to 98564f4c-0214-4 (at 10.49.30.28@o2ib1) [3196546.105533] Lustre: fir-MDT0001: Connection restored to 4251eba8-5842-4 (at 10.49.30.29@o2ib1) [3197565.967817] Lustre: fir-MDT0001: haven't heard from client 8e988288-7327-4 (at 10.49.8.31@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d75ff1c00, cur 1587749480 expire 1587749330 last 1587749253 [3197565.987876] Lustre: Skipped 5 previous similar messages [3197648.962949] Lustre: fir-MDT0001: haven't heard from client 4cb83747-634b-4 (at 10.49.8.26@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d799fc800, cur 1587749563 expire 1587749413 last 1587749336 [3197648.983009] Lustre: Skipped 1 previous similar message [3197724.985247] Lustre: fir-MDT0001: haven't heard from client 4e1c0cee-e6a4-4 (at 10.49.7.2@o2ib1) in 178 seconds. I think it's dead, and I am evicting it. exp ffff8afd77522800, cur 1587749639 expire 1587749489 last 1587749461 [3197725.005218] Lustre: Skipped 9 previous similar messages [3197800.982158] Lustre: fir-MDT0001: haven't heard from client 0a9b0342-b987-4 (at 10.49.7.19@o2ib1) in 224 seconds. I think it's dead, and I am evicting it. exp ffff8b0d562f9400, cur 1587749715 expire 1587749565 last 1587749491 [3197801.002229] Lustre: Skipped 14 previous similar messages [3200658.577641] Lustre: fir-MDT0001: Connection restored to bb56e686-3cef-4 (at 10.49.8.30@o2ib1) [3200658.586378] Lustre: Skipped 1 previous similar message [3200667.799063] Lustre: fir-MDT0001: Connection restored to 14f6955a-e3d1-4 (at 10.49.8.32@o2ib1) [3200667.807766] Lustre: Skipped 1 previous similar message [3200675.624674] Lustre: fir-MDT0001: Connection restored to c742d3fe-d253-4 (at 10.49.8.33@o2ib1) [3200697.158876] Lustre: fir-MDT0001: Connection restored to aee8fcf3-b4f4-4 (at 10.49.7.3@o2ib1) [3200716.834881] Lustre: fir-MDT0001: Connection restored to d754cff8-a642-4 (at 10.49.7.9@o2ib1) [3200750.159554] Lustre: fir-MDT0001: Connection restored to 0a9b0342-b987-4 (at 10.49.7.19@o2ib1) [3200750.168257] Lustre: Skipped 3 previous similar messages [3200839.910208] Lustre: fir-MDT0001: Connection restored to e1c903ac-38f7-4 (at 10.49.8.20@o2ib1) [3200839.918918] Lustre: Skipped 19 previous similar messages [3200976.662579] Lustre: fir-MDT0001: Connection restored to 6470996b-3104-4 (at 10.49.7.14@o2ib1) [3202648.020292] Lustre: fir-MDT0001: Connection restored to f77ab551-d023-4 (at 10.49.7.5@o2ib1) [3203867.389043] Lustre: fir-MDT0001: Connection restored to e0f8d55d-8a95-4 (at 10.49.8.21@o2ib1) [3208526.907970] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.49.0.61@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3208526.925427] LustreError: Skipped 1 previous similar message [3208656.248483] Lustre: fir-MDT0001: haven't heard from client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afc05ffc400, cur 1587760570 expire 1587760420 last 1587760343 [3208656.268542] Lustre: Skipped 3 previous similar messages [3208693.184255] Lustre: fir-MDT0001: Connection restored to 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) [3208798.653416] Lustre: fir-MDT0001: Client d3db911c-747e-4 (at 10.49.0.62@o2ib1) reconnecting [3208798.661854] Lustre: Skipped 10 previous similar messages [3208798.667370] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [3209024.257698] Lustre: fir-MDT0001: haven't heard from client b0a0091f-966f-4 (at 10.49.7.8@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed79fff400, cur 1587760938 expire 1587760788 last 1587760711 [3212605.350260] Lustre: fir-MDT0001: haven't heard from client f1e95053-36e6-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aebff627400, cur 1587764519 expire 1587764369 last 1587764292 [3212683.221149] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3226705.725650] Lustre: fir-MDT0001: haven't heard from client 1b076dfe-4a07-4 (at 10.49.27.19@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd6f599000, cur 1587778619 expire 1587778469 last 1587778392 [3246528.265189] Lustre: fir-MDT0001: haven't heard from client 0b00d912-3aaf-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad2ea97bc00, cur 1587798441 expire 1587798291 last 1587798214 [3246615.719310] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3260893.652132] Lustre: fir-MDT0001: haven't heard from client f3b8a3c2-e0a8-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8adce2977c00, cur 1587812806 expire 1587812656 last 1587812579 [3260993.328694] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3265561.777721] Lustre: fir-MDT0001: haven't heard from client 62c0bc08-3bbe-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0733637c00, cur 1587817474 expire 1587817324 last 1587817247 [3265639.511056] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3306057.844148] Lustre: fir-MDT0001: haven't heard from client 31da0f95-39f8-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad514d89800, cur 1587857969 expire 1587857819 last 1587857742 [3306142.294751] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3312767.018601] Lustre: fir-MDT0001: haven't heard from client d8e7c41e-e2d2-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b082ef0d800, cur 1587864678 expire 1587864528 last 1587864451 [3312838.893879] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3315199.197577] Lustre: fir-MDT0001: Client 80cdbcdc-4411-4 (at 10.50.10.70@o2ib2) reconnecting [3315199.203524] Lustre: fir-MDT0001: Connection restored to 933e4cd4-a6c0-4 (at 10.50.10.63@o2ib2) [3315199.214890] Lustre: Skipped 1 previous similar message [3330027.511025] Lustre: fir-MDT0001: haven't heard from client 65bc2e5d-4393-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad5fa7da400, cur 1587881938 expire 1587881788 last 1587881711 [3330103.201622] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3330103.210337] Lustre: Skipped 1 previous similar message [3334309.627529] Lustre: fir-MDT0001: haven't heard from client 0d7fc5d8-ed14-4 (at 10.50.7.68@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d799fec00, cur 1587886220 expire 1587886070 last 1587885993 [3334972.640615] Lustre: fir-MDT0001: haven't heard from client 46fffebc-0282-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad8095bbc00, cur 1587886883 expire 1587886733 last 1587886656 [3335043.675488] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3360310.360278] Lustre: fir-MDT0001: haven't heard from client 24b9c830-af4c-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afbc66c2800, cur 1587912220 expire 1587912070 last 1587911993 [3360383.991983] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3363034.657920] Lustre: fir-MDT0001: Client d3db911c-747e-4 (at 10.49.0.62@o2ib1) reconnecting [3363034.666386] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [3365179.482254] Lustre: fir-MDT0001: haven't heard from client 2b334f66-989b-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afd5a268400, cur 1587917089 expire 1587916939 last 1587916862 [3365217.485397] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3406938.199263] Lustre: fir-MDT0001: Connection restored to 846622f7-6989-4 (at 10.50.5.12@o2ib2) [3406990.619100] Lustre: fir-MDT0001: haven't heard from client 846622f7-6989-4 (at 10.50.5.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d78559000, cur 1587958899 expire 1587958749 last 1587958672 [3410328.050740] Lustre: fir-MDT0001: Client 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) reconnecting [3410328.059294] Lustre: fir-MDT0001: Connection restored to 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) [3410328.643923] Lustre: fir-MDT0001: Client ce0de346-3437-4 (at 10.49.30.10@o2ib1) reconnecting [3410328.652462] Lustre: fir-MDT0001: Connection restored to ce0de346-3437-4 (at 10.49.30.10@o2ib1) [3410328.661247] Lustre: Skipped 1 previous similar message [3410329.892998] Lustre: fir-MDT0001: Client da21b453-9110-4 (at 10.49.23.18@o2ib1) reconnecting [3410329.901528] Lustre: Skipped 10 previous similar messages [3410329.907040] Lustre: fir-MDT0001: Connection restored to da21b453-9110-4 (at 10.49.23.18@o2ib1) [3410329.915824] Lustre: Skipped 9 previous similar messages [3410331.914268] Lustre: fir-MDT0001: Client 27105f38-d1f2-4 (at 10.49.25.22@o2ib1) reconnecting [3410331.922799] Lustre: Skipped 10 previous similar messages [3410331.928312] Lustre: fir-MDT0001: Connection restored to 27105f38-d1f2-4 (at 10.49.25.22@o2ib1) [3410331.937097] Lustre: Skipped 10 previous similar messages [3410335.984531] Lustre: fir-MDT0001: Client 75eb7c2d-56d1-4 (at 10.49.26.8@o2ib1) reconnecting [3410335.992972] Lustre: Skipped 64 previous similar messages [3410335.998481] Lustre: fir-MDT0001: Connection restored to 75eb7c2d-56d1-4 (at 10.49.26.8@o2ib1) [3410336.007176] Lustre: Skipped 65 previous similar messages [3410344.034877] Lustre: fir-MDT0001: Client 49c076ac-79a3-4 (at 10.49.18.29@o2ib1) reconnecting [3410344.043406] Lustre: Skipped 106 previous similar messages [3410344.049009] Lustre: fir-MDT0001: Connection restored to 49c076ac-79a3-4 (at 10.49.18.29@o2ib1) [3410344.057797] Lustre: Skipped 105 previous similar messages [3410360.243026] Lustre: fir-MDT0001: Client 051aface-8044-4 (at 10.49.27.28@o2ib1) reconnecting [3410360.251552] Lustre: Skipped 209 previous similar messages [3410360.257153] Lustre: fir-MDT0001: Connection restored to 051aface-8044-4 (at 10.49.27.28@o2ib1) [3410360.265939] Lustre: Skipped 209 previous similar messages [3410428.383786] Lustre: fir-MDT0001: Client 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) reconnecting [3410428.392332] Lustre: Skipped 22 previous similar messages [3410428.397862] Lustre: fir-MDT0001: Connection restored to 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) [3410428.406659] Lustre: Skipped 22 previous similar messages [3410528.738571] Lustre: fir-MDT0001: Client 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) reconnecting [3410528.747102] Lustre: Skipped 427 previous similar messages [3410528.752714] Lustre: fir-MDT0001: Connection restored to 05195c7f-e5e4-4 (at 10.49.20.27@o2ib1) [3410528.761494] Lustre: Skipped 427 previous similar messages [3410579.419974] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3410610.336021] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3410610.353473] LustreError: Skipped 1 previous similar message [3410656.898318] Lustre: fir-MDT0001: Client 05d9bd56-36ab-4 (at 10.49.26.1@o2ib1) reconnecting [3410656.906758] Lustre: Skipped 773 previous similar messages [3410656.912353] Lustre: fir-MDT0001: Connection restored to 05d9bd56-36ab-4 (at 10.49.26.1@o2ib1) [3410656.921051] Lustre: Skipped 773 previous similar messages [3410755.108115] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3410971.306369] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3410996.394993] Lustre: fir-MDT0001: Client d3db911c-747e-4 (at 10.49.0.62@o2ib1) reconnecting [3410996.403437] Lustre: Skipped 106 previous similar messages [3410996.409045] Lustre: fir-MDT0001: Connection restored to eb57335a-b614-4 (at 10.49.0.62@o2ib1) [3410996.417761] Lustre: Skipped 106 previous similar messages [3411041.278357] LustreError: 42101:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b0c7ea79f80 x1661798185183808/t0(0) o37->d3db911c-747e-4@10.49.0.62@o2ib1:145/0 lens 448/440 e 1 to 0 dl 1587962955 ref 1 fl Interpret:/0/0 rc 0/0 [3411063.853003] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3411114.030432] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3411114.047883] LustreError: Skipped 1 previous similar message [3411237.201604] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3411393.966380] Lustre: 23116:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1587963295/real 0] req@ffff8aece3b32400 x1661669913208192/t0(0) o104->fir-MDT0001@10.49.0.61@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1587963302 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [3411412.281797] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3411501.086424] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 114s: evicting client at 10.49.0.61@o2ib1 ns: mdt-fir-MDT0001_UUID lock: ffff8b0c7b7f1d40/0xbe1cb1b7037183e8 lrc: 3/0,0 mode: PR/PR res: [0x24003709b:0x30:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.49.0.61@o2ib1 remote: 0x727a597244f6f77b expref: 4408 pid: 42285 timeout: 3411404 lvb_type: 0 [3411508.262624] Lustre: 105470:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1587963409/real 0] req@ffff8afe546a6780 x1661669914146432/t0(0) o104->fir-MDT0001@10.49.0.64@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1587963416 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [3411508.500264] Lustre: fir-MDT0001: Client d6bb72a8-4bc4-4 (at 10.49.25.23@o2ib1) reconnecting [3411508.508790] Lustre: Skipped 1221 previous similar messages [3411508.509950] Lustre: fir-MDT0001: Connection restored to 924fdc99-68a1-4 (at 10.49.7.11@o2ib1) [3411508.509951] Lustre: Skipped 1221 previous similar messages [3411559.918422] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.219@o2ib7 added to recovery queue. Health = 900 [3411609.936703] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.217@o2ib7 added to recovery queue. Health = 900 [3411634.952541] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.219@o2ib7 added to recovery queue. Health = 900 [3411643.090466] LustreError: 21976:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 142s: evicting client at 10.49.0.64@o2ib1 ns: mdt-fir-MDT0001_UUID lock: ffff8b0adb33a1c0/0xbe1cb1b70e6c3773 lrc: 3/0,0 mode: PR/PR res: [0x24003709b:0x30:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.49.0.64@o2ib1 remote: 0xb5dde8f5a9ef5205 expref: 55 pid: 22415 timeout: 3411546 lvb_type: 0 [3411684.970878] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.217@o2ib7 added to recovery queue. Health = 900 [3411684.984006] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 2 previous similar messages [3411709.996540] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.219@o2ib7 added to recovery queue. Health = 900 [3411760.014100] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.217@o2ib7 added to recovery queue. Health = 900 [3411760.027220] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 2 previous similar messages [3411769.765341] Lustre: fir-MDT0001: haven't heard from client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) in 203 seconds. I think it's dead, and I am evicting it. exp ffff8aed1f6df800, cur 1587963678 expire 1587963528 last 1587963475 [3411785.038898] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.216@o2ib7 added to recovery queue. Health = 900 [3411845.766060] Lustre: fir-MDT0001: haven't heard from client a5d32a00-dfb8-4 (at 10.49.18.20@o2ib1) in 160 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b332000, cur 1587963754 expire 1587963604 last 1587963594 [3411885.058478] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.216@o2ib7 added to recovery queue. Health = 900 [3411885.071597] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 2 previous similar messages [3411921.788971] Lustre: fir-MDT0001: haven't heard from client 0f2d3178-05c2-4 (at 10.49.0.64@o2ib1) in 211 seconds. I think it's dead, and I am evicting it. exp ffff8af6b0a5e000, cur 1587963830 expire 1587963680 last 1587963619 [3411951.161082] LustreError: 137-5: fir-MDT0002_UUID: not available for connect from 10.49.0.62@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server. [3411951.178543] LustreError: Skipped 1 previous similar message [3411997.765408] Lustre: fir-MDT0001: haven't heard from client 0dc74af1-84ad-4 (at 10.49.0.61@o2ib1) in 206 seconds. I think it's dead, and I am evicting it. exp ffff8b0837766c00, cur 1587963906 expire 1587963756 last 1587963700 [3411997.785466] Lustre: Skipped 3 previous similar messages [3412010.094029] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.217@o2ib7 added to recovery queue. Health = 900 [3412010.107151] LNetError: 21872:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 3 previous similar messages [3448703.009255] Lustre: 22421:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [3448703.021079] Lustre: 22421:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 189 previous similar messages [3467726.106837] Lustre: fir-MDT0001: Client b33b6709-9bde-4 (at 10.50.0.62@o2ib2) reconnecting [3467726.115292] Lustre: Skipped 1935 previous similar messages [3467726.121025] Lustre: fir-MDT0001: Connection restored to b33b6709-9bde-4 (at 10.50.0.62@o2ib2) [3467726.129731] Lustre: Skipped 1946 previous similar messages [3470420.982659] Lustre: fir-MDT0001: Client ab9422cb-105a-4 (at 10.50.1.48@o2ib2) reconnecting [3470420.991102] Lustre: Skipped 5 previous similar messages [3470420.996535] Lustre: fir-MDT0001: Connection restored to ab9422cb-105a-4 (at 10.50.1.48@o2ib2) [3470421.005297] Lustre: Skipped 5 previous similar messages [3472661.010446] Lustre: fir-MDT0001: Client 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) reconnecting [3472661.018915] Lustre: fir-MDT0001: Connection restored to 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) [3472665.625883] Lustre: fir-MDT0001: Client 51387ec6-ed83-4 (at 10.50.10.48@o2ib2) reconnecting [3472665.634414] Lustre: Skipped 3 previous similar messages [3472665.639846] Lustre: fir-MDT0001: Connection restored to 51387ec6-ed83-4 (at 10.50.10.48@o2ib2) [3472665.648652] Lustre: Skipped 3 previous similar messages [3472668.097272] Lustre: fir-MDT0001: Client 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) reconnecting [3472668.105717] Lustre: Skipped 1 previous similar message [3472668.111063] Lustre: fir-MDT0001: Connection restored to 2f3a422b-dba8-4 (at 10.50.0.63@o2ib2) [3472668.119764] Lustre: Skipped 1 previous similar message [3481242.142180] Lustre: fir-MDT0001: Connection restored to 1b076dfe-4a07-4 (at 10.49.27.19@o2ib1) [3481369.633770] Lustre: fir-MDT0001: Connection restored to d6539a1b-0c76-4 (at 10.49.7.8@o2ib1) [3495417.397490] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [3495417.409315] Lustre: 22442:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 264 previous similar messages [3495417.900188] Lustre: 63252:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [3495417.912009] Lustre: 63252:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 249 previous similar messages [3495418.900464] Lustre: 67110:0:(mdd_device.c:1811:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [3495418.912291] Lustre: 67110:0:(mdd_device.c:1811:mdd_changelog_clear()) Skipped 652 previous similar messages [3522863.845793] Lustre: fir-MDT0001: haven't heard from client 9028409b-8293-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeb51afa000, cur 1588074769 expire 1588074619 last 1588074542 [3522863.865868] Lustre: Skipped 1 previous similar message [3522905.348685] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3537129.014522] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537129.022996] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537150.777502] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537150.785948] Lustre: Skipped 2 previous similar messages [3537150.791413] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537150.800118] Lustre: Skipped 2 previous similar messages [3537179.798094] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537179.806536] Lustre: Skipped 1 previous similar message [3537179.811883] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537179.820610] Lustre: Skipped 1 previous similar message [3537187.513305] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537187.521746] Lustre: Skipped 2 previous similar messages [3537187.527173] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537187.535890] Lustre: Skipped 2 previous similar messages [3537206.547801] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537206.556246] Lustre: Skipped 2 previous similar messages [3537206.561682] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537206.570389] Lustre: Skipped 2 previous similar messages [3537225.105274] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3537225.113720] Lustre: Skipped 11 previous similar messages [3537225.119240] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537225.127960] Lustre: Skipped 11 previous similar messages [3537242.829839] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3537292.470002] Lustre: fir-MDT0001: Client 40f84ba4-941c-4 (at 10.50.0.71@o2ib2) reconnecting [3537292.470669] Lustre: fir-MDT0001: Connection restored to 86f43a12-df24-4 (at 10.50.7.61@o2ib2) [3537292.470671] Lustre: Skipped 1 previous similar message [3537292.492444] Lustre: Skipped 3 previous similar messages [3538569.623397] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3538569.631844] Lustre: Skipped 1 previous similar message [3538569.637193] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3538569.645890] Lustre: Skipped 2 previous similar messages [3542029.632700] Lustre: fir-MDT0001: Client 01162551-7145-4 (at 10.50.17.29@o2ib2) reconnecting [3542029.641274] Lustre: Skipped 1 previous similar message [3542029.646743] Lustre: fir-MDT0001: Connection restored to 01162551-7145-4 (at 10.50.17.29@o2ib2) [3542029.655570] Lustre: Skipped 1 previous similar message [3542037.656746] Lustre: fir-MDT0001: Client e6075956-5d7a-4 (at 10.50.7.60@o2ib2) reconnecting [3542037.665276] Lustre: fir-MDT0001: Connection restored to e6075956-5d7a-4 (at 10.50.7.60@o2ib2) [3542045.969655] Lustre: fir-MDT0001: Client d7d7110e-deef-4 (at 10.50.17.28@o2ib2) reconnecting [3542045.978247] Lustre: fir-MDT0001: Connection restored to d7d7110e-deef-4 (at 10.50.17.28@o2ib2) [3542081.609471] Lustre: fir-MDT0001: Client fb6eed47-408e-4 (at 10.50.8.33@o2ib2) reconnecting [3542081.617995] Lustre: fir-MDT0001: Connection restored to fb6eed47-408e-4 (at 10.50.8.33@o2ib2) [3542173.196644] Lustre: fir-MDT0001: Client d87117b6-a627-4 (at 10.50.2.11@o2ib2) reconnecting [3542173.205096] Lustre: Skipped 1 previous similar message [3542173.210453] Lustre: fir-MDT0001: Connection restored to d87117b6-a627-4 (at 10.50.2.11@o2ib2) [3542173.219183] Lustre: Skipped 1 previous similar message [3549305.937204] LustreError: 22417:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-OH' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-OH' to finish migration. [3566052.996559] Lustre: fir-MDT0001: haven't heard from client def74d96-2eea-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ada98685400, cur 1588117957 expire 1588117807 last 1588117730 [3566078.788360] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3566078.797066] Lustre: Skipped 1 previous similar message [3568013.061894] Lustre: fir-MDT0001: haven't heard from client 2fb882fb-4fa3-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae422d54400, cur 1588119917 expire 1588119767 last 1588119690 [3568036.434798] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3570990.517378] Lustre: fir-MDT0001: Client c0d8f986-a521-4 (at 10.50.10.64@o2ib2) reconnecting [3570990.525910] Lustre: Skipped 1 previous similar message [3570990.531271] Lustre: fir-MDT0001: Connection restored to c0d8f986-a521-4 (at 10.50.10.64@o2ib2) [3574095.508411] Lustre: fir-MDT0001: Client 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) reconnecting [3574095.516880] Lustre: fir-MDT0001: Connection restored to 8bb8b052-263d-4 (at 10.50.6.69@o2ib2) [3578863.990288] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [3578864.001854] Lustre: 22430:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 136 previous similar messages [3578903.452761] Lustre: 68055:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [3578903.464330] Lustre: 68055:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 33 previous similar messages [3578978.939660] Lustre: 22491:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [3578978.951230] Lustre: 22491:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 62 previous similar messages [3579129.753826] Lustre: 28970:0:(llog_cat.c:894:llog_cat_process_or_fork()) fir-MDD0001: catlog [0x2a4:0xa:0x0] crosses index zero [3579129.765393] Lustre: 28970:0:(llog_cat.c:894:llog_cat_process_or_fork()) Skipped 128 previous similar messages [3587908.587468] Lustre: fir-MDT0001: haven't heard from client dcec0564-0e94-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afb64298800, cur 1588139812 expire 1588139662 last 1588139585 [3587935.531889] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3590045.649275] Lustre: fir-MDT0001: haven't heard from client 95fc548f-457e-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec82f7e800, cur 1588141949 expire 1588141799 last 1588141722 [3602749.385785] LustreError: 31649:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '05-full-bare' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 05-full-bare' to finish migration. [3602749.402468] LustreError: 31649:0:(mdd_dir.c:4496:mdd_migrate()) Skipped 1 previous similar message [3602987.904721] LustreError: 21981:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '07-pyrochlore' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 07-pyrochlore' to finish migration. [3603740.589313] LustreError: 22430:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '09-MoN' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 09-MoN' to finish migration. [3603748.776315] LustreError: 22430:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '05-TI' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 05-TI' to finish migration. [3603748.791794] LustreError: 22430:0:(mdd_dir.c:4496:mdd_migrate()) Skipped 1 previous similar message [3603766.999785] LustreError: 22412:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '04-PMOF-Calero-Vaspsol' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 04-PMOF-Calero-Vaspsol' to finish migration. [3604091.174444] LustreError: 22413:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'a-forme' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 a-forme' to finish migration. [3604355.658358] LustreError: 22462:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '00-references' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 00-references' to finish migration. [3604626.054546] LustreError: 22494:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'a-forme' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 a-forme' to finish migration. [3605140.910773] LustreError: 22492:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'a-forme' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 a-forme' to finish migration. [3605402.620600] LustreError: 22309:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '05-Rh' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 05-Rh' to finish migration. [3605490.619952] LustreError: 22478:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '11-waterbarrier' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 11-waterbarrier' to finish migration. [3605490.637159] LustreError: 22478:0:(mdd_dir.c:4496:mdd_migrate()) Skipped 41 previous similar messages [3620264.820272] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3621901.652029] LustreError: 22470:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3622272.103281] LustreError: 22470:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3622408.385577] LustreError: 22399:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3622604.256862] LustreError: 22437:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3622604.272152] LustreError: 22437:0:(mdd_dir.c:4496:mdd_migrate()) Skipped 1 previous similar message [3632710.559916] LustreError: 22012:0:(out_handler.c:643:out_write()) fir-MDT0001: empty buf for xattr set: rc = -61 [3632767.393099] LustreError: 42990:0:(out_handler.c:643:out_write()) fir-MDT0001: empty buf for xattr set: rc = -61 [3632767.403368] LustreError: 42990:0:(out_handler.c:643:out_write()) Skipped 1 previous similar message [3653001.208565] LustreError: 22377:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3653096.730688] LustreError: 22377:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3653194.967396] LustreError: 22473:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3653194.982692] LustreError: 22473:0:(mdd_dir.c:4496:mdd_migrate()) Skipped 1 previous similar message [3653364.848089] LustreError: 22469:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. [3669460.809724] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3669580.528850] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3669700.260035] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3669821.954438] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3669940.940592] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670061.546818] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670180.273369] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670301.002956] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670421.111358] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670541.102392] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670660.859745] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670780.579965] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3670900.980849] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671020.710552] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671140.434436] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671261.154824] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671380.386069] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671502.306165] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671621.026062] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671740.746659] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671860.473462] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3671990.092632] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672100.821707] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672220.583058] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672341.760742] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672460.844555] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672581.388183] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672701.107592] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672821.626796] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3672940.341996] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673067.119638] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673180.833023] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673300.549921] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673421.268202] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673542.187195] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673661.127186] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673780.843724] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3673900.560797] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674024.155634] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674141.744777] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674260.465637] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674381.186411] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674500.907785] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674620.629667] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674741.485970] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674861.599165] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3674981.309327] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675102.167312] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675224.711830] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675341.792919] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675460.508443] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675581.243155] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675701.216714] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675820.925472] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3675940.979961] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676060.702565] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676181.225093] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676302.297691] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676421.374921] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676542.019049] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676660.730120] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676780.443843] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3676901.889684] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677021.389181] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677141.097630] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677260.803143] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677381.536767] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677502.242652] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677621.866986] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677740.620795] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677861.353672] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3677981.077918] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678101.850983] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678221.290282] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678341.000313] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678461.266668] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678580.975383] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678701.575919] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678822.547900] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3678941.263192] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679060.978885] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679180.691847] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679301.966765] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679420.677546] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679541.394425] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679663.314740] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679781.049052] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3679901.626261] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680021.346010] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680141.449886] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680261.161708] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680389.439876] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680501.162447] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680620.879241] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680740.594989] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680861.312098] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3680981.469890] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681101.185725] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681221.646487] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681341.382201] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681462.317984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681581.841410] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681701.565395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681821.287229] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3681941.052984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682062.183119] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682180.894805] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682300.615744] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682421.333800] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682541.047319] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682661.095381] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682780.808511] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3682906.573704] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683022.225227] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683142.155987] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683260.867372] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683381.581987] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683501.302112] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683621.018052] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683741.846625] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683861.808832] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3683982.769641] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684101.506055] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684221.252014] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684341.470681] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684461.188734] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684581.219906] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684701.227534] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684820.940157] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3684941.043316] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685060.761142] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685181.475136] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685301.192385] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685420.908559] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685542.118423] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685661.885359] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685781.597047] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3685901.313234] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686021.028729] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686140.737630] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686261.448442] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686381.598762] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686502.457729] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686621.171605] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686740.885841] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686861.603546] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3686981.317735] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687102.742642] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687221.457397] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687342.268691] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687460.986815] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687581.702125] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687701.621691] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687822.508584] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3687941.785229] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688061.505991] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688181.968743] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688301.682075] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688421.397123] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688541.426752] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688661.146294] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688780.863708] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3688901.803595] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689021.684736] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689141.404514] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689261.115024] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689380.827864] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689501.543876] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689621.253351] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689741.389168] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689861.201086] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3689981.099995] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690100.813556] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690221.453546] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690341.171455] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690462.770974] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690581.486873] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690702.391251] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690821.102106] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3690941.828935] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691061.757181] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691182.199783] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691301.514970] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691421.249874] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691541.118065] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691669.966263] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691781.681887] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3691901.402129] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692021.633377] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692142.274203] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692261.193110] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692380.919421] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692501.637734] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692621.363085] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692741.078776] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692862.452971] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3692981.490164] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693101.197666] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693220.913263] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693341.639634] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693470.032930] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693582.598404] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693701.985868] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693822.317751] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3693941.264647] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694061.086650] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694181.804737] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694301.514515] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694423.314167] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694541.704946] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694661.489999] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694781.204392] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3694901.918473] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695022.875136] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695141.590680] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695262.911693] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695381.624499] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695501.335661] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695621.101060] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695741.812300] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695861.525402] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3695981.233084] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696102.524528] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696222.272128] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696341.029321] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696461.737988] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696582.807676] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696701.522704] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696821.929361] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3696941.996788] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697061.702190] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697182.332777] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697301.041641] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697421.515055] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697541.222904] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697663.183012] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697782.089668] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3697901.802481] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698021.517816] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698141.224534] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698261.939958] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698381.654951] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698502.653630] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698622.480343] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698741.192975] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698861.908301] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3698981.621266] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699101.623434] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699221.339244] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699341.062970] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699461.929786] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699582.670758] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699701.385280] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699821.101928] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3699942.911298] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700061.629536] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700183.024238] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700301.741770] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700421.451450] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700541.163201] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700661.177285] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700781.947379] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3700902.953823] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701022.575413] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701142.274537] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701261.985274] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701381.756327] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701501.480663] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701621.482265] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701741.832070] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701862.043828] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3701981.754366] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702101.488168] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702221.234341] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702341.948841] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702461.663380] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702581.377848] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702703.352492] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702822.057279] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3702941.766043] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703061.477031] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703181.457757] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703301.169584] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703421.887406] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703542.498819] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703661.780352] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703781.490057] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3703902.421997] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704021.206395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704143.740310] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704261.447520] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704382.157593] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704502.225047] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704621.935694] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704743.239319] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704861.965988] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3704981.684085] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705101.851008] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705221.558208] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705342.112683] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705461.832487] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705582.805810] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705701.523778] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705821.238677] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3705947.832064] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706061.549320] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706181.258176] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706301.977908] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706422.588735] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706541.305544] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706663.644832] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706781.360335] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3706903.238742] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707021.958749] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707141.670645] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707262.221385] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707381.338281] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707503.129018] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707621.846675] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707741.561274] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707863.462097] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3707982.176755] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708102.971622] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708221.681937] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708342.663882] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708461.933621] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708581.649834] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708702.391690] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708822.110220] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3708942.370395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709062.083719] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709182.328683] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709302.113972] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709421.829976] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709541.542434] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709662.258282] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709782.003352] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3709901.716844] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710023.250941] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710141.965225] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710261.677513] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710381.392725] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710502.117265] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710622.770848] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710741.487625] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710863.506589] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3710987.926078] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711101.663697] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711222.425059] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711342.136820] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711461.847925] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711581.561258] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711702.278290] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711822.017397] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3711941.729651] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712061.438429] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712182.151832] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712301.867375] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712421.587950] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712542.303418] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712662.660258] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712783.605437] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3712902.675755] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713022.388956] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713142.845984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713262.481476] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713383.229647] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713501.948941] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713621.661577] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713742.374004] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713862.098124] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3713983.716062] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714102.439734] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714222.162672] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714342.949764] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714461.662702] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714582.409154] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714702.123733] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714823.033354] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3714941.792623] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715061.518899] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715182.237749] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715301.956966] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715421.867938] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715446.120209] Lustre: fir-MDT0001: haven't heard from client 1a6c6519-2c58-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad76dc6dc00, cur 1588267346 expire 1588267196 last 1588267119 [3715477.943798] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3715541.756566] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715662.534799] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715782.255051] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3715901.973758] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716022.266157] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716141.981822] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716261.698049] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716382.804180] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716502.018184] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716621.751709] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716741.729939] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716862.435700] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3716879.657048] LustreError: 23198:0:(out_handler.c:643:out_write()) fir-MDT0001: empty buf for xattr set: rc = -61 [3716879.667309] LustreError: 23198:0:(out_handler.c:643:out_write()) Skipped 1 previous similar message [3716982.143747] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717103.278165] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717169.042541] LustreError: 23198:0:(out_handler.c:643:out_write()) fir-MDT0001: empty buf for xattr set: rc = -61 [3717169.052801] LustreError: 23198:0:(out_handler.c:643:out_write()) Skipped 1 previous similar message [3717221.995945] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717341.716349] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717462.439742] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717582.781005] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717702.492406] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717822.206575] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3717942.780392] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718063.530566] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718182.248327] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718301.968554] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718421.698309] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718545.203807] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718661.923339] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718781.640375] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3718902.980571] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719021.707778] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719143.460076] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719262.836556] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719363.595549] LustreError: 22417:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3719382.558729] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719502.289126] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719622.013405] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719743.832479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719750.894265] LustreError: 22427:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3719862.556610] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3719982.723106] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720102.445003] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720222.160399] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720342.256925] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720461.989099] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720582.000386] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720701.722839] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720821.986820] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3720943.023213] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721025.310561] LustreError: 22495:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3721044.336945] LustreError: 22483:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3721061.967963] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721125.276750] Lustre: fir-MDT0001: haven't heard from client ab9dfcb2-2a8f-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8add60e9bc00, cur 1588273025 expire 1588272875 last 1588272798 [3721163.021518] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3721181.682672] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721303.560256] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721423.180164] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721541.901915] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721662.653261] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721782.406671] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3721903.003971] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722021.733827] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722143.329714] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722262.897001] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722269.308966] Lustre: fir-MDT0001: haven't heard from client 42eccf22-530f-4 (at 10.50.9.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8afb62ff0c00, cur 1588274169 expire 1588274019 last 1588273942 [3722312.366938] Lustre: fir-MDT0001: Connection restored to 59748e26-ab50-4 (at 10.50.9.27@o2ib2) [3722383.733147] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722502.508873] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722622.223899] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722682.098689] LustreError: 22495:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3722743.078593] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722861.800131] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3722982.521186] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723102.279708] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723223.185157] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723342.693543] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723462.411045] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723582.407862] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723702.127381] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723821.837947] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3723943.130030] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724063.920771] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724182.639510] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724302.352995] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724422.068111] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724541.973933] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724662.721870] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724782.696386] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3724903.548153] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725021.775175] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725142.501525] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725262.228834] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725381.956461] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725501.847185] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725623.754533] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725742.990611] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725862.713274] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3725982.433242] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726103.219056] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726224.169847] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726341.897043] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726462.639119] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726582.374618] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726702.855136] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726822.716164] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3726942.435498] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727062.160678] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727181.882577] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727303.662451] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727423.277608] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727427.176711] LustreError: 22383:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3727542.071678] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727662.806095] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727707.876302] LustreError: 22427:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3727782.530891] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3727844.043673] LustreError: 83152:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3727902.259222] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728021.978113] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728144.011767] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728262.768472] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728382.636853] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728502.317305] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728622.074625] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728742.807553] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728863.870460] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3728984.478176] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729102.212369] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729222.936774] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729342.718384] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729463.514161] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729582.543737] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729703.387880] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729822.113288] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3729942.841849] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730064.070242] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730182.799719] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730302.530133] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730422.276916] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730542.044698] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730662.814386] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730782.215517] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3730902.332474] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731022.059315] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731143.318046] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731262.460029] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731383.373351] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731502.127984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731622.909516] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731742.845654] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731862.567329] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3731982.945504] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732101.982994] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732224.002666] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732342.727042] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732462.455695] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732582.189797] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732702.928502] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732823.874499] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3732942.647349] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733062.379473] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733182.108479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733304.138996] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733422.868511] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733542.600433] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733663.410532] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733782.140582] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3733903.864702] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734022.586355] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734142.316436] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734262.046831] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734383.036051] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734504.202135] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734622.931899] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734742.667810] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734862.408141] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3734982.149721] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735104.300687] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735222.233076] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735343.876869] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735462.607809] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735582.412351] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735703.069276] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735822.803457] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3735943.329048] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736063.062837] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736183.609604] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736302.626767] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736422.380777] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736542.764969] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736662.700999] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736782.510101] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3736902.243752] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737023.259425] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737143.007562] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737264.048108] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737382.795594] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737503.755176] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737624.516477] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737742.248677] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737863.521196] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3737982.254708] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738103.007876] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738222.242959] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738342.986084] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738464.099355] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738582.245566] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738703.824670] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738822.553567] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3738946.020223] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739065.157953] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739182.895999] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739303.295045] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739423.071531] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739543.553618] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739663.052008] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739793.299072] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3739903.060222] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740022.825917] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740143.051771] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740262.791216] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740383.375307] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740503.118521] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740624.014292] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740742.761239] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740862.493091] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3740982.617483] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741102.362765] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741224.491432] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741342.223127] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741462.969642] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741583.944893] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741702.386697] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741823.555423] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3741942.293016] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742064.224881] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742183.981669] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742302.711995] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742423.153247] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742542.875796] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742663.245828] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742783.721461] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3742902.441200] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743034.106098] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743142.823729] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743262.545293] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743383.460164] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743503.226862] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743622.985063] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743743.658467] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743786.889522] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6b695c00, cur 1588295686 expire 1588295536 last 1588295459 [3743862.375028] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3743983.201193] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744103.717948] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744222.440337] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744343.172198] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744462.890918] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744582.692307] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744702.435556] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744823.200336] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3744943.037335] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745062.753952] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745139.932696] Lustre: fir-MDT0001: haven't heard from client f3d4dc47-07cf-4 (at 10.50.10.42@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0d6dd07c00, cur 1588297039 expire 1588296889 last 1588296812 [3745182.480838] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745303.334195] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745423.068980] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745542.786643] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745662.918179] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745783.773361] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3745902.496247] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746023.221199] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746144.105203] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746262.826173] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746384.765896] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746502.569942] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746623.297422] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746743.797276] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746862.521030] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3746983.260400] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747104.144171] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747223.902665] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747343.738028] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747462.457608] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747583.182940] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747702.909208] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747822.642890] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3747943.392267] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748063.171084] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748183.522077] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748303.250774] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748422.971325] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748542.695486] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748662.412479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748783.135162] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3748902.852980] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749023.005073] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749142.722357] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749262.442620] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749383.165643] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749503.566732] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749624.480352] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749712.312193] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3749744.000890] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749862.789620] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3749939.066764] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ad463a28800, cur 1588301838 expire 1588301688 last 1588301611 [3749982.499827] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750104.217027] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750222.952003] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750344.401354] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750464.231150] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750582.950452] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750703.940630] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750822.659320] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3750943.380175] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751063.087744] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751183.871094] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751303.594263] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751423.299860] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751543.308298] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751663.020214] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751782.732390] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3751812.362735] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3751904.757485] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752023.474419] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752039.128512] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aec77e51c00, cur 1588303938 expire 1588303788 last 1588303711 [3752143.186178] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752262.897872] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752383.158998] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752502.959423] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752536.422713] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3752622.676114] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752743.387050] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752763.148806] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae11a0cd000, cur 1588304662 expire 1588304512 last 1588304435 [3752863.124015] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3752982.874260] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753102.605533] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753223.323113] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753343.041943] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753463.886643] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753471.170574] Lustre: fir-MDT0001: haven't heard from client b8a11533-b764-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed6b64b800, cur 1588305370 expire 1588305220 last 1588305143 [3753533.616116] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3753583.405836] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753703.846696] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753822.564390] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3753943.286770] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754063.006545] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754182.722801] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754304.590143] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754423.310997] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754543.032495] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754665.100383] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754783.211086] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3754903.294901] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755023.019222] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755144.024306] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755263.623182] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755383.348687] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755503.079430] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755622.824559] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755743.553013] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755863.266843] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3755983.526405] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756103.264526] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756223.016029] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756342.733479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756464.225855] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756582.941131] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756702.676585] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756823.389278] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3756943.115438] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757063.868425] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757184.119769] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757302.835059] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757425.863800] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757543.584710] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757664.315984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757783.044012] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3757902.858988] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758025.020570] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758143.223463] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758263.312169] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758383.031146] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758502.761114] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758623.889634] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758742.989989] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758862.920787] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3758983.647521] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759103.363706] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759223.446503] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759343.754406] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759463.479097] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759583.202163] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759704.221290] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759824.578991] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3759943.305556] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760063.021633] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760183.296529] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760303.021876] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760423.937801] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760544.104302] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760662.831535] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760785.881265] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3760903.604482] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761023.991071] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761143.713315] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761263.429979] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761383.146105] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761502.867774] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761530.429336] Lustre: fir-MDT0001: Connection restored to f3d4dc47-07cf-4 (at 10.50.10.42@o2ib2) [3761623.593705] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761743.317490] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761863.642612] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3761984.950578] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762103.674903] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762223.390860] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762343.108134] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762463.312854] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762584.573802] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762703.294922] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762823.023530] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3762943.752267] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763064.688589] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763184.170004] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763303.031260] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763423.037324] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763543.754567] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763663.472081] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763783.920716] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3763904.288857] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764023.241050] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764143.855143] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764263.571425] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764383.538395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764503.252313] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764622.968222] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764743.682979] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764863.401658] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3764868.715336] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3764983.614554] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765095.501116] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae440f7c000, cur 1588316994 expire 1588316844 last 1588316767 [3765103.363940] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765223.094609] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765345.234710] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765462.948508] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765584.904523] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765703.731214] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765824.245258] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3765942.962248] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766063.682284] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766183.392521] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766303.107356] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766424.673082] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766544.684368] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766663.929145] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766783.641150] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3766903.360613] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767023.075957] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767144.241984] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767262.954026] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767383.079526] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767503.803617] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767623.523296] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767743.832224] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767863.556288] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3767983.272698] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768103.890604] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768223.629152] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768344.211757] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768463.963241] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768583.697601] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768703.453619] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768823.170028] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3768945.007747] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769063.057730] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769183.777604] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769303.494022] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769423.212170] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769543.928397] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769663.639616] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769783.354757] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3769903.071786] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770023.791396] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770143.502473] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770263.218771] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770383.945939] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770504.557421] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770624.564416] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770743.282100] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770863.994997] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3770983.703833] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771104.079385] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771223.809741] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771343.527552] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771464.307286] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771584.033950] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771703.883913] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771823.609975] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3771943.326495] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772064.040685] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772183.755365] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772300.943723] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3772305.457751] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772423.173620] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772527.710167] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aeaabf60000, cur 1588324426 expire 1588324276 last 1588324199 [3772543.892169] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772663.605141] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772783.317953] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3772903.395510] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773023.110535] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773145.174598] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773265.115822] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773383.830308] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773503.558247] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773623.280743] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773744.003709] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773864.848660] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3773984.858383] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774103.576040] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774223.291549] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774344.016921] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774464.597197] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774583.324735] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774704.042683] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774824.525338] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3774943.244389] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775064.186833] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775183.913092] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775303.632308] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775423.354801] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775544.092402] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775665.525516] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775783.245367] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3775903.966737] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776023.725594] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776143.437215] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776265.400822] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776384.116337] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776503.839819] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776624.875462] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776743.717176] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776863.488979] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3776984.202575] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777105.120954] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777224.402125] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777343.381296] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777464.106359] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777583.826407] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777703.542215] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777824.024555] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3777944.606890] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778063.321100] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778186.023248] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778303.958121] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778424.883292] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778543.593587] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778663.307403] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778784.075332] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3778903.791339] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779024.409515] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779144.129969] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779264.819476] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779383.532701] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779504.244620] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779624.844343] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779743.567689] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779864.286711] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3779984.763258] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780104.519484] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780224.237534] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780343.954442] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780464.546825] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780584.261129] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780703.983764] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780823.701721] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3780944.419564] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781064.140358] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781183.862560] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781304.048882] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781423.770491] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781543.486857] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781664.215855] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781783.929875] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3781903.648698] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782024.310873] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782144.036185] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782263.756394] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782383.472463] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782504.190890] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782623.907945] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782743.623424] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782863.979518] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3782984.817024] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783103.531730] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783224.255617] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783344.582653] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783465.453898] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783585.829441] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783703.579691] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783824.301605] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3783944.073387] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784063.808454] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784184.264137] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784209.283224] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3784304.987404] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784423.567699] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784436.046899] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8acd6b521c00, cur 1588336334 expire 1588336184 last 1588336107 [3784544.288057] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784664.003262] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784783.969382] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3784903.686472] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785024.405697] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785144.653942] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785264.849818] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785384.346853] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785505.199478] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785623.914593] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785743.628788] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785864.349554] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3785984.065096] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786105.618901] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786224.344205] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786344.061789] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786464.061682] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786583.774279] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786704.584448] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786825.587980] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3786945.134867] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787063.942800] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787183.670229] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787304.388935] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787424.105944] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787544.864170] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787663.917747] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787784.489741] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3787904.256601] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788023.968645] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788144.486657] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788264.211198] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788383.928128] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788503.645881] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788624.370706] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788745.014831] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788863.730612] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3788984.454496] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789104.168555] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789223.883974] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789344.066206] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789463.791723] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789585.031483] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789703.753789] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789824.476819] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3789944.184791] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790064.687535] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790184.402330] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790304.445291] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790425.048562] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790543.760452] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790664.471435] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790784.192672] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3790903.913571] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791023.625627] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791144.341062] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791264.899047] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791384.615811] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791505.200536] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791623.909728] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791743.898883] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791863.924992] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3791984.311316] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792104.825610] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792225.758316] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792344.476781] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792464.613842] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792584.908368] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792704.618881] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792825.432762] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3792944.150159] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793063.873259] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793184.591868] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793304.307831] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793425.677305] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793544.009472] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793663.967641] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793784.683716] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3793904.401969] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794024.612821] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794145.247041] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794265.280126] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794384.605771] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794504.320075] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794624.119640] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794743.831752] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794866.554665] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3794984.930445] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795104.881258] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795224.596396] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795344.311682] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795464.029143] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795584.062373] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795703.778911] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795824.513294] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3795944.499618] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796064.965682] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796185.517298] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796304.228665] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796423.945363] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796545.992347] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796665.546986] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796748.819719] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3796785.501520] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796904.260113] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3796975.398161] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed0e6c8c00, cur 1588348873 expire 1588348723 last 1588348646 [3797023.975311] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797144.688902] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797264.408128] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797384.731067] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797505.380474] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797625.119476] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797743.934698] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797864.667513] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3797984.382387] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798104.098479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798223.811665] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798345.080267] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798464.697683] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798485.658591] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3798584.441784] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798704.155479] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798712.448113] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ae40ed00800, cur 1588350610 expire 1588350460 last 1588350383 [3798823.874584] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3798946.603424] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799064.318802] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799184.929582] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799304.647273] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799425.309203] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799544.034354] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799664.002463] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799784.717990] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3799905.465761] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800025.696673] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800145.108284] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800264.829252] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800384.550470] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800504.272731] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800624.540535] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800744.450957] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800865.363748] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3800984.710921] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801104.451234] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801224.168331] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801344.883454] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801464.596013] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801584.321177] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801705.238937] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801824.279310] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3801950.559010] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802064.316860] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802185.225533] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802305.771563] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802425.604641] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802544.339570] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802664.081626] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802784.804936] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3802904.538755] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803024.269440] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803143.997957] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803264.731498] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803384.450221] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803504.667419] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803624.398645] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803745.246676] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803863.971309] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3803985.506156] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804104.256465] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804223.985273] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804344.730339] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804465.401291] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804584.146193] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804707.251331] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804824.967793] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3804945.411076] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805064.808421] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805184.527395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805305.226434] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805425.697165] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805544.358656] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805664.264033] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805751.642407] Lustre: fir-MDT0001: haven't heard from client 03a3ed52-d0e7-4 (at 10.50.6.54@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b08bc15cc00, cur 1588357649 expire 1588357499 last 1588357422 [3805785.180875] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3805809.592180] Lustre: fir-MDT0001: Connection restored to eceee209-ec05-4 (at 10.50.6.54@o2ib2) [3805904.912946] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806024.634585] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806144.362438] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806264.089235] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806384.805945] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806504.546865] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806624.733180] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806744.453648] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806817.892555] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3806865.203012] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3806984.932398] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807044.678802] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8aed7e50d400, cur 1588358942 expire 1588358792 last 1588358715 [3807105.730346] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807224.457390] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807338.306891] LustreError: 28226:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3807345.149997] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807464.890437] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807485.967148] LustreError: 68057:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3807584.970908] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807704.700150] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807719.857401] LustreError: 62242:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3807824.435189] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807945.212419] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3807993.873838] LustreError: 22425:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. [3808065.493299] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808184.227216] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808306.173135] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808425.165982] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808544.915312] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808664.656726] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808784.747333] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3808905.698043] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809024.409989] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809144.138012] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809264.894073] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809384.630267] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809505.071542] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809625.447804] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809744.178889] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809864.921655] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3809984.660379] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810105.328631] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810226.354774] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810345.093337] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810464.835764] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810584.560054] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810705.040830] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810824.776631] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3810944.813045] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811064.291357] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811185.025582] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811305.911399] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811424.680891] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811544.454060] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811665.557055] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811784.670518] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3811904.400057] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812025.142767] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812145.498906] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812265.096709] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812385.574211] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812504.308741] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812627.042306] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812744.771642] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812864.638353] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3812985.447507] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813104.501693] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813226.653903] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813344.376707] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813465.113208] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813584.873395] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813704.595780] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813826.345458] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3813944.334995] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814065.928423] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814184.653832] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814304.410120] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814425.779013] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814544.540776] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814665.880956] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814785.725416] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3814905.613436] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815025.290393] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815145.039223] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815264.797174] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815384.919289] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815505.529994] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815625.254975] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815744.989423] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815864.650844] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3815984.380097] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816105.126079] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816225.195308] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816346.213197] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816464.948664] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816584.723945] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816704.464847] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816831.736441] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3816944.793405] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817064.539154] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817186.924253] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817304.659667] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817425.216199] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817544.929953] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817664.454714] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817785.452027] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3817905.596637] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818025.610753] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818144.964651] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818264.696324] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818385.208684] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818504.952513] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818625.161717] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818727.759865] Lustre: fir-MDT0001: Connection restored to 0d7fc5d8-ed14-4 (at 10.50.7.68@o2ib2) [3818744.905705] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818866.613331] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3818985.357591] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819105.082214] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819224.825357] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819344.559724] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819465.645535] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819584.696909] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819706.312553] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819825.043196] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3819944.804656] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820064.846347] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820184.610384] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820304.926903] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820424.663419] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820546.098444] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820594.290446] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3820664.830515] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820784.556734] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3820821.056734] Lustre: fir-MDT0001: haven't heard from client 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b0ce7fbdc00, cur 1588372718 expire 1588372568 last 1588372491 [3820905.287483] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821025.567643] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821145.289564] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821265.019351] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821385.682694] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821505.437291] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821625.160673] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821744.883594] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821864.607017] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3821987.286261] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822105.017898] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822225.361682] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822345.071950] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822464.786549] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822584.502317] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822704.538749] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822826.636317] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3822945.359128] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823065.078382] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823185.367636] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823305.604696] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823425.352424] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823545.069802] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823666.128597] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823784.861144] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3823905.604573] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824025.318588] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824146.275036] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824265.001140] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824385.355269] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824505.079740] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824620.448341] LustreError: 22344:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x2400576ec:0x149ae:0x0]: rc = -2 [3824624.993901] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824640.928979] LustreError: 42016:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0xcf72:0x0]: rc = -2 [3824658.778134] LustreError: 22546:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0xe2f2:0x0]: rc = -2 [3824678.792561] LustreError: 42121:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0xf468:0x0]: rc = -2 [3824696.395767] LustreError: 22546:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0x10477:0x0]: rc = -2 [3824714.310806] LustreError: 42123:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0x11206:0x0]: rc = -2 [3824730.506605] LustreError: 42121:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0x120e7:0x0]: rc = -2 [3824745.343556] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824768.104569] LustreError: 22344:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0x13f7d:0x0]: rc = -2 [3824768.117096] LustreError: 22344:0:(mdd_object.c:3249:mdd_close()) Skipped 1 previous similar message [3824822.439361] LustreError: 28226:0:(mdd_object.c:400:mdd_xattr_get()) fir-MDD0001: object [0x240057703:0x16d59:0x0] not found: rc = -2 [3824840.123675] LustreError: 22344:0:(mdd_object.c:3249:mdd_close()) fir-MDD0001: failed to get lu_attr of [0x240057703:0x17b2e:0x0]: rc = -2 [3824840.136203] LustreError: 22344:0:(mdd_object.c:3249:mdd_close()) Skipped 2 previous similar messages [3824865.060885] mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) [3824898.399369] Lustre: fir-MDT0001: Connection restored to 7862f6c9-0098-4 (at 10.50.8.41@o2ib2) [3824911.223375] LustreError: 22403:0:(lod_dev.c:132:lod_fld_lookup()) fir-MDT0001-mdtlov: invalid FID [0x0:0x0:0x0] [3824911.233641] LustreError: 22403:0:(lu_object.c:146:lu_object_put()) ASSERTION( top->loh_hash.next == ((void *)0) && top->loh_hash.pprev == ((void *)0) ) failed: [3824911.248150] LustreError: 22403:0:(lu_object.c:146:lu_object_put()) LBUG [3824911.254941] Pid: 22403, comm: mdt00_022 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [3824911.265305] Call Trace: [3824911.267941] [] libcfs_call_trace+0x8c/0xc0 [libcfs] [3824911.274687] [] lbug_with_loc+0x4c/0xa0 [libcfs] [3824911.281077] [] lu_object_put+0x336/0x3e0 [obdclass] [3824911.287838] [] lu_object_put_nocache+0x16/0x20 [obdclass] [3824911.295127] [] lu_object_find_at+0x1fe/0xa60 [obdclass] [3824911.302240] [] lu_object_find+0x16/0x20 [obdclass] [3824911.308908] [] mdt_object_find+0x4b/0x170 [mdt] [3824911.315306] [] mdt_migrate_lookup.isra.40+0x158/0xa60 [mdt] [3824911.322778] [] mdt_reint_migrate+0x8ea/0x1310 [mdt] [3824911.329526] [] mdt_reint_rec+0x83/0x210 [mdt] [3824911.335765] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [3824911.342508] [] mdt_reint+0x67/0x140 [mdt] [3824911.348401] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [3824911.355537] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [3824911.363446] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [3824911.369949] [] kthread+0xd1/0xe0 [3824911.375036] [] ret_from_fork_nospec_begin+0xe/0x21 [3824911.381683] [] 0xffffffffffffffff [3824911.386893] Kernel panic - not syncing: LBUG [3824911.391339] CPU: 28 PID: 22403 Comm: mdt00_022 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 [3824911.404191] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.10.6 08/15/2019 [3824911.412016] Call Trace: [3824911.414649] [] dump_stack+0x19/0x1b [3824911.419967] [] panic+0xe8/0x21f [3824911.424938] [] lbug_with_loc+0x9b/0xa0 [libcfs] [3824911.431323] [] lu_object_put+0x336/0x3e0 [obdclass] [3824911.438044] [] ? lu_object_start.isra.35+0x8b/0x120 [obdclass] [3824911.445715] [] lu_object_put_nocache+0x16/0x20 [obdclass] [3824911.452951] [] lu_object_find_at+0x1fe/0xa60 [obdclass] [3824911.460011] [] ? lod_xattr_get+0xee/0x700 [lod] [3824911.466387] [] lu_object_find+0x16/0x20 [obdclass] [3824911.473014] [] mdt_object_find+0x4b/0x170 [mdt] [3824911.479378] [] mdt_migrate_lookup.isra.40+0x158/0xa60 [mdt] [3824911.486780] [] mdt_reint_migrate+0x8ea/0x1310 [mdt] [3824911.493499] [] ? check_unlink_entry+0x19/0xd0 [obdclass] [3824911.500654] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [3824911.508318] [] mdt_reint_rec+0x83/0x210 [mdt] [3824911.514503] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [3824911.521213] [] mdt_reint+0x67/0x140 [mdt] [3824911.527097] [] tgt_request_handle+0xada/0x1570 [ptlrpc] [3824911.534180] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [3824911.541925] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [3824911.549176] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [3824911.557038] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [3824911.564004] [] ? __wake_up+0x44/0x50 [3824911.569438] [] ptlrpc_main+0xb34/0x1470 [ptlrpc] [3824911.575911] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [3824911.583478] [] kthread+0xd1/0xe0 [3824911.588528] [] ? insert_kthread_work+0x40/0x40 [3824911.594796] [] ret_from_fork_nospec_begin+0xe/0x21 [3824911.601407] [] ? insert_kthread_work+0x40/0x40